Book Image

Secret Recipes of the Python Ninja

Book Image

Secret Recipes of the Python Ninja

Overview of this book

This book covers the unexplored secrets of Python, delve into its depths, and uncover its mysteries. You’ll unearth secrets related to the implementation of the standard library, by looking at how modules actually work. You’ll understand the implementation of collections, decimals, and fraction modules. If you haven’t used decorators, coroutines, and generator functions much before, as you make your way through the recipes, you’ll learn what you’ve been missing out on. We’ll cover internal special methods in detail, so you understand what they are and how they can be used to improve the engineering decisions you make. Next, you’ll explore the CPython interpreter, which is a treasure trove of secret hacks that not many programmers are aware of. We’ll take you through the depths of the PyPy project, where you’ll come across several exciting ways that you can improve speed and concurrency. Finally, we’ll take time to explore the PEPs of the latest versions to discover some interesting hacks.
Table of Contents (17 chapters)
Title Page
Copyright and Credits
Packt Upsell
Foreword
Contributors
Preface
Index

Using local patches and constraint files


The benefit of open-source software is the ability to view and modify source code. If you are working on a project and create a local version of a PyPI module, such as customizing for a project or creating a patch, requirements.txt can be used to override the normal download of the file.

Constraints files are a modification of requirements files that simply indicate what version of a library is installed, but they don't actually control the installation of files.

One example of using a constraints file is when using a local patched version of a PyPI module, for example, ReqFile. Some software packages downloaded from PyPI rely on ReqFile, but other packages don't. Rather than writing a requirements file for every single package from PyPI that depends on ReqFile, a constraints file can be created as a master record and implemented across all Python projects. Any package being installed that requires ReqFile will see the constraints file and install from the local repository, rather than from PyPI.

In this manner, a single file can be used by every developer and it no longer matters what a PyPI package depends on; the correct version will either be pulled down from PyPI, or the local version will be used as needed.

How to do it...

  1. Tag the in-house version of the file. Assuming you are using Git, a tag is generated by using the following:
      git tag -a <tag_name> -m "<tag_message>"
      # git tag -a v0.3 -m "Changed the calculations"
  1. Upload it to the version control system.
  2. Indicate the local version in the requirements.txt file, as shown in the following example:
      git+https://<vcs>/<dependency>@<tag_name>#egg=<dependency>
      # git+https://gitlab/[email protected]#egg=pump_laws
  1. Write the constraints.txt file in the same manner as a requirements.txt file. The following example comes from https://github.com/mldbai/mldb (this was released under the Apache v2.0 license by MLDB.ai):
      # math / science / graph stuff
      bokeh==0.11.1
      numpy==1.10.4
      pandas==0.17.1
      scipy==0.17.0
      openpyxl==2.3.3
      patsy==0.4.1
      matplotlib==1.5.1
      ggplot==0.6.8
      Theano==0.7.0
      seaborn==0.7.0
      scikit-learn==0.17

      pymldb==0.8.1
      pivottablejs==0.1.0

      # Progress bar
      tqdm==4.11.0

      # notebook and friends
      ipython==5.1.0
      jupyter==1.0.0
      jupyter-client==4.4.0
      jupyter-console==5.0.0
      jupyter-core==4.2.1

      # validator
      uWSGI==2.0.12
      pycrypto==2.6.1

      tornado==4.4.2

      ## The following requirements were added by pip freeze:
      backports-abc==0.5
      backports.shutil-get-terminal-size==1.0.0
      backports.ssl-match-hostname==3.5.0.1
      bleach==1.5.0

      ***further files truncated***
  1. Next, run the command, pip install -c constraints.txt, to make the file available to Python.

How it works...

In the preceding example, <vcs> is the version control system being used; it could be a local server or an online service such as, GitHub. <tag_name> is the version control tag used to identify this particular update to the control system.

If a required dependency was a top-level requirement for the project, then that particular line in the requirements file can simply be replaced. If it is a sub-dependency of another file, then the above command would be added as a new line.

There's more...

Constraints files differ from requirements files in one key way: putting a package in the constraints file does not cause the package to be installed, whereas a requirements file will install all packages listed. Constraints files are simply requirements files that control which version of a package will be installed, but provide no control over the actual installation.