The most convenient tool to install and manage Python packages is pip. Rather than installing the packages one by one, it is possible to define a list of packages that you want to install as the contents of a text file. We can pass the text file into the pip tool, which will then handle the installation of all packages in the list automatically. An added benefit to this approach is that the package list can be stored in version control.

Generally speaking, it is ideal and often sufficient to have a single requirements file that directly matches your production environment. You can change versions or add and remove dependencies on a development machine and then manage them through version control. This way, going from one set of dependencies (and associated code changes) to another can be as simple as switching branches.

In some cases, environments differ enough that you will need to have at least two different instances of your project:

• The development environment, where you create new features
• The public website environment, which is usually called the production environment in a hosted server

There might be development environments for other developers, or special tools that are needed during development but that are unnecessary in production. You might also have a testing and staging environment in order to test the project locally and in a public website-like setup.

For good maintainability, you should be able to install the required Python modules for development, testing, staging, and production environments. Some of the modules will be shared and some of them will be specific to a subset of the environments. In this recipe, we will learn how to organize the project dependencies for multiple environments and manage them with pip.

Before using this recipe, you need to have a Django project ready with pip installed and a virtual environment activated. For more information on how to do this, read the Working with a virtual environment recipe.

Execute the following steps one by one to prepare pip requirements for your virtual environment Django project:

1. Let's go to the Django project that you have under version control and create a requirements directory with the following text files:

• _base.txt for shared modules
• dev.txt for the development environment
• test.txt for the testing environment
• staging.txt for the staging environment
• production.txt for production

2. Edit _base.txt and add the Python modules that are shared in all environments, line by line:

# requirements/_base.txt
Django~=3.0.4
djangorestframework
-e git://github.com/omab/python-social-auth.git@6b1e301c79#egg=python-social-auth
Copy

3. If the requirements of a specific environment are the same as in _base.txt, add the line including _base.txt in the requirements file of that environment, as shown in the following example:

# requirements/production.txt
-r _base.txt
Copy

4. If there are specific requirements for an environment, add them after the _base.txt inclusion, as shown in the following code:

# requirements/dev.txt
-r _base.txt
coverage
django-debug-toolbar
selenium
Copy

5. You can run the following command in a virtual environment in order to install all of the required dependencies for the development environment (or an analogous command for other environments), as follows:

(env)$ pip install -r requirements/dev.txt
Copy

The preceding pip install command, whether it is executed explicitly in a virtual environment or at the global level, downloads and installs all of your project dependencies from requirements/_base.txt and requirements/dev.txt. As you can see, you can specify a version of the module that you need for the Django framework and even directly install it from a specific commit at the Git repository, as is done for python-social-auth in our example.

When you have many dependencies in your project, it is good practice to stick to a narrow range of release versions for Python module release versions. Then you can have greater confidence that the project integrity will not be broken because of updates in your dependencies, which might contain conflicts or backward incompatibility. This is particularly important when deploying your project or handing it off to a new developer.

If you have already manually installed the project requirements with pip one by one, you can generate the requirements/_base.txt file using the following command within your virtual environment:

(env)$ pip freeze > requirements/_base.txt
Copy

If you want to keep things simple and are sure that, for all environments, you will be using the same dependencies, you can use just one file for your requirements named requirements.txt, generated by definition, as shown in the following:

(env)$ pip freeze > requirements.txt
Copy

To install the modules in a new virtual environment, simply use the following command:

(env)$ pip install -r requirements.txt
Copy

If you need to install a Python library from another version control system, or on a local path, then you can learn more about pip from the official documentation at https://pip.pypa.io/en/stable/user_guide/.

Another approach to managing Python dependencies that is getting more and more popular is Pipenv. You can get it and learn about it at https://github.com/pypa/pipenv.

• The Working with a virtual environment recipe
• The Working with Docker containers for Django, Gunicorn, Nginx, and PostgreSQL recipe
• The Including external dependencies in your project recipe
• The Configuring settings for development, testing, staging, and production environments recipe