Writing Python at GDS
About this manual
This manual is designed to aid developers in writing Python code that is clear and consistent, within, and across, projects at GDS.
We follow PEP 8; where PEP 8 doesn’t express a view (e.g. on the usage of language features such as metaclasses) we defer to the Google Python style guide. Use these as references unless something is explicitly mentioned in the GDS Python Style Guide.
If you want to add a new rule or exception please create a pull request against this repo.
For more on linting (including standard settings and config) see the linting section of this manual.
Flake8 is the preferred linting tool. It incorporates:
- PEP 8 inspired style checks using pycodestyle
- Complexity checking using the McCabe project
- Lint checks using Pyflakes
A Python application project typically brings together Python packages from PyPI, with others written in-house (or otherwise not distributed via PyPI).
These packages are the applications immediate dependencies. Additionally, any package can have its own immediate dependencies, where they draw on other packages. From an application’s point of view, the dependencies of the packages it requires are its sub-dependencies.
The below diagram shows a simplified view of the resulting pyramid of dependencies; however it is important to note that the hierarchy can repeat itself infinitely.
Application | +-----+-----+ | | | Immediate dependencies | | | +--+--+--+--+--+--+--+--+ | | | | | | | | | ... Sub-dependencies ...
There are two ways of specifying dependencies in Python world: as a specific version or as an allowable range.
Different considerations apply to dependency management depending whether you are packaging a library, or creating an end-product such as an application or a bundle of scripts. In general, you should only specify specific versions when creating a Python system that sits at the top of the dependency pyramid; otherwise there is a danger of creating version conflicts.
These recommendations apply wherever you need a reproducible set of dependencies, such as a complete web application, perhaps with many dependencies and sub-dependencies. It also applies to a collection of scripts that are deployed into the cloud and run automatically (for example, batch jobs).
A good strategy for specifying your application’s Python dependencies has two desirable characteristics - they should be:
Pin your application’s full dependencies – specific versions, rather than ranges – or you’ll get unpredictability between your dev environment and other environments. You want a new starter to avoid small hard-to-spot problems. And you want parity between what you test locally, what is tested by CI, and what you deploy, or you risk new issues appearing on a live server. Additionally these things can be hard to diagnose.
Security issues are found in libraries, so it is important to choose libraries that are maintained and to ensure your team has a strategy to ensure security updates are installed without significant delay. The how to manage third party software dependencies section gives further context and discusses tools that can help, such as snyk.io.
Your README should document an easy-to-follow process by which all your Python dependencies can be upgraded to get bug fixes and security fixes, without introducing breaking changes to your build.
Your pinned dependencies should be fully specified in a file called
requirements.txt, and checked
into your version control system (VCS). For projects with only a small number of dependencies, maintaining
this manually (for example, installing with
pip install, then using
pip freeze) may be adequate.
For larger projects, GDS recommends the following approach, which was developed after various issues
were found in current mainstream tooling (see this commit message, and note also the
lack of support for specifying VCS dependencies
Put your top-level requirements in a
Use a “freeze” script (as seen in this Makefile) to generate the
requirements.txt, which is used whenever you need to pip-install the
- The script creates a temporary virtualenv, pip-installs the
requirements-app.txtand pip-freezes the result as
The Makefile freeze script we currently have is constructed such that your
requirements-app.txtfile can only contain pinned version numbers for your application’s immediate dependencies. This is not a particularly desirable feature: in principle we only demand that the final
requirements.txthas no ambiguity and ends up containing pinned versions for all dependencies.
List dependencies only needed for development or testing into a separate
This recommendation applies to any Python repository that intended to be installable (into a virtual environment, a container, or onto bare metal) as a dependency of some larger system or application. It may be applicable to repositories that provide scripts to be run by developers or other end-users, but is not recommended for code that’s intended to be deployed on its own into the cloud.
to specify the dependencies of your library, and the version ranges with which it
can be reasonably expected to work.
The range you choose will depend on the guarantees each dependency makes about backward-compatibility. For example, if you’re currently using version 1.3.1 of a semantically-versioned library, it would be reasonable to specify a range such as
<2.0,>=1.3.1. However, for a library that doesn’t make that guarantee, you might specify a more restricted range, such as
Update this file whenever you are ready to test and validate a new version that falls outside the existing range.
If you have dependencies that aren’t available on PyPI (for example, because you’ve fixed a bug by forking the code), then you can use a PEP 440 git reference in your
Updating this manual
This manual, and by extension the GDS Python Style Guide, is not presumed to be infallible or beyond dispute. If you think something is missing or if you’d like to see something changed then:
- (optional) Start with the #python Slack channel. See what other developers think and you’ll get an idea of how likely your proposal is of being accepted as a pull request even before you put in any work.
- Check out the making changes section of the GDS Way repo
- Create a pull request against GDS Way repo