Skip to main content

The GDS Way and its content is intended for internal use by the GDS and CO CDIO communities.

Python style guide

This manual is designed to aid developers in writing Python code that is clear and consistent, within, and across, projects at GDS.

Code formatting

We follow PEP 8; where PEP 8 does not express a view (for example, on the usage of language features such as metaclasses) we defer to the Google Python style guide. Use these as references unless something is explicitly mentioned here. As always these rules should be followed only in conjunction with the advice on consistency on the main programming languages manual page.

If you want to add a new rule or exception please create a pull request against this repo.

Maximum line length of 120 characters

PEP 8 dictates a preferred maximum line length of <= 79. This is is a hangover from developing in a Unix terminal window. The vast majority of developers are now using an IDE which can handle a greater line length comfortably.

Couple this with the fact that much of the time GDS developers are coding web apps and have to deal with nested JSON objects, ORM model definitions/ queries, and error/ url strings and this convention begins to show its age.

Do not do

if not models.Address.query(
    models.Address.street_address_line_1
    == user['address']['street_address_line_1']
):
    pass


Do

if not models.Address.query(models.Address.street_address_line_1 == user['address']['street_address_line_1']):
    pass

Linting

Flake8

This manual advises the use of the Flake8 all in one lint, codestyle and complexity checker.

What is Flake8?

Flake8 is a command line utility that acts as a drop in replacement for the pep8/pycodestyle command line checker. It includes pep8/pycodestyle checks as well as adding complexity checks, and linting. If you’re already using the pep8/pycodestylechecker you’re most of the way there. If not then, never fear. You’ll find everything you need on this page.

How to use Flake8

Implementation of Flake8 will depend on whether the repository you want to run the checks on is a module or an application, and how your dependencies, automated testing, and continiuous integration are set up.

You’ll want to add the Flake8 module (available from PyPI) to your ‘dev’ or ‘test’ requirements/dependencies. You’ll then likely want to run it before you run your unit tests.

Example usage

For a quick example run the below code (assuming you have [virtualenv][virtualenv] installed)

mkdir flake8_test; echo 'import os' > flake8_test/flake8_test_file.py
python3 -m venv flake8_test/venv
source flake8_test/venv/bin/activate
pip install flake8
flake8 flake8_test/flake8_test_file.py
deactivate
rm -rf flake8_test

The last line of output should read:

> flake8_test/flake8_test_file.py:1:1: F401 'os' imported but unused

Pyflakes catches a linting error: F401 '<module>' imported but unused

This is because our file imports the os module but never uses it. Unused imports are considered a bad thing because they pollute the namespace, increasing the number of names a developer needs to keep track of.

Unused imports are a fairly simple example but errors like F811 redefinition of unused <name> from line <N> or F841 local variable <name> is assigned to but never used can indicate there are real problems with the code, and it may not be acting as expected.

For a full list of error codes check out:

Plugins

Flake8 has been designed to be extensible and has numerous plugins. They’re worth a look to see if any would be particularly beneficial to your code base.

Examples include checks for requiring copyright/licensing strings, requiring docstrings or warnings for upcoming deprecations.

A list can be found by performing a PyPI search.

Flake8 per file ignores

A particularly useful feature of Flake8 is the ability to specify rule exemptions on a per directory, per file, or per regex match basis.

Commonly it’s used for ignoring unused imports in module level __init__.py files or imports not being at the top of a file in settings files or scripts.

The feature is documented in the Flake8 options documentation, under per-file-ignores. You can also see an example in the Digital Marketplace API repo.

Common Configuration

Digital Marketplace is already running the latest version of Flake8 on all of its repos. You can find an example of their configuration in the root of any repo in the .flake8 file.

Commonly a configuration file will live in the root of the package. By default Flake8 will look for a .flake8 file in each directory.

[flake8]
# Rule definitions: https://flake8.pycqa.org/en/latest/user/error-codes.html
# D203: 1 blank line required before class docstring
# W503: line break before binary operator
exclude = venv*,__pycache__,node_modules,bower_components,migrations
ignore = D203,W503
max-complexity = 9
max-line-length = 120

In the above file we exclude directories we want the checker to ignore completely, ignore specific rules we disagree with, set the maximum line length and set the maximum complexity. We’ve also included comments detailing what the specific exclusions are.

Note: you can also ignore rules on particular lines of code or files by adding a # noqa comment - see flake8’s noqa syntax.

Linting resources

Dependencies

A Python application project typically brings together Python packages from PyPI, with others written in-house (or otherwise not distributed through PyPI).

These packages are the applications immediate dependencies. Additionally, any package can have its own immediate dependencies, where they draw on other packages. From an application’s point of view, the dependencies of the packages it requires are its sub-dependencies.

The below diagram shows a simplified view of the resulting pyramid of dependencies; however it is important to note that the hierarchy can repeat itself infinitely.

                               Application

                                    |
                              +-----+-----+
                              |     |     |

                          Immediate dependencies

                              |     |     |
                        +--+--+--+--+--+--+--+--+
                        |  |  |  |  |  |  |  |  |

                         ... Sub-dependencies ...

There are two ways of specifying dependencies in Python world: as a specific version or as an allowable range.

Different considerations apply to dependency management depending whether you are packaging a library, or creating an end-product such as an application or a bundle of scripts. In general, you should only specify specific versions when creating a Python system that sits at the top of the dependency pyramid; otherwise there is a danger of creating version conflicts.

Applications

These recommendations apply wherever you need a reproducible set of dependencies, such as a complete web application, perhaps with many dependencies and sub-dependencies. It also applies to a collection of scripts that are deployed into the cloud and run automatically (for example, batch jobs).

A good strategy for specifying your application’s Python dependencies has two desirable characteristics - they should be:

  1. Reproducible (predictable)

    Pin your application’s full dependencies – specific versions, rather than ranges – or you’ll get unpredictability between your dev environment and other environments. You want a new starter to avoid small hard-to-spot problems. And you want parity between what you test locally, what is tested by CI, and what you deploy, or you risk new issues appearing on a live server. Additionally these things can be hard to diagnose.

  2. Kept up-to-date

    Security issues are found in libraries, so it is important to choose libraries that are maintained and to ensure your team has a strategy to ensure security updates are installed without significant delay. The how to manage third party software dependencies section gives further context and discusses tools that can help, such as snyk.io.

Your README should document an easy-to-follow process by which all your Python dependencies can be upgraded to get bug fixes and security fixes, without introducing breaking changes to your build.

Your pinned dependencies should be fully specified in a file called requirements.txt, and checked into your version control system (VCS). For projects with only a small number of dependencies, maintaining this manually (for example, installing with pip install, then using pip freeze) may be adequate.

For larger projects, GDS recommends the following approach, which was developed after various issues were found in current mainstream tooling (see this commit message, and note also the lack of support for specifying VCS dependencies in pip-compile).

Put your top-level requirements in a requirements-app.txt file.

Use a “freeze” script (as seen in this Makefile) to generate the fully specified requirements.txt, which is used whenever you need to pip-install the dependencies.

  • The script creates a temporary virtualenv, pip-installs the requirements-app.txt and pip-freezes the result as requirements.txt.

The Makefile freeze script we currently have is constructed such that your requirements-app.txt file can only contain pinned version numbers for your application’s immediate dependencies. This is not a particularly desirable feature: in principle we only demand that the final requirements.txt has no ambiguity and ends up containing pinned versions for all dependencies.

List dependencies only needed for development or testing into a separate requirements-dev.txt file.

Libraries

This recommendation applies to any Python repository that intended to be installable (into a virtual environment, a container, or onto bare metal) as a dependency of some larger system or application. It may be applicable to repositories that provide scripts to be run by developers or other end-users, but is not recommended for code that’s intended to be deployed on its own into the cloud.

Use a setup.py to specify the dependencies of your library, and the version ranges with which it can be reasonably expected to work.

  • The range you choose will depend on the guarantees each dependency makes about backward-compatibility. For example, if you’re currently using version 1.3.1 of a semantically-versioned library, it would be reasonable to specify a range such as <2.0,>=1.3.1. However, for a library that does not make that guarantee, you might specify a more restricted range, such as <1.4,>=1.3.1.

  • Update this file whenever you are ready to test and validate a new version that falls outside the existing range.

If you have dependencies that are not available on PyPI (for example, because you’ve fixed a bug by forking the code), then you can use a PEP 440 git reference in your install_requires list.

  • This requires pip 18.1, and we have not tried it much at GDS.

  • In the past, we’ve documented such dependencies in a requirements.txt file (example), but any application wanting to depend on your library then needs to manually copy those sub-dependencies into its own list (example).

Specify dependencies needed only for testing your library in tox.ini if you are using Tox (example), or in a requirements-dev.txt (example) file otherwise.

This page was last reviewed on 13 July 2021. It needs to be reviewed again on 13 January 2022 by the page owner #gds-way .
This page was set to be reviewed before 13 January 2022 by the page owner #gds-way. This might mean the content is out of date.