Automating Convention: Linting and Formatting Python Code
Table of Contents
– The Zen of Python
I’ve found that most programmers favor a consistent code style and format. However, I’ve also found that consistently applying that style and format on every commit requires a lot of discipline. Instead of manually formatting code and applying code style, we can automate and have the computer apply our format and style.
Keeping our code style and format consistent makes our code base more readable and keeps code reviews focused on the substance of the implementation. Personally, using these tools reduces the time I spend reviewing code and allows me to forget about manually formatting code. Previously, I spent a lot of time manually formatting long function signatures and nested dictionaries, but adding tools like black to my git commit process saved me time.
Project Set Up
I’ll take you through an example project with configuration that I use in my production applications.
- Clone the
git clone https://github.com/laactech/pre-commit-config.git
- Create a virtual environment
python3 -m venv pcc_venv
- Activate the virtual environment
- Install the requirements
pip install -r requirements.txt
- Install the git hooks using pre-commit
Tools for Automation
pre-commit installs and manages the environments for the code linting and formatting tools:
- flake8 a linter to enforce coding style
- .flake8 stores the configuration
- bandit a linter to check for security vulnerabilities
- The bandit section of the .pre-commit-config.yaml stores the configuration
- black an automatic code formatter
- The tool.black section of the pyproject.toml stores the configuration
- isort an automatic import formatter
- .isort.cfg stores the configuration
- seed-isort-config a tool to statically populate the known_third_party part of the .isort.cfg
This collection of tools automates the formatting of our Python code and imports, style guide linting, and security linting. With pre-commit, all of these tools run on the code staged for every commit. This ensures the quality of the code being committed to our repository. Additionally for extra insurance, pre-commit can run our tools across the entire code base as part of a continuous integration process.
Now let’s run
pre-commit run -a on the example project to walk through how the linting and
formatting works and how we would remedy any problems.
Black outputs the first failure:
It tells us that it reformatted both of our Python files. If you look at the example.py file, you see the reformatted code.
When black fails on the git hook, black reformatted our code. The only action required from
us is to
git add the changed code.
Flake8 outputs the next failure:
Flake8 gives us the file name:line number:character number:error code for each style issue. Let’s fix our flake8 issues. First we need to remove the unused imports for pendulum and Dict in example.py and requests in another_example.py. We can safely delete those import lines. Next let’s remove the -u on line 41 to fix the undefined name issues. After that on line 49, let’s replace the ellipse with a f.read() to fix our local variable never used. Finally, remove the # fmt: off and # fmt: on around the custom_formatting list so that black can reformat it.
Bandit outputs one failure:
If you read the message output, bandit says that input is safe on Python 3. Due to this, let’s add a # nosec next to our output on line 16 to silence this issue.
Seed isort output:
The seed isort output tells us that the .isort.cfg changed. If we open the file, we see that the known_third_party line updated with our third party packages pendulum and requests. No action is required on our part aside from git adding the changes.
Finally, the isort output:
The isort output tells us which files changed. If we look at both our Python files, we see that our imports were sorted. Once again, no action is required from us besides git adding the changes.
Now we’re ready to commit the changes. As a reminder, we need to add the changes to the
.isort.cfg, example.py, and another_example.py. Let’s attempt to commit our changes
Black reformatted our custom_formatting list and helped fix our flake8 issues. Seed isort
removed pendulum from the known_third_party line in our .isort.cfg since we removed that
import from our code. Finally, let’s try another
git commit after adding the changes to
example.py and .isort.cfg.
Our code committed with a full pass of our linters and formatters. These examples contained a lot of changes, but generally in my experience, the number of actions required per commit is low. The failing hooks tend to be black and isort which require no action. Occasionally, flake8 will fail, but in my opinion, most flake8 failures are easy to solve, and it helps keeps your code clean.
Using the Tools in Your Project
Hopefully, the walk through convinced you that adding these tools to your Python project is easy. Now let’s go through how to add these to your new or existing Python project.
- Copy the following files to the root of your Python project’s git repository:
git addthe previous files to your git repository
pip install pre-commit
- Add pre-commit to your project’s requirements
git committhe new configuration files
pre-commit run -ato lint and format your entire project
git committhe formatting and linting changes once you’ve resolved any issues
Now on every commit,
pre-commit will use a git hook to run the tools.
Automating the linting and formatting of your code gives you consistent code without the manual time investment. I’ve used this configuration in numerous production applications and with multiple development teams with great success. I hope that you’ll find success with them as well.