Grabbing The Snake By The Tail

Python is one of those pervasive pervasive languages used all over the place. I love the Python community's enhancement proposal process, its heavier investment in type safety in recent years (e.g. this), and the simplicity of writing web services in Python. There are also really rock solid libaries in Python that are indispensable in the machine learning world (although I hope Rust becomes even more prominent in that space)!

The only part of the Python ecosystem that has felt toothy and cumbersome for over a decade is the tooling around developing with it. The advent of build frontends led to nicer tooling like poetry, which I use almost for almost every Python project I work on. Still, moving between projects feels tedious. There are a lot of reasons for this, but here are some of the stickiest for me:

newer standards are inconsistently supported, so relying on your pyproject.toml to store your library config is not a uniform solution -- this leads to 'ini' files or 'cfg' files laying around with duplicated information or copied between projects
language server support in Python is full of many contenders that partially solve developer's needs, so you often need to cobble together many tools to build a rigorous development workflow: mypy, isort, black, ruff, pyright, et cetera
editor support for switching virtual environments is hit or miss, especially in the open source community (Jetbrains Pycharm does really well here)

Coding Python In Neovim

The above issues make the ideal Neovim setup for developing in Python a little challenging to define. Initially I solely relied on pyright as my language server, with additional verifications downstream in my pre-commit setup. pre-commit was set to format based on black, with flake8 to enforce lint rules, isort to keep python module imports sorted and pydocstyle to ensure code was thoroughly documented.^1 All of these things prevented commits in source control from dragging down code quality -- and it was good.. until the rules became a nag at commit time.

The solution was to use the same tooling in dev that pre-commit used. Neovim is an amazing editor, with a vibrant plugin community, but setting this up took some time.

More LSPs, more problems

For example, pyright does a great as a language server providing autocompletion, go to definition, and basic linting. My team prefers mypy for type checking though. mypy is a static type checker tool, not a language server.

null-ls provides a shim to glue together tools to the language server protocol, helping combine them with language servers and other plugins in the Neovim editor. We can add mypy to null-ls and have it pop open alongside pyright when we open Python code.

While it was easy to add mypy support, now my editor had two errors for each typing problem. I also noticed that depending on my active virtual environment, mypy might miss typing defined in my project.

It turned out that an easy 'fix' was to source the virtual environment for my project before I opened the code with Neovim. This fixed many issues with the python path... but it caused other problems, like breaking access to tools that were installed on the system python that pyenv pointed to on my host. Then I stumbled upon the prefer_local key in null-ls (check out that thread, lots of great dialogue and learning).

Digging into my issues, I concluded that pyright is powerful but opinionated and did not provide the flexibility I wanted. I ended up switching to pylsp, which provides similar functionality and is composable with many other Python tools. It looked something like this in my lua config for Neovim:

lspconfig.pylsp.setup({
 settings = {
   pylsp = {
     plugins = {
       black = {
         enabled = true,
         line_length = 120
       },
       pycodestyle = {
         enabled = true,
         ignore = {},
         maxLineLength = 100
       },
       pyflakes = {
        enabled = false
       },
       flake8 = {
         enabled = true,
         select = "B,C,E,F,W,T4,B9",
         ignore = "E266,E501,E402",
       },
       mypy = {
         enabled = true,
         ignore_missing_imports = true
       }
     }
   }
 }
})

The issue is that pylsp is installed into my system Python environment, not the virtualenv that each of my projects use. This leads to another version of the problem from before, where mypy cannot 'see' types that my project depends on, for example. I guess I could install that lsp as a dev dependency into each project I work on, but that feels heavy-handed and wrong.

The solution I came up with is:

use pylsp instead of pyright with only core lsp features enabled
use null-ls for the additional tools I want, using prefer_local and sourcing each project's virtualenv before entering the project as a fallback ^2
consolidate tooling as much as possible to avoid performance problems and keep my editor config as simple as I can

Game-changing tools

As I got deeper into the tooling, I found some really promising projects to keep an eye on. The coolest one is called ruff. It already has most of the features I want (including smart caching and pyproject.toml support). What is most exciting is the stated aim of being a highly performant all-in-one tool to unite approaches from so many disparate but wonderful existing tools.

The Javascript community is churning out highly performant tools in languages like Rust to support their developer platforms (e.g. swc). Python appears to be going through a similar process, and ruff looks like it is leading the way!

One of the nicest things about ruff will be taking the half-dozen Python tools and replacing them with a single tool as my language server for Python development. While it cant yet today replace things like black, the work is underway. My expectations are high and I have a feeling this project will not disappoint. Keep an eye on it!

PS

1 - It turns out that pre-commit also uses its own virtualenv, with its creator saying that is done to guarantee isolation of tooling environments. Of course, pre-commit has its own config resolution so there is more duplication per project than I would like for linter config, etc.

2 - There are many approaches to solving this problem. One of the more interesting ones on my todo list would use a Nix flake to build each project with Poetry and bundle in nvim. Additionally, many of the neovim conigurations I have seen online use lua helper functions to resolve the correct Python environment per their needs, relying on the BufWritePre autocmd.

3 - My Neovim config is always a work in progress. You can find a recent copy of it here

posts · about · home