This is mainly a note-to-self to remind me for my next upgrade – but hope it helps you too. TL;DR – install python-dev tools, export old installs into a requirements file, import into new python site-packages.

I upgraded the python on my laptop from 3.6 to 3.7 so I could use dataclasses for a little project. Then I got on a plane to Zurich and planned to get some work done on the 9 hour flight. Unfortunately I spent most of that time wrangling my python install – little cli tools I like to use like black, glances, sphinx, cookiecutter, etc – none of them worked!

When you do a python pip install of a library, it puts the library in a directory called site-packages under lib/python3.<your-version>/site-packages/<your-package-name>. If the library has defined command line entry-points, you will find it has also installed a file under bin/<your-package-name>. If bin is in your PATH, it means you can just type something like cookiecutter and it will call a function in the library to do stuff for you! Here’s what one of those files looks like:

➜  ~ bat ~/.local/bin/cookiecutter
───────┬───────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /home/matt/.local/bin/cookiecutter
───────┼───────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/usr/bin/python3
   2   │
   3   │ # -*- coding: utf-8 -*-
   4   │ import re
   5   │ import sys
   6   │
   7   │ from cookiecutter.__main__ import main
   8   │
   9   │ if __name__ == "__main__":
  10   │     sys.argv[0] = re.sub(r"(-script\.pyw?|\.exe)?$", "", sys.argv[0])
  11   │     sys.exit(main())
───────┴───────────────────────────────────────────────────────────────────────────────────────────────
➜  ~

It tells the shell to execute it using /usr/bin/python3 – that’s a link to the latest version of python3 – in this case it’s now python3.7, while the library was installed under python3.6. So the command line now fails! Python3.7 doesn’t know about libs installed for 3.6 at all.

On the flight I realized I could either copy from the python3.6 site-lib to the python3.7 one or just switch python3 to point back to python3.6 to make things work. But once you have internet access, here’s the magic to get it all working.

First, install the python3.7 dev stuff so you can compile anything that needs compilation.

➜  ~ sudo apt-get install python3.7-dev

Then let’s export everything we used to use under 3.6 and then reinstall it under 3.7.

➜  ~ python3.6 -m pip freeze > old_requirements.txt
➜  ~ python3.7 -m pip install --user -r old_requirements.txt