If you run into issues with versioning and package management when working on multiple Python projects, don't worry, you're not alone. I will introduce two projects to hopefully help you tackle these issues: Pyenv and Pipenv. You might be thinking "why didn't he mention the Conda project?" Well, I chose not to include it simply because, in my experience, Conda and the pip resolving package dependency issues cause problems 90% of the time. My approach should work 100% of the time and allow you to fully control your Python environment. Let's dive in!
Pyenv
Pyenv is a project you can use to control Python’s version. More often than not, issues arise when moving to different versions of Python. For example, jumping from 3.7 to 3.9 introduces significant features that do not exist in older versions.
One criticism I have of Pyenv is that it's installed via a script, making it hard to install code in some corporate environments. I will assume your work environment is Linux or Mac. For windows please follow these directions.
Install
Installing Pyenv is typically done via running the following script:
curl https://pyenv.run | bash
Shell Setup
Pyenv requires you to run the following code in the shell prior to usage. This can be run via the .bashrc file at the initialization of the shell.
export PATH="$HOME/.pyenv/bin:$PATH"
eval "$(pyenv init -)"
eval "$(pyenv virtualenv-init -)"
Install Python
Next, we will run the install argument with a version, this will install on the system of Python version we requested.
pyenv install 3.9.2
pyenv versions
Activate
Once your environment is created, all that's needed is to activate the environment.
pyenv global 3.9.2
IDE integration
Your IDE of choice will be able to access this version of python using the following path:
~/.pyenv/versions/3.9.2/bin/python
Pipenv
Pipenv is a project that extends the pip Python package manager. It's a significant improvement in many areas, but specifically in dependency management. Once you learn how to use Pipenv, it's very easy to accomplish any task.
Activate the Python Environment
Before we do anything, we will activate the Python environment created in the first section. Pipenv should be able to find the Python interpreter without any configuration.
pyenv global 3.9.2
Install
We will use the pip Python package manager to install pipenv, and then we will stop using pip.
pip install pipenv
Create a Pipenv Environment
Now that pipenv is installed we are going to create an example project folder, and then initialize pipenv in that folder.
mkdir my_project
cd my_project
pipenv --python 3.9.2 install
pipenv lock
Pipfile & lock file
Pipenv uses two main files: the Pipefile specifies packages for your environment and the Pipe.lock file, generated from the Pipefile with:
pipenv lock
The lock file lists all dependencies used for the project, including sub dependencies. You should commit both files to your git repo.
The Pipfile is used to generate the lock file and contains configurations like repository and Python version.
[[source]]
#url = "your pypi url for packages"
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
pandas = "==0.24.2"
requests = "*"
[requires]
python_version = "3.9"
Syncing with the Lock File
Once you have a lock file, you can now sync your environment with what is defined in the lock file:
pipenv sync
If you want to manually install a package into your environment you can:
$ pipenv install requests~=1.2
This will update your Pipenv file.
Running Code
Now that our environments are set up, running code is trivial. All we need to do is:
pipenv run python3 my_python_app.py
Shell
If you want the terminal set to the environment, set it by using:
pipenv shell
Updating packages
If you want a list of outdated packages, run this to see all the packages needing an upgrade:
pipenv update --outdated
You can choose to blindly update all packages:
pipenv update
Or, you can update one package at a time:
pipenv update pandas
Requirements.txt
If you have a legacy requirements.txt file, Pipenv can use that specific file:
pipenv install -r requirements.txt
If you find that for some reason you need to generate a requirements.txt file, you can run:
pip freeze > requirements.txt
Now, you can manage both Python interpreter and your Python packages. It's important to control your environment for every project because it increases repeatability and reduces time wasted troubleshooting odd issues.
Let me know if you try these methods and how it affects working in your Python environment. If you want to talk specifics or learn more about setting up Python, feel free to send me a message. Our data engineers at SingleStone work with multiple distribution systems and are eager to hear about your experience.