Python

When using Python the recommended way to manage Python versions and pip package versions is using virtual environments. These allow you to create isolated Python environments with separate versions of pip packages and Python installs that can be easily switched between depending on the needs of the particular project you are working on.

Guide to configure your own virtual environment

Introduction

Virtual Environments and dependency management are an integral part of software development workflow. This tool will allow users to create their own “sandbox” environments where they can use different versions of python than are provided as default on the host system, and will allow them to install and specify particular versions of python libraries to create reproducible programs that can be shared with others.

This document is intended to show a user how to set up and configure a python virtual environment using pipenv. Python has a very diverse ecosystem of popular methods for accomplishing similar tasks, and it is up to individual users to find what works best for them. Some alternatives are pip, pipx, and venv.

Step 1: Getting Ready

The following instructions for setting up a pipenv environment require running commands in the linux terminal.

To launch a terminal after logging in:

  • In XFCE: In the application bar at the bottom of the screen, click the terminal emulator application to launch the terminal (it looks like a black square with some text)
  • In GNOME: Click on the the Show Applications button on the main screen (it looks like a grid of dots), search for “terminal” and select the Terminal application.

Most linux labs have Ubuntu installed which runs GNOME by default. If using VOLE, there is an option to choose between XFCE or GNOME, so it is up to the user’s discretion.

Step 2: Creating your own Pipenv Project

NOTE: Some modification of the below commands will be required. Parts that you will need to fill in are enclosed in either square brackets, [ and ], or angle brackets, < and >.

  1. Create an environment. The /xtra storage space usually has more space available than the your home directory, and so new environments should typically be created in that location. Create the folder where the virtual environment should be located with the following command. The <env_name> is arbitrary, it is suggested to use a name associated with the project or class you are working on. 

    mkdir /xtra/$USER/<env_name>
  2. Once the directory has been created successfully, you will need to change your current directorty to the desired pipenv location:

    cd /xtra/$USER/<env_name>
  3. Once you are in the desired directory, you can install packages with pipenv like so:

    pipenv install <package1> [optional_package2[=optional version]] [etc]

    E.g.:

    pipenv install requests scrapy beautifulsoup4=4.12.2

    NOTE : it is best practice to declare a specific version of every library you want to use. This will help you avoid problems in the future if someone (including you) wants to create the environment from scratch after new versions of the libraries have come out (which possibly contain unknown breaking changes)
  4. Once the desired package(s) are installed, you can run python scripts using the pipenv run command like so:

    pipenv run python3 <python_script.py>
  5. After creating an environment, you can continue to install addtional packages into your environment whenever you want using the same pipenv install command, as long as you are running those commands in the location of your pipenv.

Sharing your Conda Project

In order to share your environment with someone else who is also using pipenv you can give them a copy of your pipfile and pipfile.lock. These act like a requirements.txt file in other Python Virtual Environment solutions, noting which version of dependencies are needed for the project. When installing packages with pipenv they are automatically created and updated in the project directory.  

If you are sharing the project with someone using a different Python Virtual Environment tool that uses a requirements.txt file you can export all of the python dependencies into a requirements.txt file using pipenv like so:

pipenv requirements > requirements.txt

To share your environment and code with someone else, using a version control & source code repository system like Github is recommended. Your repository should contain the source code, pipfile, pipfile.lock and a requirements.txt if you anticipate the project being shared with others who are using tools other than pipenv.

Using Someone Else’s pipenv Project or from requirements.txt

If someone has shared a their pipfile and pipfile.lock files or requirements.txt with you, you can install it with a simple pipenv commands. Again, you will probably want to use a path that is in the /xtra directory due to the larger space than /home directories have:

  1. Create a new directory for your pipenv project:

    mkdir /xtra/$USER/<env_name>
  2. If using pipfile and pipfile.lock files, copy them to this directory
  3. Change your current directory to the one you just created for the project:

    cd /xtra/$USER/<env_name>
  4. To install the packages from the pipfile:

    pipenv install
  5. To install the packages from a requirements.txt file:

    pipenv install -r path/to/requirements.txt

Other Helpful Information

The official pipenv guide is very thorough. There are also lots of guides and walkthroughs online if you are trying to do something specific that is not covered in this guide. Often a simple google search can give you exactly what you need.

Recommended Installation Locations:

If you are creating Python Virtual Environments using a CSE Linux machine there are a few different directories where you can install them. Below are the available options and some suggested guidelines for when to use each install location.

  • Xtra Directory (/xtra/[username])
    • The Virtual Environment is too large for your home directory
    • It does not need to be accessible to anyone else
  • Home Directory (/home/[username])
    • You anticipate your Virtual Environment to be very small
    • It does not need to be accessible to anyone else
  • Group Project Directory (/project/[group])
    • The Virtual Environment is too large for your home or xtra directories
    • /project/[group] is the working directory for your Python project
    • It needs to be accessible to other members of your group
  • Export Scratch Directory (/export/scratch)
    • Your group has a dedicated machine with local storage
    • You do not need your Virtual Environment to be backed up
    • You have no reason to want access to the Virtual Environment outside of the local storage
    • You are using a User Managed host that does not have access to Unified CSE directories (/project, /home, /xtra)
    • You do not need it to be accessible from any other machine