Python

When using Python the recommended way to manage Python versions and pip package versions is using virtual environments. These allow you to create isolated Python environments with separate versions of pip packages and Python installs that can be easily switched between depending on the needs of the particular project you are working on.

Guide to configure your own virtual environment

Introduction

Virtual Environments and dependency management are an integral part of software development workflow. This tool will allow users to create their own “sandbox” environments where they can use different versions of python than are provided as default on the host system, and will allow them to install and specify particular versions of python libraries to create reproducible programs that can be shared with others.

This document is intended to show a user how to set up and configure a python virtual environment using conda. Python has a very diverse ecosystem of popular methods for accomplishing similar tasks, and it is up to individual users to find what works best for them. Some alternatives are pip, venv, and pipenv.

Step 1: Getting Ready

The following instructions for setting up a conda environment require running commands in the linux terminal.

To launch a terminal after logging in:

In XFCE: In the application bar at the bottom of the screen, click the terminal emulator application to launch the terminal (it looks like a black square with some text)
In GNOME: Click on the the Show Applications button on the main screen (it looks like a grid of dots), search for “terminal” and select the Terminal application.

Most linux labs have Ubuntu installed which runs GNOME by default. If using VOLE, there is an option to choose between XFCE or GNOME, so it is up to the user’s discretion.

Step 2: Creating your own Conda Project

NOTE: Some modification of the below commands will be required. Parts that the user will need to fill in are enclosed in either square brackets, [ and ], or angle brackets, < and >.

Create an environment. The /xtra storage space usually has more space available than the user’s home directory, and so new environments should typically be created in that location. The following command

conda create --prefix /xtra/<internet_id>/<env_name> python=<python_version> [optional_package_names[=optional_version]]...

E.g., the following command will install a conda environment using python 3.11 at /xtra/baile320/test_out_conda and will install the pandas and matplotlib library without any version restrictions, and will install the cryptography library version 41.0.4:

conda create --prefix /xtra/baile320/test_out_conda python=3.11 pandas matplotlib cryptography=41.0.4

NOTE : it is best practice to declare a specific version of every library you want to use. This will help you avoid problems in the future if someone (including you) wants to create the environment from scratch after new versions of the libraries have come out (which possibly contain unknown breaking changes)
A prompt will ask you if you want to proceed. Press ‘y’ for yes and hit enter, and it will attempt to download the appropriate libraries & dependencies and create your conda environment.
Once the environment has been created successfully, you can activate your conda environment using the command:

conda activate /xtra/<internet_id>/<env_name>

When you are in your conda environment, your shell prompt will show you your active environment, like so:

(<current_env>) $

From here, you can run python scripts as you normally would, but the scripts would be run using the version of python and the libraries that have been specified throughout the set-up process.
After creating an environment, you can install additional packages into your environment whenever you want, as long as your conda environment is active, like so:

conda install <package1> <package_2> <etc>

E.g.
conda install requests scrapy beautifulsoup4=4.12.2
To deactivate an environment, simply run:

conda deactivate

Which will put you back into your normal terminal environment.

Sharing your Conda Project

In order to share your environment with someone else, you can export all of the python dependencies into a requirements.txt file, which another person can use to make a conda environment with those dependencies.

With the appropriate conda environment activated, the following command will output the requirements.txt file to your current directory:

conda list -e > requirements.txt

To share your environment and code with someone else, you would probably want to use a version control & source code repository system like Github (the university provides a private github instance for students & employees).

Your repository should contain the source code and requirements.txt, but it should not contain all of the other files and directories which conda uses.

Using Someone Else’s Conda Project

If someone has shared a requirements.txt file with you, you can install it with a few simple conda commands. Again, you will probably want to use a path that is in the /xtra directory due to the larger space than /home directories have:

Create a new conda environment (ask your instructor or see the source code documentation to see if there is a particular requirement for the python version):

conda create --prefix /xtra/<internet_id>/<env_name> python=<python_version>
Install the packages from requirements.txt:

conda install -p /xtra/<internet_id>/<env_name> --file requirements.txt

(this assumes requirements.txt is in the current directory, if it is somewhere else, you will need to entire the full path to the file)

You would then activate the virtual environment by running

conda activate /xtra/<internet_id>/<env_name>

Other Helpful Information

The official conda user guide is very thorough. There are also lots of guides and walkthroughs online if you are trying to do something specific that is not covered in this guide. Often a simple google search can give you exactly what you need.

Autoloading your environment

If you do not want to manually activate your environment every time you need to use it, you can add the activation line to your .bashrc file using your favorite text editor (VS Code, vi, vim, nano, etc).

At the end of the .bashrc file, add the line

conda activate /xtra/<internet_id>/<env_name>

For the environment you want to activate.

PLEASE NOTE: This will activate every time you log into linux until you remove it from the .bashrc file. That means your “default shell” will be your conda environment. You can still deactivate it using conda deactivate. If you have multiple courses or different environments per homework or per project, this is probably not a wise change to make, as it could easily lead to confusion.

List Your Environments

conda env list

Remove an Environment

conda remove --prefix /xtra/<internet_id>/<env_name> --all

Recommended Installation Locations:

If you are creating Python Virtual Environments using a CSE Linux machine there are a few different directoires where you can install them. Below are the available options and some suggested guidelines for when to use each install location.

Home Directory (/home/[username])
- You do not anticipate your Virtual Environment to be very large
- It does not need to be accessible to anyone else
Xtra Directory (/xtra/[username])
- The Virtual Environment is too large for your home directory
- It does not need to be accessible to anyone else
Group Project Directory (/project/[group])
- The Virtual Environment is too large for your home or xtra directories
- /project/[group] is the working directory for your Python project
- It needs to be accessible to other members of your group
Export Scratch Directory (/export/scratch)
- Your group has a dedicated machine with local storage
- You do not need your Virtual Environment to be backed up
- You have no reason to want access to the Virtual Environment outside of the local storage
- You are using a User Managed host that does not have access to Unified CSE directories (/project, /home, /xtra)
- You do not need it to be accessible from any other machine