Python virtual environments#

We describe how to create conda and venv environments. These environments have an independent set of packages.

See Python for our Python/Conda modules and JupyterHub for our JupyterHub instances.

Install or build packages inside an interactive job on the target cluster

In order to install or build packages with GPU support and hardware feature detection, you should get an interactive job on the target cluster (Alex, Fritz, Helma, TinyGPU, Woody and TinyFat) Make sure to load modules that might be needed by your Python code, e.g. CUDA/CUDNN for GPU support.

Not all compute nodes have direct internet access. Configure a proxy to enable access, either in the shell:

export http_proxy=http://proxy.nhr.fau.de:80
export https_proxy=http://proxy.nhr.fau.de:80

or as argument, e.g. for pip:

in the shell: pip install --proxy http://proxy.nhr.fau.de:80 <package-name>
inside Jupyter Notebook/JupyterHub: %pip install --proxy http://proxy.nhr.fau.de:80 <package-name>

If you are not inside a conda/venv environment add the --user flag to pip.

Conda environments#

First time only initialization#

The following steps only have to be performed once on our systems. They will initialize conda and cause conda to store packages and environments under $WORK instead of $HOME in order to save space in the latter.

Summary of all the following steps for easy copy and paste:

if [ ! -f ~/.bash_profile ]; then
  echo "if [ -f ~/.bashrc ]; then . ~/.bashrc; fi" > ~/.bash_profile
fi
module add python
conda config --add pkgs_dirs $WORK/software/private/conda/pkgs
conda config --add envs_dirs $WORK/software/private/conda/envs

Detailed explanation of the steps

If not already exists, create the file ~/.bash_profile (located in your $HOME) with the following content:
```
if [ -f ~/.bashrc ]; then . ~/.bashrc; fi
```
- This ensures the file ~/.bashrc, which will later be created/changed, to be loaded.
- See the bash documentation for details.

Ensure you have a Python module loaded:

module list
# output should contain a Python module:
# Currently Loaded Modulefiles:
# 1) python/3.9-conda

Store newly installed conda packages and conda environments under $WORK to save space in $HOME by executing:

conda config --add pkgs_dirs $WORK/software/private/conda/pkgs
conda config --add envs_dirs $WORK/software/private/conda/envs

Check the configuration is used, note that the variable $WORK will be expanded to the real path:

conda info
# output should contain:
#     package cache : /apps/python/...
#                     <real path of $WORK>/software/private/conda/pkgs
#  envs directories : /apps/python/...
#                     <real path of $WORK>/software/private/conda/envs

For more options see .condarc.

Conda environments can also be used for package management (and more). You can share conda environments with co-workers by having them add your environment path to their envs_dirs entry in ~/.condarc as well.

Creating conda environments#

You can create conda environments on our systems. Ensure you have initialized conda as described above. For GPU support of your packages you must use an interactive job.

Create a conda environment by executing:
```
conda create -n <env. name> python=<py. version>
```
- This creates a conda environment named <env. name>
- The new environment uses Python of the specified version <py. version>.
Activate the environment:
```
conda activate <env. name>
```
Your prompt should now be prefixed with the name of the environment like (<env. name>).

When you install now packages via conda or pip they will end up within the conda environment, therefore no --user option is needed.

An alternative way to build a conda environment is by using an environment.yml file. This method comes in very handy, if you intend to copy your environment from a local machine to our hpc systems. Or to copy between clusters.

Create a yml-file on your system with the original conda environment activated: conda env export --from-history --file environment.yml
Copy the environment.yml to the target system and create a new environment with: conda env create -f environment.yml

To activate a conda environment inside a batch job use:

module add python
conda activate <env. name>

Virtual environments with `venv`#

With venv you can create lightweight virtual environments. When using pip inside an activated venv environment packages are automatically installed there.

conda does not recognize venv environments

Installing packages via conda from inside an activated venv environment will fail. Conda does not work together with such an environment. For this you need a to create a conda environment.

Preparation: ensure a Python module is loaded.
Create a venv virtual environment in <path to env>:
```
python3 -m venv <path to env>
```
For example: python3 -m venv $WORK/venvs/testing
Activate the environment:
```
source <path to env>/bin/activate
```
For example: source $WORK/venvs/testing/bin/activate
The environment is activated when your prompt is prefixed with the directory name of the virtual environment like (<directory>). In case of our example your prompt would start with (testing).

Using environments in Jupyter Notebooks and JupyterHub#

Python environments can be made available in Jupyter Notebooks and JupyterHub by installing the ipykernel package.

conda environment:
1. Install the ipykernel package
```
conda install -n <env. name> ipykernel
```
  where <env. name> denotes the conda environment you want to use.
2. Only required for Jupyter Notebooks: register environment
```
conda run -n <env. name> python3 -m ipykernel install --user --name <env. name>
```
  An existing kernel with the same name as <env. name> will be overridden.
Python venv environment:
1. Activate the venv environment, here named <env. name>.
2. Install ipykernel package and register it
```
pip install ipykernel
python3 -m ipykernel install --user --name=<env. name>
```
  An existing kernel with the same name as <env. name> will be overridden.

Compressing Python environments at rest#

All of our filesystems have inode quotas, so the number of files is limited. Python environments contain a lot of rather small files and are therefore not suited well for the available filesystems.

Compressing Conda environments#

After environment creation, compress it to a known location

conda info -e | grep -E "^<env. name>"
<env. name>       <env. path>
[...]
tar -cjf /path/for/resting/environments/<env. name>.tar.gz <env. path>

Usage on the HPC systems. At the beginning of the job, unpack it to a suitable location like $TMPDIR

tar -C $TMPDIR -xf /path/for/resting/environments/<env. name>.tar.gz
module add python
conda config --add envs_dirs $TMPDIR
conda activate <env. name>

Please check out Conda page for more info about Conda.

Compressing `venv` environments#

After environment creation, compress it to a known location

python3 -m venv <path to env>
tar -cjf /path/for/resting/environments/<env. name>.tar.gz <path to env>

Usage on the HPC systems. At the beginning of the job, unpack it to a suitable location like $TMPDIR

tar -C $TMPDIR -xf /path/for/resting/environments/<env. name>.tar.gz
module add python
source $TMPDIR/<env. name>/bin/activate

Using an Apptainer container#

Another option is to create a container with contains your environment. It is possible to install whatever you need using package managers inside the container. The codeforge/miniforge3 images use Ubuntu, thus apt-get.

Bootstrap: docker
From: dockerhub-mirror3.rrze.uni-erlangen.de/condaforge/miniforge3:latest

%files
    requirements.txt /

%environment
    export PATH=/opt/conda/envs/myenv/bin:$PATH
    export CONDA_DEFAULT_ENV=myenv

%post
    apt-get update -y
    apt-get install -y <packages>
    apt-get clean

    # Create a new conda environment named 'myenv' with Python installed
    conda create -y -n myenv python=3.11

    # Activate the conda environment
    . /opt/conda/etc/profile.d/conda.sh
    conda activate myenv

    # Install pip dependencies inside the conda environment
    pip install --upgrade pip
    pip install -r /requirements.txt

    # Optional: clean up
    conda clean --all -y

%runscript
    exec "$@"

Please check out Apptainer page for more info about apptainer.