# Extending containers with virtual environments for faster testing

This is a short example of how to extend the containers built via `cotainr` via virtual environments. This approach can be useful for developing and testing as it doesn't require rebuilding a container from scratch every time a new package is added.

## Requirements

We assume you have built a container from a `conda` environment file via something like:
```bash
module load LUMI/25.03 cotainr

BASE_CONTAINER=/appl/local/laifs/containers/lumi-multitorch-u24r70f21m50t210-20260513_121430/lumi-multitorch-mpich-u24r70f21m50t210-20260513_121430.sif
   
cotainr build minimal_pytorch.sif --base-image=$BASE_CONTAINER --conda-env=minimal_pytorch.yml --accept-license
```

## Set up a virtual environment

First, we run a shell inside the container
```bash
singularity shell --bind /pfs,/scratch,/projappl,/project,/flash,/appl minimal_pytorch.sif
```
Note that setting `--bind` is optional, you achieve the same by
```bash
module use /appl/local/laifs/modules
module load lumi-aif-singularity-bindings
singularity shell minimal_pytorch.sif
```

In order to install additional packages, we create a virtual environment via `venv` and activate it inside the container
```bash
python -m venv myenv --system-site-packages
source myenv/bin/activate
```
The `--system-site-packages` flag gives the virtual environment access to the packages from the container.

## Install custom packages

After activating the virtual environment, we can now install custom packages via pip, for example:
```bash
pip install torchmetrics
```

## Run container with `venv` packages
If we want to run the container with the freshly installed packages in a batch script, we need to first source the `venv` before executing the python script:
```bash
singularity exec minimal_pytorch.sif bash -c "source myenv/bin/activate && python my_script.py"
```

> [!WARNING]
> You should not stop here, as this way of installing python packages creates typically thousands of small files. This puts a lot of strain on the Lustre file system and might exceed your file quota. Choose one of the following options next:


## Option 1: Create a new container with `cotainr`
After having found all packages needed for our project, we should create a new container with an updated `conda` environment file. The virtual environment should then be deleted
```bash
cotainr build updated_pytorch.sif --base-image=$BASE_CONTAINER --conda-env=updated_pytorch.yml --accept-license
rm -rf myenv
```


## Option 2: Turn `myenv` into a SquashFS file
Alternatively, we can turn the `myenv` directory into a SquashFS file and bind mount it to the container.

Create a SquashFS file:
```bash
mksquashfs myenv myenv.sqsh
```
Or better, use the following options:
```bash
mksquashfs myenv myenv.sqsh -processors 1 -all-root -action "chmod(a+rX) @true"
```

Where the additional options mean the following:
- `-processors 1`: use 1 core to the keep load on the login node low. Use a value between 1 and 16.
- `-all-root`: reset user and group IDs of all files to root. This is optional and useful to anonymize the container.
- `-action "chmod(a+rX) @true"`: reset all file and directory permissions for everybody to read and execute where needed.

Then remove the myenv directory and bind mount the SquashFS file to the container:
```bash
rm -rf myenv # the myenv directory can be deleted
export SINGULARITYENV_PREPEND_PATH=/user-software/bin # gives access to packages inside the container
singularity exec -B myenv.sqsh:/user-software:image-src=/ minimal_pytorch.sif python my_script.py
```
