Conda environment and local installation#

Useful links:

Conda environments#

Managing conda environment in Python is a very useful skill that allows you to have multiple working spaces in the same machine, from where you can run different libraries or even different version of Python. It is usually recommendable to have a base environment with a default version of Python and standard libraries, such as numpy, pandas and matplotlib, and then create new virtual environmnets when working is specific projects.

In the JupyterHub, we have already setted up a series of conda environments that you can use during the course. You can see all the availables environments by oppening a new terminal and entering conda env list. In order to activate a new environment (for example, in case you want to add a package), you can use conda activate <environment name>. You will find these environments listed:

  • base and notebook: These are default environmnts you can use to explore.

  • Finse_MB_oggm: Virtual environment use for Project No1

  • Finse_CNN: Virtual environment use for Project No2 When opening a Jupyter notebook, be sure the kernel is seted up with the right virtual envrionment. In order to do so, you can go to the upper right corner of the notebook, select the kernel option and click the option with the desired name of the conda virtual environment.

Local installation: JupyterLab in your machine#

The goal of these notes is to help you setup your local machine with the JupyterLab we are using for the workshop. Even if you already have conda installed in your local machine, it’s not a bad idea to reset the conda install. The idea is to have a very simplistic base environment from which you can launch JupyterLab and connect it to the kernels associated to other environmnets you create (eg, the ones used during the projects).

Step 1: Installation and configuration of miniconda#

Instead of installing conda with the full package management system and many Python libraries, we are going to install Miniconda, a minimalist version of conda that included the package management tool and a small number of useful packages. If you already have conda installed in your machine, you may not need to do this. An easy way to check if conda is installed in your computer is by entering conda --help in the terminal. You are also welcome to install conda from zero by removing all conda directories and installing miniconda.

You can install the last version of miniconda depending on you operating system from the Miniconda website. For unix machines (Linux, MacOS), a more simple way of doing this is by using the wget command followed by the installer link (silent installation mode). For example, for a MacOSX machine (without M1 chip) you can use

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh -O ~/miniconda.sh
mkdir -p $HOME/local/conda
bash ~/miniconda.sh -b -p $HOME/local/conda

and for a Linux machine

wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
mkdir -p $HOME/local/conda
bash ~/miniconda.sh -b -p $HOME/local/conda

If you don’t have wget installed, you can do it using brew with brew install wget; or download the .sh file directly from the miniconda installation website and replace ~/miniconda.sh by the name of the downloaded file (don’t change the name of the file and be sure that the direction to the file is correct, either by adding the full path to the file or by moving the sh file to your home directory).

Note for Windows users

If you are working from a Windows machine, you can directly install Miniconda using the link provided in the website. Most cloud-hosted computing is based on a Unix foundation, which both MacOS and Linux provide out of the box, so some of our instructions fit that workflow a little more easily. But for Windows platforms, Microsoft now officially supports something very similar, which is excellent and we highly encourage you to play with and learn about: the Windows Subsystem for Linux (WSL). It provides top-notch Unix functionality next to your regular Windows tools and workflow, and is probably the ideal setup for modern scientific and research computing on Windows. Discussing it in detail is beyond the scope of this course, but we highly encourage you to explore this tool on your own (and let us know how it goes!).

In order to specify the path where conda is installed, you need to add the following line to your .bashrc file

# add path to conda
export PATH="$HOME/local/conda/bin:$PATH"

and execute it by source .bashrc. After doing this, you will see that now conda is recognized. You can fix this change by now running conda init, so you don’t need to worry again about PATH. You can see where conda has been installed by typing which conda in your terminal.

You can change the configuration of conda inside .condarc manually or directly though the shell by using the following commands. In this case, we specify the channel used by conda/mamba to download the libraries.

conda config --add channels conda-forge
conda config --set channel_priority strict

Once you have installed conda, you can use it to install mamba:

conda install --yes mamba

If this doesn’t work, you can also try conda install mamba -n base -c conda-forge. If this still doesn’t work, you can always replace mamba by conda in the following section, but it will be much slower and unefficient.

Step 2: Setting the Hub environment in local#

Once you have installed miniconda, you can create virtual environments from a yml file just as we did in the Hub. As a first step, we are going to create a virtual environment with the same configuration we have in the Hub. The configuration file environment.yml for the Hub is available at the site repository for the course. Before using environment.yml to create a new environment, take a look to the text file and try to identify parts of the file that are not necessary or irrelevant to create the Hub environment; you will have to remove the last lines of the file that include the pip instructions for the requirements of the JupyterBook we use for the website, which are not necessary for the creation of the virtual environment.

Then, you can create a the virtual environment with

mamba env create -f environment.yml

You can always use conda env create -f <yml file>, but mamba is much faster than conda. It will take a few minutes until all the dependencies get installed in your machine. It could be the case that one specific library produces conflict in certain machines. If conda prints a error message associated to a specific library included in the environment file, try to comment such line in environment.yml and try again.

Now you can activate the environment from the terminal using conda activate <environment name>. From there, you can lunch the classical notebook by typing jupyter notebook but you also have installed the JupyterLab, which you can lunch with jupyter lab.

In order to reproduce the same environment we have in the Hub, we are going to install JupyterLab in our base environment and then create a Python kernel with the environment of the Hub. In your base environment (conda activate base), be sure to have installed both JupyterLab and ipykernel. If not, you can install then using conda:

mamba install -c conda-forge jupyterlab
mamba install ipykernel

From here, the story is pretty much the same of what we did in the Hub to create a kernel with a new virtual environment (see Using an environmnet in your notebooks). Moving back to the new environment we just created (conda activate <environment name>) we run

python -m ipykernel install --user --name <environment name> --display-name "ML+Glaciology"

Now, if we come back to base and launch Jupyter Lab with jupyter lab, we should see the option creating a new notebook with the ML+Glaciology kernel.