Software libraries

We use the 'module' engine to access available software packages.  

1.  Locally installed packages

To find and setup an environment for a particular software , do
  • to list available modules:
    module avail
    module keyword keyword
  • to load a module
    module load moduleName
  • to find a description of the moduleName module
    module help moduleName
  • to purge loaded modules (restore the defaul environment)
    module purge

To install a tagged version of specific package that is not listed please submit HelpDesk ticket.

3.  Personal installations 

We provide the /opt/meties/el8/ucontrib folder for personal installations.    
To build a test version from the source, follow the steps below (replace ProjectFolder, UserName,  PackageName, and ModuleName with actual names) :

  • create a build folder in your project area
    mkdir -p /lstr/sahara/ProjectFolder/build
  • download the package source into the /data1/ProjectFolder/build/PackageName
  • follow the build instructions from the package source documentation
  • use /opt/metis/el8/ucontrib/UserName/packageName as an installation target
    Example:
    cd /lstr/sahara/ProjectFolder/build/PackageName
    ./configure --prefix=/opt/gaea/ucontrib/UserName/packageName [build options]
    make all -j8; make install
  • create an environment module for the installed package in your home area
    mkdir -p /home/UserName/mymodulefiles; cd /home/UserName/mymodulefiles
    touch packageName/moduleName
  • edit packageName/moduleName
  • load packageName/moduleName in a shell or in a batch script to run the installed executables:
    module use /home/UserName/mymodulefiles;
    module purge; module load packageName/moduleName

4. Python modules

We allow user-level installation of the python modules via pip3 and conda managers.

4.1  pip3 manager

  • A python package from the pip3 predefined repository will be placed in the user's home directory ~/.local/lib/python3.6/site-packages under user ownership. All dependencies will be checked/resolved:
    python3 -m pip install -U pip --user
    pip3 install pkgName  --user

  • Same as above but using the exact address for a package:
    python3 -m pip install -U pip --user
    pip3 install git+git://github.com/xxxx/pkgName.git

4.2  conda manager

Conda allows a more flexible way to manage personal python packages, which is also more complex.
To be used as an extension of pip3  functions when special python modules are required.
On Gaea, conda is primarily required to install TensorFlow packages.

  • Create a new conda environment (confirm the installation of proposed packages). 
    For the example below, all packages will be stored under   ~/.conda/envs/p39test:
    conda create --name p39test python=3.9

  • List available conda environments:
    conda info --env
  • Activate an environment (do each time to use/modify p39test installation):
    conda activate p39test

  • Install conda-specific packages (torch-tensorflow in the example below)
    (p39test) conda install pytorch tensorflow
  • Install additional packages if needed (one can use pip3, which is local to the p39test
    environment in the example below and will not interfere with the system pip3)

    (p39test) pip3 install --upgrade pip
    (p39test) pip3 install matplotlib
  •  Run a custom code 
    (p39test) python3 mycode.py

  •  Deactivate 'py39test'
    (p39test) conda deactivate

4.3  Jupyter Notebook on Metis or Gaea

  • Install  Jupyter Notebook
    • [@metis ~]$ pip3 install jupyter
    • [@metis ~]$ pip3 install urllib3==1.26.6
  • Launch the notebook using a random port number of your choice in the [10000,50000] interval  (12345 in the example below)
    • [@metis ~]$ jupyter notebook --no-browser --port=12345 --ip=0.0.0.0
      metis ~]$ jupyter notebook --no-browser --port=12345 --ip=0.0.0.0 
      [I 23:26:55.768 NotebookApp] Serving notebooks from local directory: /home/username
      [I 23:26:55.768 NotebookApp] Jupyter Notebook 6.4.10 is running at:
      [I 23:26:55.768 NotebookApp] http://metis:12345/?token=f7586e3b06bddb86f130f762860dfc4af4ce70223a8b8e77
      [I 23:26:55.769 NotebookApp]  or http://127.0.0.1:12345/?token=f7586e3b06bddb86f130f762860dfc4af4ce70223a8b8e77
      [I 23:26:55.769 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
      [C 23:26:55.790 NotebookApp]
      ......
      
  • Open a terminal on your computer and create the ssh tunnel for the chosen port to the Jupyter Notebook server (example below is for the Linux or MacOS environment, MS Windows users can follow this instructions):
    • [@myComputer ~]$ ssh -f username@metis.niu.edu -L 12345:metis.niu.edu:12345 -N
  • Open a browser on your local computer and copy/paste the access URL
  • Detailed tutorial (including ssh tunnel setup using PuTTY): https://thedatafrog.com/en/articles/remote-jupyter-notebooks/

 

 

Prospective user?

Request an account.

Back to top