Software libraries

3.  Personal installations 

We provide the /opt/meties/el8/ucontrib folder for personal installations.    
To build a test version from the source, follow the steps below (replace ProjectFolder, UserName,  PackageName, and ModuleName with actual names) :

• create a build folder in your project area
  mkdir -p /lstr/sahara/ProjectFolder/build
• download the package source into the /data1/ProjectFolder/build/PackageName
• follow the build instructions from the package source documentation
• use /opt/metis/el8/ucontrib/UserName/packageName as an installation target
  Example:
  cd /lstr/sahara/ProjectFolder/build/PackageName
./configure --prefix=/opt/metis/el8/ucontrib/UserName/packageName [build options]
make all -j8; make install
• create an environment module for the installed package in your home area
  mkdir -p /home/UserName/mymodulefiles; cd /home/UserName/mymodulefiles
• create and edit packageName/moduleName
touch packageName/moduleName
nano packageName/moduleName

• load packageName/moduleName in a shell or in a batch script to run the installed executables:
  module use /home/UserName/mymodulefiles;
module purge; module load packageName/moduleName

4. Python modules

We allow user-level installation of the python modules via pip3 and conda managers.

4.1  pip3 manager

• A python package from the pip3 predefined repository will be placed in the user's home directory ~/.local/lib/python3.6/site-packages under user ownership. All dependencies will be checked/resolved:
  python3 -m pip install -U pip --user
pip3 install pkgName  --user


• Same as above but using the exact address for a package:
  python3 -m pip install -U pip --user
pip3 install git+git://github.com/xxxx/pkgName.git

4.2  conda manager

Conda allows a more flexible way to manage personal Python packages, which is also more complex.
It is to be used as an extension of pip3  functions when special Python modules are required.
On Metis, conda is primarily required to install TensorFlow packages.

• Create a new conda environment (confirm the installation of proposed packages). 
  For the example below, all packages will be stored under   ~/.conda/envs/p312tf:
  conda create --name p312tf python=3.12


• List available conda environments:
  conda info --env

• Activate an environment (do each time to use/modify p39test installation):
  conda activate p312tf

• Install TensorFlow and dependent packages
  (p312tf) pip3 install --upgrade pip
(p312tf)  pip3 install tensorflow[and-cuda] pandas matplotlib scikit-learn

•  Run a custom code 
  (p312tf) python3 mycode.py


•  To use the GPU version of TensorFlow, load the CUDA module first
  (p312tf) module purge; module load cuda/cuda-12.2
(p312tf) python3 mycode.py


•  Deactivate 'py312tf'
  (p312tf) conda deactivate


To use the Conda environment in a PBS batch script, add after the PBS directives:

source /etc/profile.d/conda.sh
conda activate p312tf
#Load CUDA module if needed
module purge; module load cuda/cuda-12.2
python3 mycode.py
conda deactivate



4.3  Jupyter Notebook on Metis or Gaea

1. 1. Install  Jupyter Notebook
      
      • [@metis ~]$ pip3 install jupyter
      • [@metis ~]$ pip3 install urllib3==1.26.6
   2. Launch the notebook using a random port number of your choice in the [10000,50000] interval  (12345 in the example below)
      • [@metis ~]$ jupyter notebook --no-browser --port=12345 --ip=0.0.0.0

        metis ~]$ jupyter notebook --no-browser --port=12345 --ip=0.0.0.0 
        [I 23:26:55.768 NotebookApp] Serving notebooks from local directory: /home/username
        [I 23:26:55.768 NotebookApp] Jupyter Notebook 6.4.10 is running at:
        [I 23:26:55.768 NotebookApp] http://metis:12345/?token=f7586e3b06bddb86f130f762860dfc4af4ce70223a8b8e77
        [I 23:26:55.769 NotebookApp]  or http://127.0.0.1:12345/?token=f7586e3b06bddb86f130f762860dfc4af4ce70223a8b8e77
        [I 23:26:55.769 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
        [C 23:26:55.790 NotebookApp]
        ......
        

        At the beginning of the above output, look for the URL to copy/past in your local browser in step 4; in this example, "http://127.0.0.1:12345/?token=f7586e3b06bddb86f130f762860dfc4af4ce70223a8b8e77"
        
        
   3. Open a terminal on your computer and create the SSH tunnel for the chosen port to the Jupyter Notebook server (the example below is for the Linux or MacOS environment; MS Windows users can follow these instructions):
      
      [@myComputer ~]$ ssh -f username@metis.niu.edu -L 12345:metis.niu.edu:12345 -N
      
      
   4. Open a browser on your local computer and copy/paste the access URL
      
      
   5. Detailed tutorial (including SSH tunnel setup using PuTTY): https://thedatafrog.com/en/articles/remote-jupyter-notebooks/