After you have loaded the necessary modules for the type of programs you going to run on Gaea, you are ready to build your and run first application.
We have created a set of examples that illustrate how to build and run a variety of different types of programs on Gaea (e.g., MPI, CUDA, OpenMP, some of the above, all of the above, or none of the above).
To get a copy of the examples first login to Gaea and enter the following command:
gaea% cp -RL /home/examples/examples.bash ./
This will create a directory called examples. Enter that directory and you will see a set of directories, one directory for each example.
gaea% cd examples.bashgaea% ls
mpi-cuda-noopenmp/ mpi-nocuda-noopenmp/ nompi-cuda-noopenmp/ nompi-nocuda-noopenmp/mpi-cuda-openmp/ mpi-nocuda-openmp/ nompi-cuda-openmp/ nompi-nocuda-openmp/
gaea%
Each directory is described as follows:
| Directory | Uses MPI | Uses CUDA | Uses OpenMP |
|---|---|---|---|
| nompi-nocuda-noopenmp | |||
| mpi-nocuda-noopenmp | |||
| nompi-cuda-noopenmp | |||
| nompi-nocuda-openmp | |||
| mpi-cuda-noopenmp | |||
| mpi-nocuda-openmp | |||
| nompi-cuda-openmp | |||
| mpi-cuda-openmp |
Note, by default, all MPI examples are configured for use with the 'OpenMPI' implementation of the MPI standard. There are other MPI implementations installed on Gaea and instructions on how to choose a different MPI are included in the examples above. To use Intel's MPI instead of the default OpenMPI, see the included "pbs" scripts and make any recommended adjustments.
Select the example directory you want and enter that directory. For example, if you want to see an example that uses both MPI and CUDA then you would 'cd mpi-cuda-noopenmp' into that directory. Once there you will find source code and a Makefile.
You should be able to build the program like this (yuo may skip the first 'module purge ... module load' line if these modules already loaded via your ~/.bashrc file)
gaea% module purge; module load cuda/cuda-7.5; module load openmpi/openmpi-2.1.0;
gaea% cd mpi-cuda-noopenmp
gaea% make clean
gaea% make world
rm -f *.o progmpic++ -O2 -Wall -Werror -c -o main.o main.ccmpic++ -O2 -Wall -Werror -c -o addvec_mpi.o addvec_mpi.ccnvcc -arch=sm_20 -g -G -Xcompiler '-O2 -Wall -Werror' -c addvec_cuda.cu -o addvec_cuda.ompic++ -O2 -Wall -Werror -L/usr/local/cuda/7.5/lib64 -lcudart -lcuda -o prog main.o addvec_mpi.o addvec_cuda.o
gaea%
This will create the executable file with filename prog.
All of the work that you have done thus far on Gaea has been on the Gaea computer that you logged on to. That computer is called the login node. In addition to serving as the entry point for users to login to Gae, the login computer is also there for users to create, edit, store, and build their programs. The programs that are created and built on Gaea's login node are executed on a different set of computers, Gaea's compute nodes.
Before executing your program on Gaea's compute nodes you must first create a small file that describes the program you want to run. This file is called a PBS script. It is typically created only once for each application you want to run and thereafter it is simply re-used each time you want to run your program. Each example directory already has a PBS script that has been created for you and that is ready to use.
Once you have your PBS script you submit it to Gaea using the 'qsub' command. Gaea's scheduler reads the contents of PBS script, creates what it calls a 'job' for that execution of your program, and places the newly created job in its queue.
You can submit the PBS script created for you in the example directory using the qsub command like this:
gaea% qsub pbs.bash
269761.kvm-srv1.its.niu.edugaea%
In the example above, Gaea's scheduler created a job based on the contents of the PBS script having filename pbs.bash and assigned that job a unique ID, 269761.
In most cases, jobs sit in the queue for a time as they wait for Gaea's compute nodes to become available, then they are loaded onto Gaea's compute nodes for execution, and then end by entering a brief completion stage as Gaea's scheduler removes the application from the compute nodes. So the three stages that most jobs go through are queued, running, and completion. In some (lucky) cases a submitted job might spend little or no time in the queue and start running immediately after submission.
To observe the status of the job over time, use the qstat command.
A job in the queue (Q) waiting to run on 2 nodes looks like this:
gaea% qstat -a 269761
kvm-srv1.its.niu.edu:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
gaea%The output from qstat (i.e., running qstat again) will look like that as long as the job is in the queue. Eventually, the job will be removed from the queue and loaded onto Gaea's compute nodes for execution. When that happens the status of the job changes in qstat's output from Q to Q for "running" and will look like this:
gaea% qstat -a 269761
kvm-srv1.its.niu.edu:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Once the job has completed its execution there will be a brief period of time when Gaea will perform some cleanup operations. If you happen to run
qstat during the cleanup phase you will see the job status has changed from R to C and will look like this:
gaea% qstat -a 269761
kvm-srv1.its.niu.edu:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Eventually, the cleanup phase will complete and the job will be removed from qstat's view altogether. When that happens qstat's output will look like this:
gaea% qstat -a 269761
qstat: Unknown Job Id Error 269761.kvm-srv1.its.niu.edugaea%
When your program runs, a file will be created in the same directory where you submitted the qsub command. All output (stdout and stderr) output generated by your program will go into that file. The filename will have the form pbs.bash.o269761.
In this example, the application does nothing more than print 'SUCCESS'. The other lines that appear in the output are explained in 'A closer look at a PBS file'
gaea% cat pbs.bash.o269761
Disabled persistence mode for GPU 0000:06:00.0.Disabled persistence mode for GPU 0000:11:00.0.All done.Enabled persistence mode for GPU 0000:06:00.0.Enabled persistence mode for GPU 0000:11:00.0.All done.Compute mode is already set to DEFAULT for GPU 0000:06:00.0.Compute mode is already set to DEFAULT for GPU 0000:11:00.0.All done.****************************************************Job starting at: Wed Mar 21 13:25:52 CDT 2018****************************************************SUCCESS!****************************************************Job completed at: Wed Mar 21 13:25:54 CDT 2018****************************************************
gaea%
The example program generated exactly one line in the output file (SUCCESS!). All the other lines are generated either by (a) Gaea's system or (b) print (i.e., 'echo') statements in the PBS script.
Below are the contents of this example application's PBS file:
gaea% cat pbs.bash
#!/bin/bash
#PBS -j oe
#PBS -l nodes=2:ppn=2:gpus=2
#PBS -l walltime=00:15:00
# #PBS -A <account>
# Change to the directory where the 'qsub' command was executed.
cd $PBS_O_WORKDIR
echo '****************************************************'
echo "Job starting at: `date`"
echo '****************************************************'
#set -x # Comment this in for useful debugging output
# Load required modules
module purge; module load openmpi/openmpi-2.1.0; module load cuda/cuda-7.5
module list
mpirun ./prog
set +x
echo '****************************************************'
echo "Job completed at: `date`"
echo '****************************************************'
gaea%
#!/bin/bash#PBS -j o#PBS -l nodes=2:ppn=2:gpus=2
#PBS -l walltime=00:15:00##PBS -A project
# and place #PBS starting in the leftmost column) and replace project with the project associated with this job.cd $PBS_O_WORKDIR
The $PBS_O_WORKDIR variable will be set to the location where you executed the qsub command. Presumably, you would have done that from the directory where your application program is as shown above. If not, then you can hard-code any directory in its place that makes sense for your project.
4) This example logs the time it starts and finishes using the echo commands shown above.
5) Run the job as required. In this case, it is an MPI job. We used the mpirun command because it is compatible with both Intel and OpenMPI variations of MPI.
To remind, the output file contains a combination of output from the example program (SUCCESS!), Gaea's system and print (i.e., 'echo') statements in the PBS script.
gaea% cat pbs.bash.o269761
Disabled persistence mode for GPU 0000:06:00.0.
Disabled persistence mode for GPU 0000:11:00.0.
All done.
Enabled persistence mode for GPU 0000:06:00.0.
Enabled persistence mode for GPU 0000:11:00.0.
All done.
Compute mode is already set to DEFAULT for GPU 0000:06:00.0.
Compute mode is already set to DEFAULT for GPU 0000:11:00.0.
All done.
****************************************************
Job starting at: Wed Mar 21 13:25:52 CDT 2018
****************************************************
SUCCESS!
****************************************************
Job completed at: Wed Mar 21 13:25:54 CDT 2018
****************************************************
gaea%
Long (24 hrs and more) multinode jobs need careful construction to help the batch job system run efficiently. Before submitting such jobs, make sure that
check the load of nodes running your jobs using the "jobstat" command. For a well-balanced MPI job, the output will be similar to those below:gaea% jobstat
Remaining Used Average Job User Account Class time % Nodes load------------------------------------------------------------------------325035 xxxxxx hpc extra 3:09:05:18 18.9% 6 12.33
Specifically, "average load" indicated that all 12 CPUs on each nodes are busy with tasks.
The lower this parameter the less CPUs are in use and the less efficient the usage of requested CPUs.
!!! ! Note:: if hh:mm:ss <= 00:30:00 and nodes=1, a job will be assigned to the special (reserved) node; useful to minimize the waiting time for test jobs when the cluster is under a heavy load.