c. Run UFS

Now that we’ve installed UFS and extracted the test case. We’re able to run the forecast.

Create a Slurm sbatch script to run the test case:

cd /shared/simple-test-case/
cat > slurm-ufs.sh << EOF
#!/bin/bash

#SBATCH --chdir=/shared/simple-test-case/
#SBATCH --job-name=UFS
#SBATCH --output=slurm-ufs-%j.out
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --exclusive

export I_MPI_OFI_LIBRARY_INTERNAL=0
spack load intel-oneapi-mpi
spack load ufs-weather-model
module load libfabric-aws
set -x
ulimit -s unlimited
ulimit -a

export FI_PROVIDER=efa
export I_MPI_DEBUG=4
export I_MPI_FABRICS=ofi
export I_MPI_OFI_PROVIDER=efa
export I_MPI_PIN_DOMAIN=omp
export KMP_AFFINITY=compact
export OMP_NUM_THREADS=6

# run the model
time mpiexec.hydra -np \$SLURM_NTASKS --ppn \$SLURM_NTASKS_PER_NODE ufs_weather_model
EOF

Submit the job:

sbatch slurm-ufs.sh

Monitor the job’s status with squeue:

squeue

Using 192 cores, the job took 1 mins 56 seconds to complete.