CARLsim6 Installation

This documentation provides further information about installing a Hippocampome branch of CARLsim6.

GMU Supercomputer Installation

These instructions will cover methods to install a Hippocampome branch of CARLsim6 on the GMU Hopper supercomputer.

A Hippocampome branch of CARLsim6 should be downloaded. For this documentation, the ca3net_nm4cstp version of the Hippocampome branch can be downloaded. The branch should be downloaded from here. On that branch page, one should click the “Code” button and select “Download ZIP”. One should not enter the command git clone https://github.com/nmsutton/CARLsim6.git because that will download the wrong branch. If one needs to clone the branch with git then one needs to specify the “feat/ca3net_nm4cstp” branch name in that clone command to retrieve the right branch (reference).

It is recommended to extract the zip file in a “git” directory in one’s home directory on the supercomputer. One will need to make that directory. A program such as FileZilla (link) can be used to transfer files to Hopper with the sftp protocol. Specifically, currently, the hopper.orc.gmu.edu server address should be used and one’s GMU username and password should be used as login credentials. One may need to use two-factor authentication to login also. Currently, sftp can require two-factor authentication for each file transfer’s initiation. This can be somewhat impractical.

One will also need to use ssh to access Hopper. One can do that from a command prompt, such as Windows’ Powershell, or Linux’s Gnome Terminal. The command ssh gmu_username@hopper.orc.gmu.edu should be used and gmu_username should be replaced with one’s GMU username. One’s GMU password is then entered. Two-factor authentication may also be needed here.

Gencode

One will need to change the gencode information in CMakeLists.txt of the CARLsim6’s base directory to one that fits with Hopper. As of April ‘25, this is:
vim CMakeLists.txt and change “-gencode arch=compute_86,code=sm_86” to “-gencode arch=compute_80,code=sm_80”. Also “vim” is a text editor command.
Compatibility with the gencode depends on what the GPUs in use on Hopper are.

CMake

cmake needs to be installed. Specifically, version 3.22.0 is recommended to be used. Download “cmake-3.22.0-linux-x86_64.sh” from here. Copy the file to the git folder. Make it executable:
chmod +x ./cmake-3.22.0-linux-x86_64.sh
Install with the command:
./cmake-3.22.0-linux-x86_64.sh --prefix=/home/gmu_username/cmake-3.22
where gmu_username is one’s GMU username.
use full path when running cmake. E.g,:
/home/gmu_username/cmake-3.22/bin/cmake

Google Test Suite

Download Google test, and version 1.10.0 is recommended. Download the zip file from here source. Upload the zip file to the “git” directory and extract it there. Navigate to the base directory for Google test’s source code. Make the directory “build” in the googletest-release-1.10.0 base directory. Enter the “build” directory. Switch to the GPU node (reference) with:
salloc -p gpuq -q gpu --nodes=1 --ntasks-per-node=1 --gres=gpu:1g.10gb:1 -t 0-01:00:00
Note this GPU node session will have a 1 hour time limit.
Install with commands:
/home/gmu_username/cmake-3.22/bin/cmake \ -DCMAKE_INSTALL_PREFIX=/home/gmu_username/gtest-1.10 \ /home/gmu_username/git/googletest-release-1.10.0
where gmu_username is one’s GMU username.
make
make install

In one’s home directory the following should be set in the file “.bashrc”:
export GTEST_LIBRARY=/home/gmu_username/gtest_1.10/lib/libgtest.a
export GTEST_MAIN_LIBRARY=/home/gmu_username/gtest_1.10/lib/libgtest_main.a
export GTEST_ROOT=/home/gmu_username/gtest_1.10/

Compile CARLsim6

Go to the base directory of CARLsim6’s source code. Run these commands:
mkdir .build
cd .build
switch to gpu node (if not still on it):
salloc -p gpuq -q gpu --nodes=1 --ntasks-per-node=1 --gres=gpu:1g.10gb:1 -t 0-01:00:00
The code included in this CARLsim6 branch will need the C++ Boost library and therefore it should be loaded with:
module load boost/1.73.0
export CPATH=/opt/ohpc/pub/libs/gnu9/mpich/boost/1.73.0/include:${CPATH:+:${CPATH}}
/home/gmu_username/cmake-3.22/bin/cmake \
-DCMAKE_INSTALL_PREFIX=/home/gmu_username/CARLsim6 \
-DCARLSIM_NO_CUDA=OFF \
/home/gmu_username/git/CARLsim6
make all
make install

Test install by trying to run hello_world at /.build/projects/hello_world/hello_world . This may not run correctly if it is just a hello_world version not compatible with this CARLsim6 branch, and this may be fine. Oblomem projects may run correctly even if hello_world does not run.

General Hopper Commands

Some general commands are:
submit job to queue:
sbatch ./slurm_wrapper.sh
view jobs:
squeue -u gmu_username
/home/gmu_username/view_jobs.sh (when view_jobs.sh a script with the view jobs command)
cancel job
scancel <jobid>
where <jobid> is the id found from squeue -u gmu_username
load module:
module load boost/1.73.0
search for module:
module spider boost
example slurm_wrapper.sh
#!/bin/bash
#SBATCH --partition=gpuq
#SBATCH --qos=gpu
#SBATCH --gres=gpu:A100.80gb:1
#SBATCH --ntasks-per-node=1
#SBATCH --job-name="ca3full1"
#SBATCH --time=0-3:00:00
#SBATCH --output /scratch/gmu_username/ca3full1.txt
#SBATCH --mem=120G
srun ./ca3_snn_GPU

srun ./ca3_snn_GPU can be replaced with ./rebuild.sh when a rebuild script is made.
Example rebuild.sh:
make clean && make && compiled_project_name
It should be noted that the software will need to be rebuilt with make clean and make anytime the project source code is updated. Therefore, it can be helpful to have a rebuild.sh script that automates this process anytime the compiled program is to be run.
It should be also noted that the first time a project is run may require one to manually run the make command. This rebuild.sh script will exit on make clean if it does not find an already compiled file.
The rebuild.sh script will also need to be made executable with chmod +x ./rebuild.sh

Example .bashrc File

An example .bashrc file has been provided here. One may not have a .bashrc file present by default. In that case, one should copy the contents of the example .bashrc file and place it in their .bashrc file. One should replace gmu_username with their GMU username.

If one already has a .bashrc file, then one should copy all lines below ${CLUSTER} == "hopper" to else in their .bashrc file when hopper is detected as the system in use. This will add to the modules loaded and environment variables included when using Hopper. Some of these are nessisary, e.g., CUDA, for working with Oblomem. It may also be useful to include the lines in the example .bashrc after else until fi that specify code to be run when Hopper is not detected. These lines should only be included in a section when hopper is not detected.

It is possible that software versions, e.g., specified by module may change over time. If such a software version changes, one may need to use module spider to find the closest version available as described in the example .bashrc file. The ORC department may also be able to help one understand what updated software versions are available.