Resource management
Login nodes
The CCB login nodes are provided in order to allow file managment, light development work and submission of jobs to the cluster. Since the login nodes are a shared resource which can be in use by dozens of people at once, it's not possible for us to permit a single user to monopolise on the resources. As such, your sessions are restricted to a small quantity of CPUs and memory comparable to a desktop, ensuring one user doesn't monopolise the system:
login1: 2 CPUs and 6GB memorylogin2: 4 CPUs and 15GB memory
Resource watchdogs
Our login nodes, and our R/Jupyter server, each run resource watchdogs to monitor total CPU and memory use. These ensure that total resource use does not reach 100%, which would effectively lock the system for everyone. These do not apply to our batch nodes, which instead are managed by our Slurm scheduler.
CPU watch
Our CPU watchdog program automatically applies a nice value to all user processes. The more resources you use, the higher your processes are niced - this means your processes have a lower priority than other users, so will only consume CPU resources if not required by other users.
mem watch
Our memory watchdog program monitors total memory use, and if it approaches 100% then the watchdog will terminate the largest consumer of memory. Given the total memory available it's unlikely this will happen under normal circumstances, but if it does, you'll receive an e-mail from us to tell you what's happened. The threshold is set at around 90-95%; this ensures use doesn't reach 100% between watchdog checks, and reserves some memory for caching.
How much memory a process can consume without being terminated, particularly on our R/Jupyter server, depends on the resource use of other users also present. As this server is a popular shared resource, memory consumption varies signficantly depending on use, and so free memory varies. Below is real memory consumption on our R/Jupyter server over a 3 month period - free memory has ranged between 150GB and 500GB.
