Currently Hive has limits on public partitions, the limits are set per partition and the limitations are equal for all the users using public partitions.
Now there are 3 Hive public partitions, 3 preempt public partitions 3 ckptdell public partitions, 3 ckpthp public partitions, one ckptqueen partition, one queen partition(Fat memory node) and one mpi partition.
In order to view the real time partition info in Hive terminal, type command sinfo , also there is a login message for every user that contains real-time QOS limitations on each partition, user information and nodes information.
hive partition containing 40 public compute nodes and is a default. Running jobs on Hive partition is recommended for new users and for users that want to be sure that their job will run until it end without any preemption.
queen partition containing high memory compute nodes(queen's), this partition is made for a jobs that require a lot of memory.
preempt partition are preempt-able that contain all the compute nodes(bee's) as well as public and private. Preempt-able partition means that every job that submitted on higher priority partitions(hive,queen,private partitions) will kick off your job that running on preempt partition, and be restart from the beginning to the queue. on preempt partition you get a benefit of lower limitations per user.
ckpt partition are preempt-able that contain all the HP or DELL compute nodes(bee's) as well as public and private nodes. Preempt-able partition means that every job that submitted on higher priority partitions(hive,queen,private partitions) will kick off your job that running on ckpt partition, and restart the job to the queue. On ckpt partition you get a benefit of lower limitations per user. This partition is very recommended for jobs that can be checkpointed, with combination of checkpoint-able job and low limitations on ckpt partition you will be able to use more resources than from the public Hive partition.
mpi partition that contain all the public compute nodes(bee's) that made for MPI jobs that using a large amount of resources
vespa partition that contain GPU's nodes(vespa's) that made for jobs that using GPU resources.
guest partition contains thin compute nodes, made for Hive guest users(non Faculty of Science members), partition is low-priority and preempt-able by other partitions.
Private partition's partitions that contain private nodes that different groups has bough't, those partitions usually receive names of their group name. Running jobs on private partitions is allowed only by the users that belong the the partition group. private nodes has no limitations and will get 100% priority over any other partition that containing the private nodes(ckpt partitions for example).
Below you can see information about the limits each partition has:
hive1d - DEFAULT partition, limits: WallTime: 1 day, MaxCPU’sPerUser = 400 , MaxNodesPerUser=20
hive7d - limits: WallTime: 7 days, MaxCPU’sPerUser = 100 , MaxNodesPerUser=10
hiveunlim limits: WallTime: 31 day, MaxCPU’sPerUser = 60 , , MaxNodesPerUser=7
*Hive contains 40 public compute nodes
preempt1d - limits: WallTime: 1 days, MaxCPU’sPerUser = 800 Preemption=REQUEUE Low priority preempt-able partition
preempt7d - limits: WallTime: 7 days, MaxCPU’sPerUser = 600 Preemption=REQUEUE Low priority preempt-able partition
preempt31d - limits: WallTime: 31 days, MaxCPU’sPerUser = 400 Preemption=REQUEUE Low priority preempt-able partition
*Preempt contains 63 compute nodes
ckpthp1d - limits: WallTime: 1 days, MaxCPU’sPerUser = 400 Preemption=REQUEUE Low priority preempt-able partition
ckpthp7d - limits: WallTime: 7 days, MaxCPU’sPerUser = 300 Preemption=REQUEUE Low priority preempt-able partition
ckpthp31d - limits: WallTime: 31 days, MaxCPU’sPerUser = 200 Preemption=REQUEUE Low priority preempt-able partition
*HP contains 30 compute nodes
ckptdell1d - limits: WallTime: 1 days, MaxCPU’sPerUser = 400 Preemption=REQUEUE Low priority preempt-able partition
ckptdell7d - limits: WallTime: 7 days, MaxCPU’sPerUser = 300 Preemption=REQUEUE Low priority preempt-able partition
ckptdell31d - limits: WallTime: 31 days, MaxCPU’sPerUser = 200 Preemption=REQUEUE Low priority preempt-able partition
*Dell contains 32 compute nodesqueen - limits: WallTime: 31 days, MaxNODE’sPerUser = 1
ckptqueen - limits: WallTime: 31 days Preemption=REQUEUE Low priority preempt-able partition
mpi - limits: MinCPU’sPerUser = 200
guest - WallTime: 7 day, MaxCPU’sPerUser = 400 *Low priority
*WallTime= maximum time that each job can run.
*MaxCPU’sPerUser= maximum number of CPU’s user can use under specific partition.
*MaxNODE'sPerUser= maximum number of NODE's user can run jobs under specific partition.
*MinCPU'sPerUser= minimum resource allocation per job.
*Preemption=REQUEUE= low priority partition, all jobs running on that partition could be preempted/checkpointed by higher priority partition. preempted job will be restarted from beginning or from last checkpoint.
*ckpt partitions made for checkpoint-able jobs and has lower priority then hive and private partitions, that means that every ckpt job can be preempted and restarted by any higher priority job from hive or private partitions.
.
Hive has two types of slim compute nodes(bee's) and two types of fat compute nodes (queen's), please see the below ranges of different nodes:
bee001-032 has 20 CPU's per node and 128 GB memory
bee033-063 has 24 CPU's per node and 128 GB memory
queen01 has 32 CPU's and 768 GB memory
queen02-03 has 56 CPU's and 768 GB memory
vespa01 is a GPU node that has Nvidia K80 GPU with 4 CPU's/Devices that using 28 node CPU's
For additional information regarding nodes hardware, please refer to hardware section.
