Set gpumem and gpuutil to zero, if a gpumem infinite value is detected

[b-reyes]

Summary

In this PR I set gpumem and gpuutil to zero, if a gpumem infinite value is detected when collecting the metrics for each job step and there are no infinite values detected. The reason why this is necessary is that some jobs have job steps that have GPU utilization and some of them are CPU-only. In the CPU-only ones, the Slurm database will report infinite values for gpumem and gpuutil, if other job steps do have some GPU utilization. By setting the values to zero for these, we can correctly obtain the gpumem and gpuutil values. We do not want to just outright set all infinite values to zero as later in the code we use all infinite values in gpumem/gpuutil to determine an incompatible GPU.

Note. Given that we transitioned over to the new GRES for the A100s, any old job that just used a100 in the gres will not behave correctly in seff. For example, seff 20704335. Although this is the case, users will still be able to see the results in sacct. Given there is a workaround for this case, I do not think it is worth it to address this corner case in the current version of seff

Testing

New GPU partitions

./seff 30738094

Job ID: 30738094
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: FAILED (exit code 1)
Nodes: 1
Cores per node: 128

-------- CPU Metrics --------
CPU Utilized: 03:57:58
CPU Efficiency: 1.39% of 11-21:45:36 core-walltime
Job Wall-clock time: 02:13:57
Memory Utilized: 35.09 GiB
Memory Efficiency: 2.44% of 1.40 TiB (11.23 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 4
GPU Type: h200
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 100%
Max GPU Memory Utilized: 134.97 GiB

./seff 30738167

Job ID: 30738167
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: FAILED (exit code 1)
Nodes: 1
Cores per node: 128

-------- CPU Metrics --------
CPU Utilized: 01:39:30
CPU Efficiency: 0.72% of 9-13:09:20 core-walltime
Job Wall-clock time: 01:47:25
Memory Utilized: 35.73 GiB
Memory Efficiency: 2.49% of 1.40 TiB (11.23 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 4
GPU Type: rtx_pro_6000
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 99%
Max GPU Memory Utilized: 92.46 GiB

Tests using new GRES layout on Alpine:

./seff 29426746

Job ID: 29426746
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 10

-------- CPU Metrics --------
CPU Utilized: 00:00:26
CPU Efficiency: 2.08% of 00:20:50 core-walltime
Job Wall-clock time: 00:02:05
Memory Utilized: 358.98 MiB
Memory Efficiency: 0.93% of 37.50 GiB (3.75 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: a100_3g.20gb
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 0%
Max GPU Memory Utilized: 16.20 GiB

./seff 28711080_66

Job ID: 29399393
Array Job ID: 28711080_66
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4

-------- CPU Metrics --------
CPU Utilized: 00:32:36
CPU Efficiency: 24.87% of 02:11:04 core-walltime
Job Wall-clock time: 00:32:46
Memory Utilized: 12.28 GiB
Memory Efficiency: 81.86% of 15.00 GiB (3.75 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: a100_80gb
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 100%
Max GPU Memory Utilized: 50.39 GiB

./seff 29437179 (This is a CPU example)

Job ID: 29437179
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: FAILED (exit code 127)
Cores: 1

-------- CPU Metrics --------
CPU Utilized: 00:00:00
CPU Efficiency: 0.00% of 00:00:06 core-walltime
Job Wall-clock time: 00:00:06
Memory Utilized: 3.76 MiB
Memory Efficiency: 0.10% of 3.75 GiB (3.75 GiB/core)```

Tests using old gres format on Alpine (checks for backward compatibility)

./seff 20522520

Job ID: 20522520
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 21

-------- CPU Metrics --------
CPU Utilized: 7-14:11:13
CPU Efficiency: 75.77% of 10-00:27:21 core-walltime
Job Wall-clock time: 11:27:01
Memory Utilized: 21.25 GiB
Memory Efficiency: 27.96% of 76.00 GiB (76.00 GiB/node)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: l40
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 41%
Max GPU Memory Utilized: 874.00 MiB

./seff 19920255 --> This is a GH200 example (in the old style the GPU type and GPU metrics will not work due to new gres)

Job ID: 19920255
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 72

-------- CPU Metrics --------
CPU Utilized: 00:00:01
CPU Efficiency: 0.00% of 261-21:21:36 core-walltime
Job Wall-clock time: 3-15:17:48
Memory Utilized: 10.12 MiB
Memory Efficiency: 0.00% of 270.00 GiB (3.75 GiB/core)

./seff 20704335 (example of jobs that uses old a100 gres so only CPU metrics will display now)

Job ID: 20704335
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2

-------- CPU Metrics --------
CPU Utilized: 00:00:26
CPU Efficiency: 40.62% of 00:01:04 core-walltime
Job Wall-clock time: 00:00:32
Memory Utilized: 2.43 GiB
Memory Efficiency: 32.37% of 7.50 GiB (3.75 GiB/core)

./seff 20374346

Job ID: 20374346
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 35

-------- CPU Metrics --------
CPU Utilized: 00:02:02
CPU Efficiency: 4.59% of 00:44:20 core-walltime
Job Wall-clock time: 00:01:16
Memory Utilized: 1.30 GiB
Memory Efficiency: 1.02% of 128.00 GiB (128.00 GiB/node)

-------- GPU Metrics --------
Number of GPUs: 2
GPU Type: mi100
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: N/A
Max GPU Memory Utilized: N/A

./seff 27802287 --> old CPU example

Job ID: 27802287
Cluster: alpine
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Cores: 1

-------- CPU Metrics --------
CPU Utilized: 00:38:44
CPU Efficiency: 81.77% of 00:47:22 core-walltime
Job Wall-clock time: 00:47:22
Memory Utilized: 277.07 MiB
Memory Efficiency: 7.22% of 3.75 GiB (3.75 GiB/core)

Blanca (all old examples, as we did not change GRES settings on Blanca)

./seff 21743835

Job ID: 21743835
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2

-------- CPU Metrics --------
CPU Utilized: 2-13:21:31
CPU Efficiency: 70.70% of 3-14:47:34 core-walltime
Job Wall-clock time: 1-19:23:47
Memory Utilized: 3.50 GiB
Memory Efficiency: 43.71% of 8.00 GiB (4.00 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: a100
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 98%
Max GPU Memory Utilized: 22.65 GiB

./seff 21925455

Job ID: 21925455
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4

-------- CPU Metrics --------
CPU Utilized: 03:46:43
CPU Efficiency: 15.25% of 1-00:46:24 core-walltime
Job Wall-clock time: 06:11:36
Memory Utilized: 127.97 GiB
Memory Efficiency: 99.98% of 128.00 GiB (128.00 GiB/node)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: h100_3g.40gb
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 0%
Max GPU Memory Utilized: 6.88 GiB

./seff 20237774

Job ID: 20237774
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: TIMEOUT (exit code 0)
Nodes: 1
Cores per node: 16

-------- CPU Metrics --------
CPU Utilized: 23:55:31
CPU Efficiency: 6.23% of 16-00:04:48 core-walltime
Job Wall-clock time: 1-00:00:18
Memory Utilized: 640.30 MiB
Memory Efficiency: 1.95% of 32.00 GiB (2.00 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: l40
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: N/A
Max GPU Memory Utilized: N/A

./seff 21921652

Job ID: 21921652
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4

-------- CPU Metrics --------
CPU Utilized: 05:02:27
CPU Efficiency: 55.54% of 09:04:32 core-walltime
Job Wall-clock time: 02:16:08
Memory Utilized: 3.69 GiB
Memory Efficiency: 46.11% of 8.00 GiB (2.00 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 2
GPU Type: rtx6000
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 69%
Max GPU Memory Utilized: 2.54 GiB

./seff 21920467

Job ID: 21920467
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4

-------- CPU Metrics --------
CPU Utilized: 08:22:50
CPU Efficiency: 39.95% of 20:58:40 core-walltime
Job Wall-clock time: 05:14:40
Memory Utilized: 77.04 GiB
Memory Efficiency: 60.19% of 128.00 GiB (128.00 GiB/node)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: t4
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 99%
Max GPU Memory Utilized: 2.15 GiB

./seff 20036817

Job ID: 20036817
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 10

-------- CPU Metrics --------
CPU Utilized: 00:00:01
CPU Efficiency: 0.83% of 00:02:00 core-walltime
Job Wall-clock time: 00:00:12
Memory Utilized: 71.61 MiB
Memory Efficiency: 0.35% of 20.00 GiB (2.00 GiB/core)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: p100
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 0%
Max GPU Memory Utilized: 0.00 MiB

./seff 21735102

Job ID: 21735102
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Cores: 1

-------- CPU Metrics --------
CPU Utilized: 03:19:06
CPU Efficiency: 99.54% of 03:20:01 core-walltime
Job Wall-clock time: 03:20:01
Memory Utilized: 3.15 GiB
Memory Efficiency: 53.72% of 5.86 GiB (5.86 GiB/node)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: Unknown
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 91%
Max GPU Memory Utilized: 1.78 GiB

./seff 21915919

Job ID: 21915919
Cluster: blanca
User/Group: <redacted>/<redacted>pgrp
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 4

-------- CPU Metrics --------
CPU Utilized: 03:29:11
CPU Efficiency: 28.99% of 12:01:36 core-walltime
Job Wall-clock time: 03:00:24
Memory Utilized: 49.11 GiB
Memory Efficiency: 38.37% of 128.00 GiB (128.00 GiB/node)

-------- GPU Metrics --------
Number of GPUs: 1
GPU Type: v100
NOTE: GPU metric availability may vary by GPU type.
      Please refer to our documentation for details: curc.readthedocs.io/en/latest/getting_started/faq.html#why-am-i-getting-unexpected-results-for-my-gpu-memory-or-utilization-metrics
Max GPU Utilization: 97%
Max GPU Memory Utilized: 2.35 GiB```

[mohalkh5]

Looks good to me, thanks for tracking down the issue and for the thorough testing.

One question: regarding seff not working for the older A100 GRES jobs, do you think it's worth adding a brief note in the output/display message to let users know why GPU metrics aren't shown, or would that be overkill?

Other than that all looks good! Approved!