Kokkos::View related segfaults when using the SYCL backend

[ThomasJannaudCAM]

Hi all,

In Cambridge we just received some new Intel Ponte Vecchio GPUs. With a software engineer, we have been trying to run IDEFIX on them. She was able to compile Idefix on a number of the test cases using the SYCL backend provided by Kokkos, but at runtime it segfaults with an error about Kokkos::View. We are wondering if this is a fault with Idefix or with Kokkos. She has been able to reproduce it on two different PVC systems (each with slightly different environments - see below).

Thanks in advance !

Below is her description of how to reproduce the error:

In the test/MHD/disk directory type:

cmake ${IDEFIX_DIR} -DIdefix_MPI=ON -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DCMAKE_CXX_COMPILER=mpiicpx

then when it has finished configuring the build type ‘make'.

Run the Idefix executable (I used 4 MPI ranks but it doesn’t really matter, I’ve also tried with 1):

mpirun -np 4 ./idefix

This is the output:

jk945@sw-pvc03:~/idefix/test/MHD/disk $ mpirun -np 4 ./idefix
[0] MPI startup(): ===== GPU topology on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): NUMA nodes     : 2
[0] MPI startup(): GPUs           : 4
[0] MPI startup(): Stacks (Tiles) : 8
[0] MPI startup(): Hierarchy mode : flat
[0] MPI startup(): NUMA Id
GPU Id         Stacks (tiles)                 Ranks on this NUMA
[0] MPI startup(): 0      0,1            (0,1)(2,3)                     0,1
[0] MPI startup(): 1      2,3            (4,5)(6,7)                     2,3
[0] MPI startup(): ===== GPU pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): Rank
Pin stack (tile)
[0] MPI startup(): 0
{0,1}
[0] MPI startup(): 1
{2,3}
[0] MPI startup(): 2
{4,5}
[0] MPI startup(): 3
{6,7}
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[0] MPI startup(): Intel(R) MPI Library, Version 2021.13  Build 20240515 (id: df72937)
[0] MPI startup(): Copyright (C) 2003-2024 Intel Corporation.  All rights reserved.
[0] MPI startup(): library kind: release
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
MPI startup(): Warning: I_MPI_PMI_LIBRARY will be ignored since the hydra process manager was found
[0] MPI startup(): libfabric version: 1.20.1-impi
[0] MPI startup(): libfabric provider: mlx
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400_x1.dat" not found
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx_400.dat" not found
[0] MPI startup(): File "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi_mlx.dat" not found
[0] MPI startup(): Load tuning file: "/alt/applic/intel/oneapi/mpi/2021.13/opt/mpi/etc/tuning_spr_shm-ofi.dat"
[0] MPI startup(): Number of NICs:  1 
[0] MPI startup(): ===== NIC pinning on [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/) =====
[0] MPI startup(): Rank    Thread id  Pin nic
[0] MPI startup(): 0       0          mlx
[0] MPI startup(): 1       0          mlx
[0] MPI startup(): 2       0          mlx
[0] MPI startup(): 3       0          mlx
[0] MPI startup(): ===== CPU pinning =====
[0] MPI startup(): Rank    Pid      Node name                 Pin cpu
[0] MPI startup(): 0       2296614  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27}
[0] MPI startup(): 1       2296615  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54
                                                ,55}
[0] MPI startup(): 2       2296616  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82
                                                ,83}
[0] MPI startup(): 3       2296617  [sw-pvc03.maths.cam.ac.uk](http://sw-pvc03.maths.cam.ac.uk/)  {84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,
                                                108,109,110,111}
                                  .:HMMMMHn:.  ..:n..
                                .H*'``     `'%HM'''''!x.
         :x                    x*`           .(MH:    `#h.
        x.`M                   M>        :nMMMMMMMh.     `n.
         *kXk..                XL  nnx:.XMMMMMMMMMMML   .. 4X.
          )MMMMMx              'M   `^?M*MMMMMMMMMMMM:HMMMHHMM.
          MMMMMMMX              ?k    'X ..'*MMMMMMM.#MMMMMMMMMx
         XMMMMMMMX               4:    M:MhHxxHHHx`MMx`MMMMMMMMM>
         XM!`   ?M                `x   4MM'`''``HHhMMX  'MMMMMMMM
         4M      M                 `:   *>     `` .('MX   '*MMMM'
          MX     `X.nnx..                        ..XMx`     'M*X
           ?h.    ''```^'*!Hx.     :Mf     xHMh  M**MMM      4L`
            `*Mx           `'*n.x. 4M>   :M` `` 'M    `       %
             '%                ``*MHMX   X>      !
            :!                    `#MM>  X>      `   :x
           :M                        ?M  `X     .  ..'M
           XX                       .!*X  `x   XM( MMx`h
          'M>::                        `M: `+  MMX XMM `:
          'M> M                         'X    'MMX ?MMk.Xx..
          'M> ?L                     ...:!     MMX.H**'MMMM*h
           M>  #L                  :!'`MM.    . X*`.xHMMMMMnMk.
           `!   #h.      :L           XM'*hxHMM*MhHMMMMMMMMMM'#h
           +     XMh:    4!      x   :f   MM'   `*MMMMMMMMMM%  `X
           M     Mf``tHhxHM      M>  4k xxX'      `#MMMMMMMf    `M .>
          :f     M   `MMMMM:     M>   M!MMM:         '*MMf'     'MH*
          !     Xf   'MMMMMX     `X   X>'h.`          :P*Mx.   .d*~..
        :M      X     4MMMMM>     !   X~ `Mh.      .nHL..M#'%nnMhH!'`
       XM      d>     'X`'**h     'h  M   ^'MMHH+*'`  ''''   `'**'
    %nxM>      *x+x.:. XL.. `k     `::X
:nMMHMMM:.  X>  Mn`*MMMMMHM: `:     ?MMn.
    `'**MML M>  'MMhMMMMMMMM  #      `M:^*x
         ^*MMttnnMMMMMMMMMMMH>.        M:.4X
                        `MMMM>X   (   .MMM:MM!   .
                          `'''4x.dX  +^ `''MMMMHM?L..
                                ``'           `'`'`'`

              Idefix version 2.2.00-54482d58
              Built against Kokkos 40500
              Compiled on Mar  5 2025 at 17:00:09


Main: initialization stage.
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_xr0")
Backtrace:
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x14acd1a29d90] 
[0x14acd1a29e40] __libc_start_main
[0x40eec5] 
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x15455c029d90] 
[0x15455c029e40] __libc_start_main
[0x40eec5] 
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x14f2af629d90] 
[0x14f2af629e40] __libc_start_main
[0x40eec5] 
[0x6c5ab0] 
[0x69f3e2] 
[0x421166] 
[0x4210fd] 
[0x5c27b2] 
[0x48f765] 
[0x67d06f] 
[0x151e84429d90] 
[0x151e84429e40] __libc_start_main
[0x40eec5] 

I do not see this error with the CUDA backend.

Here are the versions of Intel oneAPI packages that I have loaded:

jk945@sw-pvc03:~/idefix/test/MHD/disk $ module list
Currently Loaded Modulefiles:
 1) intel/tbb/2021.13                          
 2) intel/compiler-rt/2024.2.0                 
 3) intel/compiler-intel-llvm/2024.2.0         
 4) intel/mpi/2021.13                          
 5) intel/mkl/2024.2                           
 6) glibc/2.35/gcc/spmhgmal                    
 7) intel-oneapi-dpl/2022.5.0/oneapi/jxepuwvw  
 8) ncurses/6.2/gcc-7.2.0-awbmlbc              
 9) cmake/3.22.3/gcc-7.2.0-ffrdpvs

[glesur]

can't really help much as we don't have any intel gpu here. A few thoughts:

• you should try to run in serial first (ie without MPI enabled)
• try to enable IDEFIX_Debug=ON (in addition to KOKKOS_Debug, as done above)
• can you tell me the output of cmake (when called from a clean build directory)

[glesur]

additional thoughts: maybe add -DKokkos_ENABLE_SERIAL=ON to cmake, not sure it is enabled by default with syscl.

[julianakwan]

Hi @glesur - thanks very much for your suggestions.
I have tried a serial build with both Idefix and Kokkos debugging enabled but the same error with Kokkos::View still occurs.
This is the line I use to compile:
cmake ${IDEFIX_DIR} -DKokkos_ENABLE_SYCL=ON -DKokkos_ARCH_INTEL_PVC=ON -DKokkos_ENABLE_DEBUG=ON -DIdefix_DEBUG=ON

Here is the output from cmake

-- The C compiler identification is IntelLLVM 2025.0.0
-- The CXX compiler identification is IntelLLVM 2025.0.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icx - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/aurora/24.180.3/updates/oneapi/compiler/eng-20240629/bin/icpx - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Setting default Kokkos CXX standard to 17
-- Kokkos version: 4.5.0
-- The project name is: Kokkos
-- Using internal gtest for testing
-- SERIAL backend is being turned on to ensure there is at least one Host space. To change this, you must enable another host execution space and configure with -DKokkos_ENABLE_SERIAL=OFF or change CMakeCache.txt
-- Using -std=gnu++17 for C++17 extensions as feature
-- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL
-- Looking for SYCL_EXT_ONEAPI_DEVICE_GLOBAL - found
-- Looking for SYCL_EXT_ONEAPI_GRAPH
-- Looking for SYCL_EXT_ONEAPI_GRAPH - found
-- Built-in Execution Spaces:
--     Device Parallel: Kokkos::SYCL
--     Host Parallel: NoTypeDefined
--       Host Serial: SERIAL
-- 
-- Architectures:
--  INTEL_PVC
-- Found TPLLIBDL: /usr/include  
-- Looking for C++ include oneapi/dpl/execution
-- Looking for C++ include oneapi/dpl/execution - found
-- Looking for C++ include oneapi/dpl/algorithm
-- Looking for C++ include oneapi/dpl/algorithm - found
-- Performing Test KOKKOS_NO_TBB_CONFLICT
-- Performing Test KOKKOS_NO_TBB_CONFLICT - Success
-- Using internal desul_atomics copy
-- Experimental mdspan support is enabled
-- Looking for C++ include experimental/mdspan
-- Looking for C++ include experimental/mdspan - not found
-- Looking for C++ include mdspan
-- Looking for C++ include mdspan - not found
-- Using internal mdspan directory /lus/flare/projects/grteclyn/idefix/src/kokkos/tpls/mdspan/include
-- Kokkos Backends: SERIAL;SYCL
-- Including problem-specific CMakeLists in '/lus/flare/projects/grteclyn/idefix/test/MHD/sod'
--     Explicitely setting Idefix_MHD = ON
-- Idefix final configuration
--     MHD:  ON
--     MPI:  OFF
--     HDF5: OFF
--     Python: OFF
--     Reconstruction: Linear
--     Precision: Double
--     Version: 2.2.00-54482d58
--     Problem definitions: 'definitions.hpp'
-- Configuring done (27.4s)
-- Generating done (2.8s)

Then I build using make and everything is fine at this stage.
When I run the executable, this is the output:

./idefix
----> Profiler::Init...
----> ...returned
                                  .:HMMMMHn:.  ..:n..
                                .H*'``     `'%HM'''''!x.
         :x                    x*`           .(MH:    `#h.
        x.`M                   M>        :nMMMMMMMh.     `n.
         *kXk..                XL  nnx:.XMMMMMMMMMMML   .. 4X.
          )MMMMMx              'M   `^?M*MMMMMMMMMMMM:HMMMHHMM.
          MMMMMMMX              ?k    'X ..'*MMMMMMM.#MMMMMMMMMx
         XMMMMMMMX               4:    M:MhHxxHHHx`MMx`MMMMMMMMM>
         XM!`   ?M                `x   4MM'`''``HHhMMX  'MMMMMMMM
         4M      M                 `:   *>     `` .('MX   '*MMMM'
          MX     `X.nnx..                        ..XMx`     'M*X
           ?h.    ''```^'*!Hx.     :Mf     xHMh  M**MMM      4L`
            `*Mx           `'*n.x. 4M>   :M` `` 'M    `       %
             '%                ``*MHMX   X>      !
            :!                    `#MM>  X>      `   :x
           :M                        ?M  `X     .  ..'M
           XX                       .!*X  `x   XM( MMx`h
          'M>::                        `M: `+  MMX XMM `:
          'M> M                         'X    'MMX ?MMk.Xx..
          'M> ?L                     ...:!     MMX.H**'MMMM*h
           M>  #L                  :!'`MM.    . X*`.xHMMMMMnMk.
           `!   #h.      :L           XM'*hxHMM*MhHMMMMMMMMMM'#h
           +     XMh:    4!      x   :f   MM'   `*MMMMMMMMMM%  `X
           M     Mf``tHhxHM      M>  4k xxX'      `#MMMMMMMf    `M .>
          :f     M   `MMMMM:     M>   M!MMM:         '*MMf'     'MH*
          !     Xf   'MMMMMX     `X   X>'h.`          :P*Mx.   .d*~..
        :M      X     4MMMMM>     !   X~ `Mh.      .nHL..M#'%nnMhH!'`
       XM      d>     'X`'**h     'h  M   ^'MMHH+*'`  ''''   `'**'
    %nxM>      *x+x.:. XL.. `k     `::X
:nMMHMMM:.  X>  Mn`*MMMMMHM: `:     ?MMn.
    `'**MML M>  'MMhMMMMMMMM  #      `M:^*x
         ^*MMttnnMMMMMMMMMMMH>.        M:.4X
                        `MMMM>X   (   .MMM:MM!   .
                          `'''4x.dX  +^ `''MMMMHM?L..
                                ``'           `'`'`'`

              Idefix version 2.2.00-54482d58
              Built against Kokkos 40500
              Compiled on Mar 11 2025 at 11:14:32


Main: initialization stage.
----> Grid::Grid(Input)...
----> ...returned
----> GridHost::GridHost(Grid)...
----> ...returned
----> GridHost::MakeGrid...
----> ...returned
----> GridHost::SyncToDevice...
----> ...returned
----> DataBlock::DataBlock...
--------> GridHost::GridHost(Grid)...
--------> ...returned
--------> GridHost::SyncFromDevice...
--------> ...returned
--------> DataBlock::ExtractSubdomain...
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
------------> idefix_for(coordinates)...
------------> ...returned
--------> ...returned
--------> DataBlock::MakeGeometry()...
------------> idefix_for(Volumes)...
Kokkos::View ERROR: attempt to access inaccessible memory space (label="DataBlock_dx0")
Backtrace:
[0x6069d0] 
[0x5e02d2] 
[0x411d76] 
[0x41af1d] 
[0x51e5c8] 
[0x45879b] 
[0x5d0c2a] 
[0x14d60757124d] __libc_start_main
[0x40ef0a] 
Aborted

[julianakwan]

Hi @glesur - I'm happy to report that that worked! There was a lot of output because I compiled on debug so let me know if you want to see it. I also checked with MPI (but always 1 MPI rank to 1 GPU) and it was fine as well. Thanks so much!

[ThomasJannaudCAM]

Thank you both very much !

[glesur]

Excellent, I'll propagate the fix to the develop branch. @ThomasJannaudCAM , whenever you have time, I'd like to know how it performs on Intel PVCs on the OT3D test with 128^3 points and no tracer.

[ThomasJannaudCAM]

What would you want ? Frequency of cell updates and parallel efficiency ?

[glesur]

yes, cell update/s (don't really care about the parallel efficiency for now), as in https://idefix.readthedocs.io/latest/performances.html

[glesur]

A fix for this has been merged into develop (#331) and will be part of the next Idefix release.