Skip to content

Add video encoding support with SYCL RGB→NV12 kernel and CPU fallback - #103

Open
eromomon wants to merge 9 commits into
intel:mainfrom
eromomon:eromomon/encoding_no_patch
Open

Add video encoding support with SYCL RGB→NV12 kernel and CPU fallback#103
eromomon wants to merge 9 commits into
intel:mainfrom
eromomon:eromomon/encoding_no_patch

Conversation

@eromomon

Copy link
Copy Markdown
Contributor

Adds a hardware video encoding path to the torchcodec-xpu plugin (supersedes #58)

What changes:

  • XpuDeviceInterface.{h,cpp} — implements three encoder overrides:
    • get_encoding_pixel_format (pinned to NV12).
    • setup_hardware_frame_context_for_encoding (VAAPI hw_frames_ctx, BT.709 + JPEG-range defaults, B-frames disabled, low_power=0, quality=1).
    • convert_tensor_to_av_frame_for_encoding dispatching to:

      • -SYCL fast path — exports the VA surface as DMA-BUF, imports via Level Zero USM, runs convertRGBToNV12 directly on the surface (handles linear + Intel Tile-Y, and both NV12 export layouts on iHD/i915).
      • -CPU fallback — GBRP→NV12 via libswscale, then av_hwframe_transfer_data onto the VA surface. Used when SYCL is disabled or FP64 is unavailable (DG2 / ATS-M).
    • find_codec extended to resolve VAAPI encoders, with H.264 / HEVC / AV1 fallback when the requested codec id has no VAAPI encoder.
  • ColorConversionKernel.{h,cpp} — adds convertRGBToNV12 SYCL kernel (BT.601 / BT.709, limited + full range); get_tile_offset reused for encode.

  • patches/0002-Add-XPU-support-to-video-encoder-tests.patch — parametrizes upstream torchcodec encoder tests over CUDA + XPU, and primes the encoder with one frame in test_write_frames_different_devices_errors (VAAPI segfaults on empty flush; NVENC doesn't). Depends on 0001-Add-XPU-support-to-tests.patch.

Compatibility
No change to decode behavior, plugin loading, or public API.

Testing
Encoder test suite (patches 0001 + 0002) passes on Intel Arc / Battlemage.

@eromomon
eromomon requested review from Copilot and removed request for dvrogozh and luis-real July 27, 2026 22:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an Intel XPU (VAAPI) hardware video encoding path to the torchcodec-xpu plugin, including an on-device SYCL RGB→NV12 conversion kernel with a CPU/libswscale fallback, and updates upstream TorchCodec encoder tests (via patch) to run on XPU as well as CUDA.

Changes:

  • Implement XPU encoding overrides in XpuDeviceInterface (pixel format selection, VAAPI hw_frames_ctx setup, tensor→AVFrame upload path with SYCL fast-path + CPU fallback).
  • Add SYCL convertRGBToNV12 kernel supporting BT.601/BT.709 and full/limited range, including tiled (Intel Tile-Y) surface handling.
  • Extend/parameterize upstream encoder tests (patch) to exercise XPU and avoid VAAPI flush-on-empty segfault by priming with a frame.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.h Declares XPU encoding overrides and helper conversion entry points.
packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Implements VAAPI frames context setup, VAAPI encoder discovery logic, and SYCL/CPU tensor→NV12 surface conversion.
packages/torchcodec-xpu/src/torchcodec_xpu/ColorConversionKernel.h Exposes the new SYCL RGB→NV12 conversion API.
packages/torchcodec-xpu/src/torchcodec_xpu/ColorConversionKernel.cpp Implements RGB→NV12 SYCL kernel and supporting YUV conversion matrices.
packages/torchcodec-xpu/patches/0002-Add-XPU-support-to-video-encoder-tests.patch Updates upstream TorchCodec encoder tests to include XPU device coverage and VAAPI-specific priming behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/patches/0002-Add-XPU-support-to-video-encoder-tests.patch Outdated
@eromomon
eromomon requested a review from dvrogozh July 27, 2026 23:29
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/patches/0002-Add-XPU-support-to-video-encoder-tests.patch Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
@eromomon
eromomon requested a review from dvrogozh July 29, 2026 00:41
@dvrogozh

dvrogozh commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

@eromomon , please, rebase on top of latest main and resolve conflicts.

@eromomon
eromomon force-pushed the eromomon/encoding_no_patch branch from 5386ae9 to 652caef Compare August 5, 2026 16:01
Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>
@eromomon
eromomon force-pushed the eromomon/encoding_no_patch branch from 652caef to 2023c4f Compare August 5, 2026 16:51
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
Comment thread packages/torchcodec-xpu/src/torchcodec_xpu/XpuDeviceInterface.cpp Outdated
eromomon and others added 3 commits August 5, 2026 12:21
Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
@dvrogozh

dvrogozh commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Last concern I have on this PR is that non-SYCL path might not be working:

USE_SYCL_KERNELS=0 TORCHCODEC_XPU_LOG_LEVEL=4 pytest -rsf --capture=no test/test_encoders.py::TestEncoder::test_add_video_and_encode_frames[xpu-to_file-mp4]

...
AssertionError: Expected at least 96% of values to be within atol=2, but only 21.34965705871582% were.

dvrogozh and others added 5 commits August 6, 2026 07:34
Signed-off-by: Dmitry Rogozhkin <dmitry.v.rogozhkin@intel.com>
Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>
…te DG2.

Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>
Signed-off-by: Edgar Romo Montiel <edgar.romo.montiel@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants