AI for Media - Latest Release

AI & Data Science Deep Learning (Training & Inference) AI for Media
,
1

NVIDIA AI for Media R22 Is Here

NVIDIA AI for Media introduces new models and capabilities across the collection of SDKs, NIM microservices, and NVIDIA Broadcast. Highlights include a new body pose model, real-time video super resolution and frame generation, an upgraded Synthetic Video Detector, and Windows-on-ARM support.

AI for Media SDKs

  • Augmented Reality:

    • 3D Body Pose

      • Markerless human motion single-camera capture.

      • 77-point NVIDIA SOMA (Solving Optical Marker-Based MoCap Automatically) output.

      • Latency and performance improvements.

    • Active Speaker Detection (ASD)

      • Available as both an SDK feature and a NIM microservice.

      • A multimodal feature that identifies which person in a video frame is currently speaking by analyzing synchronization between the audio and the visual lip-movement signal.

      • ASD now supports multi-camera and multi-microphone input.

    • LipSync (Private Access)

      • Occlusion handling for microphones and hands, improved lip and teeth texture preservation.

      • Added robustness to silent audio, reducing unintended mouth movement.

      • Support for a wider range of head poses.

    • Eye Contact

      • Now supports multi-person instances.

  • Video Effects:

    • Video Frame Generation (VFG).

      • Increases frame rates by 2x or 4x while preserving visual quality and temporal consistency.

      • Supports frame-rate conversion and frame boosting for generative AI video workflows.

      • 6x and 8x frame generation available as preview options.

Video Super Resolution (VSR)

  • New performance gains and quality improvements.

  • Now includes 10-bit support.

  • Available in the Video Effects SDK, as a NIM microservice, and as a Python wheel.

  • TrueHDR

    • Converts SDR content to HDR using the perceptual quantizer (PQ) transfer function.

  • Unified VFX API

    • Video Frame Generation, Video Super Resolution and TrueHDR run through the same API, so effects can be combined in a single pipeline with one runtime initialization.

  • Audio Effects (AFX) SDK

    • Studio Voice

      • Adds profile-based tuning for the input microphone.

  • Updated NIM microservices includes the latest Synthetic Video Detector and generally available releases for Active Speaker Detection, 3D Body Pose, and Video Super Resolution. The SDKs and NVIDIA Broadcast also provide feature parity on Windows on ARM, while Python wheel 2.0.0 adds Video Super Resolution, Video Frame Generation, and TrueHDR.

  • NVIDIA Broadcast 2.3.0 adds offline-file processing, an MCP server, Video Super Resolution, Video Frame Generation, Studio Voice Mic Profiles, and RTX Spark x86 feature parity.

AI for Media NIM Microservices

  • Synthetic Video Detector (SVD)

    • Model improvements

      • Substantially improved temporal stability.

      • Increased robustness with compressed content.

      • State-of-the-art accuracy on the SynthForensics and AIGVDBench datasets.

      • Scores calibrated to interpretable probabilities.

      • Approximately 2× performance improvement as measured on NVIDIA L40 GPUs.

    • NIM Improvements

      • Delivers 2× to 4× higher end-to-end throughput improvements.

      • Supports concurrent stream processing: NVIDIA B200 handles 11 or more concurrent 4Kp30 streams, while L40S handles approximately four.

      • Supports streaming responses over gRPC and MP4 Fast Start.

      • Supports deployment on AWS, with S3 URLs accepted as input.

      • Expands codec support to H.265, AV1, VP8 and VP9, with hardware-accelerated decoding through NVDEC.

      • Expands GPU support to NVIDIA A100, H100, H200 and B200 with no NVENC requirement. RTX PRO 6000 and RTX 5090 Blackwell are also supported.

      • Supports variable-frame-rate input.

      • Maintains API compatibility with the V1 NIM.

  • LipSync

    • Adopts the latest SDK quality improvements.

    • Now includes a Hindi model.

    • Now includes a key/fill feature.

  • Active Speaker Detection

    • Updated model with voice activity detection improvements.

  • 3D Body Pose

    • Adopts the latest SDK quality improvements.

  • Video Super Resolution

    • gRPC now availablelive. SMPTE ST 2110 variant delayed on cluster availability.

  • Background Noise Removal

    • Now available with SMPTE ST 2110 support.

    • NIM built on NFP3

Packaging

  • Windows on ARM SDK parity. At launch:

    • Augmented Reality (AR) SDK — Face Detection, Landmark Detection, Eye Contact, Face Expressions, Body Pose.

    • Video Effects SDK — AI Green Screen, Relighting, Video Super Resolution, Video Frame Generation, Video Denoising, TrueHDR.

    • Audio Effects SDK — Denoiser, Dereverb, Studio Voice Low Latency, Audio Super Resolution, Speaker Focus, Studio Voice High Quality, Studio Voice Mic Profiles.

  • Python Wheel 2.0.0

    • Video Super Resolution, Video Frame Generation and TrueHDR.

  • Sports Intelligence Playbooks

Deprecated

  • Acoustic Echo Cancellation is no longer available.

  • Voice Font is no longer available.

Explore the complete release and get started with NVIDIA AI for Media.

You can also try NVIDIA AI for Media NIM microservices in the API catalog.