MLVC: Microsoft's Open-Source AI Video Codec That Could Redefine Video Compression

 

MLVC: Microsoft's Open-Source AI Video Codec That Could Redefine Video Compression

Published: August 2026

Introduction

Video is everywhere—from video conferencing and online education to streaming services and social media. Every second, enormous amounts of video data travel across the internet, making efficient compression more important than ever.

For decades, traditional codecs such as H.264, H.265 (HEVC), and AV1 have powered this ecosystem. While these codecs have evolved significantly, each new generation demands years of engineering, hardware acceleration, and standardization before becoming widely available.

Microsoft is taking a different approach.

Recently, Microsoft open-sourced MLVC (Machine Learning Video Codec), a neural-network-based video codec designed for real-world deployment. Instead of relying on handcrafted compression algorithms, MLVC uses deep learning to compress video more efficiently while taking advantage of modern AI hardware such as NPUs (Neural Processing Units). The project is released under the MIT License, allowing researchers and developers to explore, improve, and integrate the technology into their own applications.



What Is MLVC?

MLVC is Microsoft's production-ready implementation of a learned (AI-powered) video codec.

Unlike conventional codecs that rely on manually engineered components like:

  • Motion estimation

  • Transform coding

  • Quantization

  • Entropy coding

MLVC replaces much of this pipeline with deep neural networks trained end-to-end to optimize video quality while minimizing bitrate.

Rather than asking engineers to design every compression rule manually, the neural network learns the most efficient representation directly from training data.

The result is better compression efficiency, particularly in scenarios such as video conferencing where bandwidth is limited but visual quality is critical.

Why Traditional Codecs Are Reaching Their Limits

Traditional codecs have served the industry exceptionally well.

However, they also have several limitations:

  • Every new codec generation requires years of standardization.

  • Specialized hardware support often takes years to become mainstream.

  • Improvements between generations become increasingly incremental.

  • Engineering complexity continues to grow.

Machine learning offers an alternative.

Instead of manually designing compression algorithms, researchers can train neural networks to discover better compression strategies automatically.

MLVC represents Microsoft's latest effort to bring this idea into production rather than keeping it as an academic experiment.

The Biggest Challenge: Cross-Platform Compatibility

One of the major reasons learned video codecs have struggled outside research labs is cross-platform reliability.

Imagine encoding a video on:

  • Apple Neural Engine

  • Intel NPU

  • Qualcomm AI accelerator

Even tiny numerical differences between these processors can cause encoder and decoder states to diverge.

Traditional neural codecs often require bit-exact computation.

If two devices compute slightly different values, the decoder may completely fail.

MLVC solves this problem through a novel architecture that explicitly transmits entropy-model scale parameters using its hyperprior, making decoding robust across different hardware platforms.

This is one of the project's most significant engineering achievements because it enables real-world deployment across heterogeneous devices instead of requiring identical hardware.

Real-Time Performance

Performance has always been another obstacle for neural codecs.

According to Microsoft's published research, MLVC can achieve:

  • Around 100 FPS

  • Running on commodity NPUs

  • Across Apple, Intel, and Qualcomm platforms

This makes real-time applications such as:

  • Microsoft Teams

  • Video conferencing

  • Remote collaboration

  • Live streaming

much more practical than earlier generations of neural codecs.

What Does the Open-Source Release Include?

Microsoft's GitHub repository contains far more than just inference code.

Developers receive access to:

  • Complete MLVC neural network architecture

  • Pre-trained model weights

  • Training scripts

  • Documentation for dataset preparation

  • Platform conversion scripts for different NPUs

  • MIT License for open collaboration

Microsoft has also announced that a future release will include a C++ codec library to simplify integration into production software.

Relationship with DCVC

MLVC did not appear overnight.

It is the latest evolution of Microsoft's Deep Contextual Video Compression (DCVC) research series that has been developed over several years.

The earlier DCVC family demonstrated impressive compression efficiency in research settings.

MLVC builds upon that work by emphasizing:

  • Cross-platform robustness

  • Production readiness

  • Real-time execution

  • Practical deployment

This shift marks the transition from research prototype to engineering product.

Why This Matters

Video traffic already accounts for a large portion of global internet bandwidth.

Even modest improvements in compression efficiency can translate into:

  • Lower cloud infrastructure costs

  • Reduced bandwidth usage

  • Better quality over poor networks

  • Lower latency during video calls

  • Improved battery efficiency when paired with dedicated AI hardware

As NPUs become standard components in laptops, smartphones, and edge devices, learned codecs become increasingly practical.

Rather than waiting years for specialized codec hardware, MLVC leverages AI accelerators that are already shipping in modern devices.

Potential Applications

MLVC has broad potential across multiple domains:

Video Conferencing

Lower bandwidth while maintaining higher visual quality.

Cloud Streaming

Reduce CDN costs without sacrificing user experience.

Remote Work

Improve call quality on unstable or low-speed connections.

Edge AI Devices

Run efficient video compression directly on-device using NPUs.

Future Media Platforms

Enable AI-native video formats optimized for modern hardware.

Looking Ahead

Microsoft's roadmap includes several milestones:

  • Stable real-time 540p performance

  • Broader hardware compatibility

  • Improved resilience to transmission errors

  • Support for higher resolutions such as 1080p

  • Expansion into more streaming scenarios

The long-term vision is an open ecosystem where learned video codecs can compete directly with—or even replace—legacy compression technologies where appropriate.

Final Thoughts

MLVC represents more than another open-source AI project.

It signals a shift in how video compression may evolve over the coming decade. By combining machine learning, modern AI hardware, and an open-source development model, Microsoft is challenging decades of traditional codec design.

While conventional codecs like H.264, H.265, and AV1 will remain essential for years due to their widespread adoption and hardware support, MLVC demonstrates that AI-powered codecs are moving beyond research papers into deployable systems.

For developers, researchers, and anyone interested in the future of multimedia, MLVC is a project worth watching. It offers a glimpse into a future where video compression is learned rather than handcrafted—and where AI accelerators become as important to media processing as GPUs have become for graphics.

Project Repository: Microsoft MLVC on GitHub