WPF Video Playback Problems

There is one severe problem with video playback in WPF/Silverlight that, for some reasons unknown to me, is getting very little (almost none) attention from both the community and Microsoft. Basically, it boils down to WPF being absolutely unable to properly play video content. Let me explain by using the following diagram:

WPF Video Playback Poblems

First line shows video frames as they are in the video content. Second line depicts variable frame rate of the WPF renderer – as you know, WPF is changing the frame rate dynamically, and CPU load will affect frame rate too. There are two possible outcomes:

  1. WPF is using vsync (version 1), i.e. it synchronizing rendering to the screen with the screen refresh frequency. This is what you will get on Vista, and sometimes on XP (normally XP video drivers are not using vsync). The resulting video shown on the screen will have variable duration of frames, and the viewer will be present with artifacts like skipped video frames, “jerky” playback, where motion is not “smooth”/uniform, etc.
  2. WPF is not using vsync (version 2). It is the case normally seen on XP. The resulting video is even worse, as there is the tearing effect present now – upper and lower parts of the displayed video frame are not matching. This problem is not present on some XP machines, e.g. I do not see it on my home PC, while it is present on 100% of other machines.

This is a bit oversimplified picture, but good enough for our purpose. In reality, the resulting on-screen picture will depend also on the screen refresh frequency, but artifacts will be more or less the same. In both cases the video playback it totally unacceptable though.

The issue is very serious, and, unfortunately, it is unsolvable in terms of today’s WPF. I think it will be not solved in the near future either, as solving it would require WPF to adopt constant-frame rate rendering paradigm, and synchronizing it with the display refresh rate. This is not likely to happen, as it is a drastic change to the fundamentals of WPF, and, at the same time, it will waste resources in non-video related applications (now WPF can decide what frame rate it will use, and can dynamically change it, e.g. dropping frame rate for “still” scenes and increasing it when animations start).

In our software we had to go through a lot of extra pain to “overcome” this issue – we render video using DirectShow by incorporating Win32 host into our WPF application and implement all kinds of “tricks” to allow WPF content to get into the video window airspace (to display OSD’s, menus, etc.) by using e.g. clipping regions (say Goodbye! to transparency though). There are a few third party projects out there (WPF MediaKit by Jeremiah Morrill is an excellent example) that allow one to mix DirectShow and WPF, but none of them will fix the above mentioned issues, as the resulting picture is rendered by WPF renderer and all the artifacts will be there still.

Probably, for majority of people this is not really an issue (at least until you point them to it; after that they can see it on their own and they become very annoyed), especially if you are playing some kind of home-made content from the YouTube (although YouTube has HD content nowadays too), but it is completely unacceptable for people at least somehow concerned about the video playback quality. DVD/DivX player costing $30 will not have these problems, so it is difficult to explain why $1000 machine (33 times more expensive!!!) cannot produce even remotely as good video playback on $2000 HD TV set.

In addition to these video playback problems, WPF on XP suffers from the tearing issue not only during video playback, but also during “normal” WPF animations. And this has even more profound effect, as it makes it impossible to implement animated WPF applications on XP.

Microsoft’s reaction to the tearing issue on XP is problematic. We have tried to approach them via all possible channels – Microsoft Connect program, Microsoft Metro program, newsgroups, personal contacts, everything. They either deny the issue, or say that it is unsolvable for XP in general case, so they are not going to solve it (but they could easily solve it for something like 95% of all XP installations by solving it at least for ATI and Nvidia cards), or they say that they will not solve it for XP anymore, as XP is not available so nobody cares. And this last reaction is problematic again, as the issue is present not only on Windows XP Home/Pro, but on Windows XP Embedded and Windows Embedded Standard as well. These OS’s will be around for some time still, and Windows 7 Embedded will not come before 2010 and, being Vista-based, will be a resource hog – something not necessarily good for embedded system.

Webcam Control with WPF

Webcam control with WPF or how to create high framerate player with DirectShow by using InteropBitmap in WPF application by Tamir Khason shows how to do exaclty what it claims to show :) in a way alternative to what Jeremiah Morrill is proposing (now, I do not have an opinion as of which is better or worse – they both have their uses).

Side Note Unfortunately, you cannot play decent quality video in WPF application (meaning in WPF “airspace”) no matter what you do – it has to do with non-constantness of the WPF rendering frame rate (and, on XP, add here WPF tearing artefacts), and I do not see how it can be fixed in the future as this problem is rooted extremely deeply in the core of WPF rendering pipeline. Microsoft knows about this issue, but does nothing to fix it.

DirectShow Resources

There is no need to say anything about any of the following:

  • Geraint Davies Consulting Ltd or GDCL – the home of Geraint Davies, the author of DirectShow, and famous GMFBridge. The site has a lot of interesting information about DirectShow, e.g. introduction to filters, technical Q&A, as well as some pretty elaborative samples like demuxers
  • The March Hare – the guy is always present at DirectShow forums :) . Filter Graph Library, despite being written by another guy can be downloaded from this site as well
  • Alessandro Angeli’s Programming FAQ – a lot of useful DirectShow tips and tricks
  • GraphEditPlus – is GraphEdit done right :) . Especially nice time-saving feature is its graph code generation – you build the graph and GraphEditPlus generates its C++ creation code

MSDN Forums

Vista Software Development MSDN Forums

Here are WPF-related forums:

And these forums are not related to WPF but I need them anyway: