WPF Video Playback Problems

There is one severe problem with video playback in WPF/Silverlight that, for some reasons unknown to me, is getting very little (almost none) attention from both the community and Microsoft. Basically, it boils down to WPF being absolutely unable to properly play video content. Let me explain by using the following diagram:

WPF Video Playback Poblems

First line shows video frames as they are in the video content. Second line depicts variable frame rate of the WPF renderer – as you know, WPF is changing the frame rate dynamically, and CPU load will affect frame rate too. There are two possible outcomes:

  1. WPF is using vsync (version 1), i.e. it synchronizing rendering to the screen with the screen refresh frequency. This is what you will get on Vista, and sometimes on XP (normally XP video drivers are not using vsync). The resulting video shown on the screen will have variable duration of frames, and the viewer will be present with artifacts like skipped video frames, “jerky” playback, where motion is not “smooth”/uniform, etc.
  2. WPF is not using vsync (version 2). It is the case normally seen on XP. The resulting video is even worse, as there is the tearing effect present now – upper and lower parts of the displayed video frame are not matching. This problem is not present on some XP machines, e.g. I do not see it on my home PC, while it is present on 100% of other machines.

This is a bit oversimplified picture, but good enough for our purpose. In reality, the resulting on-screen picture will depend also on the screen refresh frequency, but artifacts will be more or less the same. In both cases the video playback it totally unacceptable though.

The issue is very serious, and, unfortunately, it is unsolvable in terms of today’s WPF. I think it will be not solved in the near future either, as solving it would require WPF to adopt constant-frame rate rendering paradigm, and synchronizing it with the display refresh rate. This is not likely to happen, as it is a drastic change to the fundamentals of WPF, and, at the same time, it will waste resources in non-video related applications (now WPF can decide what frame rate it will use, and can dynamically change it, e.g. dropping frame rate for “still” scenes and increasing it when animations start).

In our software we had to go through a lot of extra pain to “overcome” this issue – we render video using DirectShow by incorporating Win32 host into our WPF application and implement all kinds of “tricks” to allow WPF content to get into the video window airspace (to display OSD’s, menus, etc.) by using e.g. clipping regions (say Goodbye! to transparency though). There are a few third party projects out there (WPF MediaKit by Jeremiah Morrill is an excellent example) that allow one to mix DirectShow and WPF, but none of them will fix the above mentioned issues, as the resulting picture is rendered by WPF renderer and all the artifacts will be there still.

Probably, for majority of people this is not really an issue (at least until you point them to it; after that they can see it on their own and they become very annoyed), especially if you are playing some kind of home-made content from the YouTube (although YouTube has HD content nowadays too), but it is completely unacceptable for people at least somehow concerned about the video playback quality. DVD/DivX player costing $30 will not have these problems, so it is difficult to explain why $1000 machine (33 times more expensive!!!) cannot produce even remotely as good video playback on $2000 HD TV set.

In addition to these video playback problems, WPF on XP suffers from the tearing issue not only during video playback, but also during “normal” WPF animations. And this has even more profound effect, as it makes it impossible to implement animated WPF applications on XP.

Microsoft’s reaction to the tearing issue on XP is problematic. We have tried to approach them via all possible channels – Microsoft Connect program, Microsoft Metro program, newsgroups, personal contacts, everything. They either deny the issue, or say that it is unsolvable for XP in general case, so they are not going to solve it (but they could easily solve it for something like 95% of all XP installations by solving it at least for ATI and Nvidia cards), or they say that they will not solve it for XP anymore, as XP is not available so nobody cares. And this last reaction is problematic again, as the issue is present not only on Windows XP Home/Pro, but on Windows XP Embedded and Windows Embedded Standard as well. These OS’s will be around for some time still, and Windows 7 Embedded will not come before 2010 and, being Vista-based, will be a resource hog – something not necessarily good for embedded system.

About these ads

27 thoughts on “WPF Video Playback Problems

  1. Hi,

    I’m now using HwndHost interoperating method you mentioned, to developing my TV app. All the TV part are directshow under win32, and my UI is WPF. But now I met a problem that I cannot display any WPF control over my TV video, it seems that the TV video (ie. hwndhost) is always on top. Is this also a known issue and still a limitation? Or could you help to provide some solution? Thank you so much. Any suggestion will be appreciated.

    Best Regards,
    Virginie

  2. Hi Virginie,

    Yes, you are right – HwndHosts are showing on top of WPF controls. That’s how it is designed, and I am not sure if they will fix it in .NET 4.

    The only reasonable way we found to at least somehow mitigate the issue is to use clipping regions: you can define a region, and set it for the video window by using SetWindowRgn Win32 API function, and it would “cut out” holes in the video, so that you can see the underlying WPF content. The approach sucks, as you cannot get nice things like partial transparency, but at least it allows you to “mix” WPF and HWND (though fully opaque). Additionaly, it is a bit complicated to do animations with regions, e.g. slide the menu in or something like that, at least we had some difficulties with this part (although we did not try hard enough perhaps) – they are “jerky”, “jumpy”, and not always end in the point they should.

    I hope this helps…

  3. Hi Roman,

    Actually I tried another way before, I made the OSD menu as another independent WPF app, but still fail, they will flash over tv video…

    I will try the “clipping” way later, Thanks a lot.

  4. Just to save your time – the same flickering will happen in case of layered windows (at least when windows become bigger, e.g. bigger than half of the screen), despite all the Microsoft’s noise about “fixing and improving” them.

    If it would be possible to get video frames to two destinations (HWND and in-memory DC) then there would be one more solution of getting OSD over the video with transparency, but, unfortunately, it is not really possible to render the same video to two places simultaneously in general case, at least not without writing own renderer, and even then you will have to say goodby to DXVA.

  5. Came up against this exact same issue in my tv app (http://pvr.sichbo.ca) and have been bitching about it ever since WPF 1.0. I had gone through various hacks over the years, but ended up dropping embedded WPF video and just cropped a hwnd like you’ve done for OSD.

    Apparently for MediaElement, MS do an internal rapid update call somewhere in the wpfgfx_v0XXX.dlls to achieve a smoother framerate in their EVR custom presenter. Like you say though, the variable timings caused by vsync still result in dropped frames.

    It’s my opinion that HD playback with EVR on XP systems is forever doomed. If you’ve ever looked at writing a MediaFoundation presenter, you basically create multiple surface and roll them in a ring-buffer type of affair. Net result is significantly more video memory used (wasted) with an barely noticeable framerate improvement over VMR7. Try 1080i mpeg2 content with hardware accelerated decoder in vmr7/9 and you get ~20% CPU. Same decoder running through EVR peggs the CPU ~99% — why, I have no idea, in theory it should just waste memory, but somehow acceleration is lost on it too. This also obviously cripples the WPF MediaElement for HD on XP. This is what got me started on writing a WPF/VMR9 presenter, the need to bypass the EVR rubbish. However D3DImage, the only way to inject a surface into WPF, also suffers from cripplingly slow updates. You can hand VMR9 the same surface that you give D3DImage (no stretch rect required), but the Lock/Unlock texture update is woefully inadequate. Undocumented direct calls to wpfgfx_v0300’s AddDirtyRect doesn’t help either.

    Another path was to make a rendertarget bitmap of your OSD (usually a slow 1fps frame rate there, right? not too much animation going on..) in WPF, throw together a source filter with ARGB32 media type output (all this can be done in C# easily), connect that to the second input of your VMR7/9 and use SetVideoPosition() to place your over the video in 1:1 ratio. If you don’t need to stretch your background video’s aspect ratio, this works. But if you do not want black bars on your TV, you have to fight the VMR’s unconscionable output rectangle calculations. It seems to take the largest source (usually your WPF OSD bitmap at y’know 1034×791 or some random thing) and adjust the output of the smallest source, based on some weird rectangle calc I could never figure out. The net effect is an OSD which never quite overlays 1:1, as to prevent blurring.. or it works on “some” WPF window sizes, but others result in unpredictable output.

    .NET 4.0 beta1 released today. Doesn’t look like we have a solution coming on the horizon.

  6. My company is using WPF to play videos and we’ve seen the same issues. For videos of small sizes, the performance is acceptable. For full screen videos, we pop-up a separate window and use direct show, with great performance. But for mid-sized videos, we’d like to continue to use WPF to play videos. For some videos, it works fine, but others have “jerky” playback.

    My question is: what can I do to optimize the playback of videos using WPF? We have control over the frame rate of the videos and the video encoder / file formats. Is there a way to adjust the video frame rate to minimize the jerky playback (our monitors runs at 60hz)? Also, is there anyway to “force” WPF to refresh more often, even if not needed? The hardware we run have quad-core CPUs, so we’re not concerned about wasting CPU (we just want quality playback).

    Thanks for the help / great blog…

  7. Hi AllureEE,

    I am afraid you cannot do anything in this case, as WPF rendering engine is non-deterministic in terms of framerate. You can look into the DesiredFrameRate attached property. The issue – it is not a guaranty, but rather a guideline, so, likely, it won’t help you much in case of video playback.

    One thing that might help somehow (I am not saying that it will, but it might) is to make your video framerates integer divisors of 60Hz, i.e. 10fps, 15fps, 20fps, or 30fps. But again – it might work for your case, or might not… try it, and see it.

    Video codec (or decoder in fact) also has great impact on jerkiness of the playback, so you might want to experiment with various video codecs to see what works for you better.

  8. I have run into the same problems while developing a WPF app that needs to scroll text smoothly. The text is jumpy and I have tried many different methods but nothing at all seems to eliminate this jumpiness. I am VERY dissappointed with Microsoft for completely ignoring this major requirement in their design phase for the WPF architecture. The fact that they are ignoring all feedback on this issue and denying the problem is making me have serious doubts about the integrity of the leadership on the WPF team.

    I am going to have to take the same path as everyone else and use Win32 to display the scrolling text. What a pain!!!

    Jeremy Walker
    Sr. Software Engineer
    Vancouver, BC

  9. Jeremy,

    “[...] they are ignoring all feedback on this issue and denying the problem is making me have serious doubts about the integrity of the leadership on the WPF team.”

    Well, I think they do not have any leadership there at all for quite a long time already, as all relevant people are long gone not only from the WPF team, but also from Microsoft. Which IMHO shows that there were severe problems during the development phase, either on interpersonal level (which I doubt), or between marketing and R&D (which is more likely).

    In fact, I started to get serious doubts about the WPF’s future in general. Microsoft is pushing Silverlight heavily, and, on the other hand, all their “minimalistic” effort to do some product with WPF are so far either not visible to the public, or get very-very bad reviews from users and critics (Visual Studio 2010 anyone?). And then there is plenty of information about the .NET 4 and all its “cool” things like C# with support of VARIANT (or is it nowadays called “dynamic”?), WCF (OMFG! puke!), and others, but practically nothing about WPF improvements, except about some minor bug-fixes and text rendering improvements.

    Very sad story…

  10. Hello Roman, all,

    Thanks for the good post.

    I too am puzzled about the lack of attention on “the web” regarding this issue. The application my company is developing suffers from the same problems as you have described and we will probably have to walk down the same DirectShow/Win32 road as you have (in fact we have tearing problems as well as problems with dropped frames (running on Vista)).

    I am curious though, in your application did you manage to get rid of the tearing problems completely? As you all know the tearing problem goes beyond WPF and I am still confused about the inconsistency between different software and hardware when it comes to tearing.

    – Kristofer.

  11. @Kristofer

    If you have a complex WPF scene with video actually utilized on brushes, you’re out of luck. My final solution (not yet live, still tweaking) is to render D3D scene with custom VMR9 and EVR Presenters (always using vmr9 for XP for DVXA1/HD support, and evr on vista+ for DXVA2), and then use RenderTargetBitmap to grab the WPF scene, copy to d3d surface and then overlay ontop of full frame-rate video. Effectively, instead of trying to host video in WPF we must host WPF in the video. I only get away with this because it’s for a simple OSD overlay which updates only every 200ms. You then translate mouse events which hit the Hwnd airspace, and divert click events to the underlying WPF application so that buttons/trackbar etc still work.

    Performance seems fine on both modern and old hardware with this setup.

    The current problem for me is I’ve implemented EVR presenter in C#, which renders correctly, but CPU is a lot higher than I want it to be for HD h264 content. Sitting ~70% instead of ~10% where it should be. I think invoking the IMFTransform ProcessOut() through a C# delegate (GetDelegateForFunctionPointer) is the cause, since it gets called so rapidly. I can set D3D present interval to “ONE” and the CPU problem is somewhat relieved, but playback becomes choppy.

    – Si

  12. Hey guys,

    Just a quick follow up. I really didn’t want to use Win32 C++ when my application is written in C#. I investigated the possibility of using XNA (C# windows platform for games) instead. As it turned out, this worked out beautifully for me.

    To reiterate, I am not rendering video but I am rendering smooth scrolling text where tearing and frame drops was a huge problem in WPF. With XNA I had none of these problems and I get very, very fast and smooth rendering of the text. I didn’t have to fuss with C++/C# integration either.

    Jeremy

  13. Kristofer,

    “[...] I am curious though, in your application did you manage to get rid of the tearing problems completely? [...]”

    No, not really. We are going to move to Windows Embedded Standard 7 and .NET 4 – this will fix WPF tearing as far as I know, but it won’t fix video playback issues (as I noted earlier, there is a conflict between video rendering and scene rendering expectations/demands).

    Regarding the video tearing issue, we use DirectShow for playback, and then it depends on what video hardware you have, what video renderer you use, and in what mode it is. For example, in our system we have to use windowed mode of VMR7 to get tearing-free video playback. All other modes of VMR7, VMR9 and EVR are tearing – full stop. Very-very-very sad story.

  14. Sichbo,

    “[...] The current problem for me is I’ve implemented EVR presenter in C#, which renders correctly, but CPU is a lot higher than I want it to be for HD h264 content. Sitting ~70% instead of ~10% where it should be. [...]”

    I think the issue here is that you are not supporting DXVA (and you are not, I am sure), so you will never reach 10% you wish. My unfortunate feeling is that you will be not able to improve your solution too much…

  15. Jeremy Walker,

    “[...] With XNA I had none of these problems and I get very, very fast and smooth rendering of the text. I didn’t have to fuss with C++/C# integration either. [...]”

    Do you mix XNA and WPF? Or you just implemented the whole app in XNA?

  16. Hey Roman,

    Yep I figured it out in the end just a couple nights ago. I now have ~8% CPU playing HD h264 with WPF overlayed – full framerate plaback.

    My problem was I was following MSDN’s EVR presenter example which I assumed was a comprehensive and a full EVR implementation. The example doesn’t actually pass back the IDirect3dDeviceManager9 during IMFGetService.GetService(..) when requested the MR_VIDEO_ACCELERATION_SERVICE. This is the key. And it’s why as you correctly pointed out, DXVA was not in use… all along I was convinced DXVA was being utilized but my direct3d or interop implementation was somehow wrong.

    All in all, it turns out to be very simple to implement EVR presenter in C#, and with a lot less code than others I’ve seen online. They make it more complex that it needs to be in order to demonstrate the full API feature set I guess.

  17. Roman,

    Yes, I am mixing XNA and WPF. You can do this with XNA rendering to its own window, or with a bit more effort you can get XNA to render to a panel in the WPF app.

    XNA is meant to be a gaming platform, where tearing is unacceptable and smooth animation without frame drops is critical. WPF was never engineered to be rigirous enough for gaming requirements, and why would Microsoft bother? They would just tell you to use XNA.

    Jeremy

  18. Excellent article. Too bad I didn t read it before. Are the things changing today ? Did you find workarounds?

  19. Hi Mike,

    No, it is all same in WPF world :) . .NET 4 did not improve video playback, and, as I wrote earlier, WPF will probably never have it fixed because of properties of the rendering engine. So, DirectShow (or MF) is still the only way to go for high quality video playback. On the other hand, UI tearing is not present at Windows 7 anymore, so at least that part became better.

  20. Hi Roman
    thanks for the follow up.
    I guess I will have to keep a good mixture of dshow for playback video content and to keep good synchronization and perfect transition. As right now testing all the paralleltimeline with mediaelement in wpf could not challenge the quality I got with dshow. However I was wondering what would be the best way to also playback flash content or webpages together with dshow and came out to use hosted activex together with directshow implementation. But what would be best using standard c# winform to do this mixture or using wpf ?
    Thanks again for your involvment in this topic

  21. Hi Mike,
    Well, it depends on what you want to reach when you talk about video and flash integration in your app. Personally, I would use WPF, as it is “cooler” :) , but then again it is my personal view. For many applications WinForms are absolutely appropriate, and using WPF will not improve anything. It is easier to host HWNDs in WinForms, as WinForms are windows-based, and WPF is not, but it is not too difficult to do it in WPF either, so all arguments will likely boil down to personal feelings and opinions. To summarize: if you do not have any specific WPF-related requirements (e.g. animations, transitions, vector-based scalable UI, etc.), all other conditions being the same, then select the technology based on your familiarity with it. I hope it answers your question.

  22. Thanks again for your advice Roman
    I was afraid of performance issue (and/or space issues) when calling from single wpf both directshow and hosted flash activex. But if it is not so then I guess we ll go for wpf.

  23. Well, mixing (I mean real mixing here – not hosting) non-WPF (DirectShow and ActiveX/Flash) and WPF content is not easy – almost impossible. There are few third party solutions that might be of interest for you, e.g. Jeremiah Morrill’s WPF MediaKit (there was also a project by Jeremiah and some other guy where they were hosting/mixing any HWND, e.g. browser in WPF by intercepting all the rendering; check http://jmorrill.hjtcentral.com/ for more info on all these). But then again, if you go this route you will not get high quality WPF playback, as it is WPF who will render the video to the screen (and we are back to my original post then). On the other hand, these solutions might be OK for Flash. By the way, depending on what you want from the video playback, WPF’s native MediaElement might even satisfy your demands – its quality is not good for high-end scenarios I need to target, but then again such quality might be more than adequate in many “PC” usage scenarios, as PCs are somewhat inherently “retarded” in this respect :) . For example, if you are playing 25 fps video on 60Hz screen, no matter how and what you try you will never get good quality video playback, as it ALWAYS will be jerky, so WPF cannot make situation much worse.

  24. Hi Roman
    Thanks again for your contribution through this interesting blog.
    After making some sample apps it turns out to be that directshow are really the solution for best optimization both of transition and preload. It seems that whatever you tend to do with wpf mediaelement you will always get the last image of the last video before new playback and the only possibility would be to hide it with a black bmp at transition time. Except there s some other techniques that I m not aware of yet. Besides and it might be out of this topic, but I would appreciate your opinion on this too; in fact discovering the advantage of dxva mode both through directshow and mediafoindation i was positively surprised by the low CPU it was requiring to play full HD also on Atom based proc with good nvidia gpu. However I wanted to add a transparent window(ie: red circle made with wpf) over the playbackvideo and discover that suddenly the video is getting choppy although CPU is still very low (<30%). and this happens only when you drag the transparent window over the playbackvideo. Could you think of what could interfere (some kind of bottleneck)? I v tried to use directcompute to make this transparent window but still could not avoid the random choppy playback. Thanks for your help.

  25. Hey Mike.

    I’m getting the same issue… as soon as I place transparent content over video in WPF, the video becomes choppy.

    Did you manage to find a solution?

  26. Hi Mike, B,

    (Mike, I somehow did not notice your post earlier…)

    Unfortunately, there is no real solution with transparent windows. I am doing video playback in WPF like this: hosting a HWND in WPF window, and “cutting” holes in the video window e.g. by using SetWindowRgn. Transparency is gone, of course, but there can be at least WPF content “overlapping” (well, kind of) with the video – you can show OSDs, pop-up, or whatever else.

    By the way – you will likely see the problem you are describing with some other video player programs as well, when e.g. Skype is showing its notification windows in the corner – video playback will slow down badly while the pop-up is visible.

    Hope this helps.

  27. Pingback: Troubleshooting Video Playing in WPF | Mr Pfisters Random Waffle

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s