Forcing Firefox to Cache Secure Silverlight Video Content

The latest GridRoom release moves all video playback to the Web using a Silverlight-based video player. During my testing I had some problems getting Firefox caching to work properly with video content delivered via https.

At first I tried to deliver all video content over http, not https. But since the secure areas of the GridRoom site are accessed via https I was forced by Silverlight security restrictions to also deliver video content via https for now. (The Silverlight MediaElement does not support cross-scheme access--or mixed http/https content delivery).

During my initial testing with https Firefox seemed to cache video intermittently. Sometimes video was loaded from the cache and sometimes it was streamed again from the Web site, even during the space of a few minutes. Firefox provides an easy way to inspect cache contents using the "about:cache" URL, and after some poking around I noticed that https-delivered video was only held in the memory cache, while http-delivered video was stored in the disk cache. Caching appeared intermittent because my large video files were randomly evicting each other from the relatively small memory cache as I clicked around.

By default Firefox 3 does not use the disk cache for content delivered over https. Users may force this content to be disk-cached by setting the hidden browser.cache.disk_cache_ssl option to true, but this isn't something developers can depend on. The solution is to set a "Cache-Control: public" header for whatever content you want to be disk-cached. This makes Firefox behave like Internet Explorer, which disk-caches https-delivered content by default.

Be careful when using this workaround. The "Cache-Control: public" header tells any caching proxy servers between your server and the client that it's OK to cache your data for use by anyone!

When Competitors Are Using Your Code...Does That Mean You're Being Too Transparent?

Openness and transparency are all the rage in startups these days. Don't hide the fact you're small. Be accessible to customers. Blog about your ideas and plans. Openly dissect your successes and failures. Share your work.

About 2 years ago I posted a video scene detection algorithm written in C# on this blog, while I was still just tinkering with the ideas behind GridRoom (and before I had decided to make a business of it).

Last night I was contacted by a developer who works for Agile Sports, makers of a high-end competitor to GridRoom. Their product is used by a number of NFL and Division I college football teams. The bottom end of their market is probably the largest high schools, while that's the top end of my intended market for GridRoom.

Apparently they've been using my scene-detection code with great success, but he had a couple of questions about how it worked. I'm pretty sure he didn't realized he was asking a competitor for help.

Video Scene Detection with DirectShow.NET

For some time I've been working on a video-related personal project. I'm using the fantastic DirectShow .NET library, which provides a nice C# interface to Microsoft's DirectShow C++ API. At one point some folks on the DS .NET forums asked about the scene detection algorithm I referenced in one of my forum posts. I promised to follow up with some sample code and explanations and--finally--here they are.

I've created a sample solution to demonstrate my scene detection algorithm. It's based on the DxScan sample available with other DS .NET samples on the DS .NET download page. My algorithm is not yet production code but has proven very reliable in my own testing. It is 100% accurate against my test video library, which is 600 minutes of actual sports video with 1,800 scene changes (including both night and daytime events) plus several short test videos created explicitly to strain the algorithm.

At a high level, scene detection involves the following steps:

1. Randomly select 2,000 of the RGB values composing a single video frame. These are the values on which we'll perform a longitudinal (or cross-frame) analysis to detect scene changes for the entire duration of the video.
2. Analyze the current frame:
   1. Calculate the average RGB value for the current frame. If the RGB values are unusually low or high we're detecting scenes shot in bright or dim light conditions and will need to raise or lower our scene detection thresholds accordingly.
   2. Perform an XOR diff between the RGB values in the previous and current frames. The XOR diff amplifies minor differences between frames (vs a simple integer difference) which improves detection of scene changes involving similar scenes as well as detection in low-light conditions where we tend to be dealing with lower RGB values.
   3. Calculate the average RGB difference between the current and previous frames. In other words, add up the XOR diff values from step 2.2 and divide by the number of sample frames.
   4. Calculate the change in average RGB difference between the current and previous frames. This is a bit tricky to understand, but it's critical to achieving a high level of accuracy when differentiating between new scenes and random noise (such as high-motion close-ups or quick pans/zooms). If the previous frame's change in average RGB difference is above a defined, positive threshold (normalized for light conditions detected in step 2.1) and the current frame's change in average RGB difference is below a defined, negative threshold, then the previous frame is flagged as a scene change. In simple terms, we're taking advantage of the fact that scene changes nearly always result in a two-frame spike/crash in frame-to-frame differences; while pans, zooms, and high-motion close-ups result in a gradual ramp-up/ramp-down in frame-to-frame differences.
   5. Advance to the next frame and repeat step 2.

I'll try to expand and clarify the above steps when I have time, but for now you'll have to read the code if you need to understand the algorithm in more detail. The only limitations in the current implementation (that I'm aware of) are the following:

1. Dropped frames are interpreted as scene changes. This issue can be minimized in most applications by choosing a minimum scene duration and discarding new-scene events fired by the SceneDetector inside the minimum-duration window.
2. Scene transition effects (fades, dissolves, etc.) are not supported and scene changes involving such effects are not detected.

If you encounter any other issues with the algorithm, I'd love the opportunity to see and analyze the video that broke it!