Tuesday, August 26, 2008

Converting a Partitioned Table to a Nonpartitioned Table In Sql Server 2005

Several months ago while working with Sql Server 2005 partitioned tables for the first time, I discovered an interesting bug/hidden feature that doesn't seem to be documented anywhere: Adding a clustered primary key constraint can quietly revert a partitioned table to a nonpartitioned one. At the time I found this behavior quite annoying, but it actually came in handy today when I needed to change the data type of a column used in the table's partition scheme from smalldatetime to datetime. Microsoft's knowledge base article on modifying partitioned tables indicates only that you may collapse multiple partitions into a single partition. It doesn't provide any options for departitioning tables--other than dropping and recreating them from scratch, of course.

To demonstrate departitioning we must first create a simple partitioned table. To do so execute this Sql:

   CREATE PARTITION FUNCTION MyPartitionRange (INT) 
   AS RANGE LEFT FOR VALUES (1,2) 

   CREATE PARTITION SCHEME MyPartitionScheme AS 
   PARTITION MyPartitionRange 
   ALL TO ([PRIMARY]) 

   CREATE TABLE MyPartitionedTable 
          ( 
          i INT NOT NULL, 
          s CHAR(8000) , 
          PartCol INT 
          ) 
   ON
    MyPartitionScheme (PartCol)     

(If you don't understand what's happening in each of the steps above, read this tutorial for more complete instructions--see "Creating the Partitioned Table"). Execute this Sql to see a list of partitions for the new table (you should see three):

   SELECT *
   FROM sys.partitions
   WHERE OBJECT_ID = OBJECT_ID('MyPartitionedTable')

Now, let's say this table has been running in production for several months, has lots of data, and you realize you need to expand PartCol to a bigint. You can't change the PartCol data type with an alter table statement:

   ALTER TABLE MyPartitionedTable ALTER COLUMN PartCol bigint

The Sql above fails with the somewhat obscure error "The object 'MyPartitionedTable' is dependent on column 'PartCol'." This won't change even if you collapse multiple partitions into a single partition using ALTER PARTITION with the SPLIT and MERGE options (the approach recommended by Microsoft), because the table is still partitioned on PartCol. Instead, you can execute this Sql to departition the table completely:

   ALTER TABLE MyPartitionedTable
   ADD CONSTRAINT [PK_MyPartitionedTable] PRIMARY KEY CLUSTERED([i]) ON [PRIMARY]

You should now see just one partition for this table in sys.partitions:

   SELECT *
   FROM sys.partitions
   WHERE OBJECT_ID = OBJECT_ID('MyPartitionedTable')

Other important points to note:

  1. This only works if the primary key column is not included in your partition function definition.
  2. If you forget to include "ON PRIMARY" when creating the primary key you'll run into another obscure error: "Column 'PartCol' is partitioning column of the index 'PK_MyPartitionedTable'. Partition columns for a unique index must be a subset of the index key." Sql Server is trying to tell you that you can't create a clustered index on "i" because it's not used in your partition function. In other words, you could create a clustered index including both "i" and "PartCol" and still maintain partitioning.
  3. A non-clustered primary key won't departition the table, even if you specify "ON PRIMARY".
  4. Adding any clustered index should accomplish the same thing--but I haven't verified this.

Finally you are free to change the PartCol data type. You can also repartition the table if necessary by following the steps in this article.

Saturday, May 31, 2008

Video Scene Detection with DirectShow.NET

For some time I've been working on a video-related personal project. I'm using the fantastic DirectShow .NET library, which provides a nice C# interface to Microsoft's DirectShow C++ API. At one point some folks on the DS .NET forums asked about the scene detection algorithm I referenced in one of my forum posts. I promised to follow up with some sample code and explanations and--finally--here they are.

I've created a sample solution to demonstrate my scene detection algorithm. It's based on the DxScan sample available with other DS .NET samples on the DS .NET download page. My algorithm is not yet production code but has proven very reliable in my own testing. It is 100% accurate against my test video library, which is 600 minutes of actual sports video with 1,800 scene changes (including both night and daytime events) plus several short test videos created explicitly to strain the algorithm.

At a high level, scene detection involves the following steps:

  1. Randomly select 2,000 of the RGB values composing a single video frame. These are the values on which we'll perform a longitudinal (or cross-frame) analysis to detect scene changes for the entire duration of the video.
  2. Analyze the current frame:
    1. Calculate the average RGB value for the current frame. If the RGB values are unusually low or high we're detecting scenes shot in bright or dim light conditions and will need to raise or lower our scene detection thresholds accordingly.
    2. Perform an XOR diff between the RGB values in the previous and current frames. The XOR diff amplifies minor differences between frames (vs a simple integer difference) which improves detection of scene changes involving similar scenes as well as detection in low-light conditions where we tend to be dealing with lower RGB values.
    3. Calculate the average RGB difference between the current and previous frames. In other words, add up the XOR diff values from step 2.2 and divide by the number of sample frames.
    4. Calculate the change in average RGB difference between the current and previous frames. This is a bit tricky to understand, but it's critical to achieving a high level of accuracy when differentiating between new scenes and random noise (such as high-motion close-ups or quick pans/zooms). If the previous frame's change in average RGB difference is above a defined, positive threshold (normalized for light conditions detected in step 2.1) and the current frame's change in average RGB difference is below a defined, negative threshold, then the previous frame is flagged as a scene change. In simple terms, we're taking advantage of the fact that scene changes nearly always result in a two-frame spike/crash in frame-to-frame differences; while pans, zooms, and high-motion close-ups result in a gradual ramp-up/ramp-down in frame-to-frame differences.
    5. Advance to the next frame and repeat step 2.

I'll try to expand and clarify the above steps when I have time, but for now you'll have to read the code if you need to understand the algorithm in more detail. The only limitations in the current implementation (that I'm aware of) are the following:

  1. Dropped frames are interpreted as scene changes. This issue can be minimized in most applications by choosing a minimum scene duration and discarding new-scene events fired by the SceneDetector inside the minimum-duration window.
  2. Scene transition effects (fades, dissolves, etc.) are not supported and scene changes involving such effects are not detected.

If you encounter any other issues with the algorithm, I'd love the opportunity to see and analyze the video that broke it!

Thursday, February 07, 2008

Unit testing too difficult? Change your design

Lessons and processes from building construction, physical goods manufacturing, and other engineering disciplines are often misapplied to software creation. Nevertheless, these disciplines occasionally provide very useful analogies. One of these is the idea that a given design must accommodate more than just functional and aesthetic needs.

For example, designers of physical goods must consider how much it will cost to actually build what they're designing. The costs to procure materials, tool-up a factory, and train an army of workers are a major portion of the costs to market a physical good. Want to design a car you can sell for $20k? Skip the Italian leather seats. Forget about the brand-new, high-compression engine that would double factory tooling costs. Drop the independent rear-suspension.

One good thing about design-related manufacturing costs is that they are well-understood and naturally visible. They may be miscalculated, but there's little chance they'll be forgotten or ignored. This is not true in the software world. Instead, the cost implications of many design choices are invisible even to the designer—let alone the rest of the organization. One of these is the recurring cost to validate functionality as the software changes. When validation is 100% manual (meaning a person—whether a QA tester or developer—must explicitly execute and review the results of each test case) it becomes extremely expensive (in terms of time as well as money). (Not to mention that top-down, manual validation is simply too inefficient to exercise more than a small fraction of possible code paths and thus will allow many code defects to escape into the wild.)

We need ways to expose design-related validation costs early in the development cycle so they can be properly accounted for when planning and choosing features—and so the design can be altered when necessary to reduce those costs. We also need ways to minimize validation costs so our software can be tested thoroughly and still be profitably sold and supported.

Rigorous automated unit testing helps us both expose design-related validation costs and minimize those costs:

  • It exposes validation costs because we're forced to invest labor in creating automated tests at the time a feature is added or created, incorporating more of the long-term costs of the feature into the initial implementation schedule.
  • It minimizes validation costs because the labor invested in test creation returns benefits each time the software is modified for an indefinite period of time and dramatically reduces total validation costs.

Automated unit testing forces validation costs to become a first-class design consideration: designs that are not unit-testable are failed designs and must be replaced by designs that are unit-testable and still meet functional and performance goals.

Rigorous unit testing should change our mindset toward one of designing for testability. With this mindset we should

  • Focus the same level of energy and creativity on the design of our unit tests as on the components being tested.
  • Take the same care in organizing and maintaining our test code and projects as we take with the rest of our codebase
  • Believe that a software feature is inseparable from its unit tests.
  • Alter our designs as necessary to support unit testing in both personal and automated build environments

Wednesday, September 05, 2007

Open-source .NET HL7 Parser

I recently spent some time searching for a good open-source .NET HL7 parser. The pickings were pretty slim, and the best option appears to be NHapi--which is a fairly new port of the popular Hapi Java HL7 parser. I had to dig surprising deep into the search results to find NHapi, and though the code seems solid, the project is not extremely active. Another big weakness is the complete lack of API documentation and code samples. You can figure out the essentials from the Hapi Javadoc and sample code, but that's a bit of a pain so I've generated MS Help API docs for each HL7 version supported by NHapi. They are available in a single zip file here (80MB).

Monday, November 07, 2005

Buy this book!

I haven't written much here lately, but I wanted to mention that my friend Tom Copeland, founder of the PMD project, has just published his first book! It's called PMD Applied and you can buy a copy here. What most impresses me about Tom is that he could write this book with five small children in the house! I can barely find enough quiet time to blog with just four!

If you care about code quality why aren't you using PMD?!

Wednesday, August 24, 2005

Ant Humor

Was just reading the NAnt docs and ran across something funny: "The name NAnt comes from the fact that this tool is Not Ant." Since the Ant FAQ says that Ant is an acronym for "Another Neat Tool", NAnt is really an acronym for "Not Another Neat Tool". I wonder if they did that on purpose?

Wednesday, August 17, 2005

Running OpenNETCF 1.3 On NET PC

Since I'm porting a complex application framework from NET PC to NET CF I sometimes find it useful when debugging to run our NET CF code on the NET PC framework. This also comes in handy if, for instance, you want to run unit tests against your NET CF code on the desktop, since on-device automated testing is nearly impossible at this point. (There are sometimes major differences between the two implementations so this is not a good long-term strategy.)

The hardest part about this was getting OpenNETCF 1.3 to run on the desktop. Here's why: The OpenNETCF.Configuration.ConfigurationSettings class uses System.IO.Path internally when figuring out where to look for config files. It passes a Pocket PC compliant path string to Path.Combine() at one point, and, since the code is actually running on the NET PC framework, Path.Combine() throws an ArgumentException.

The only solution (short of rebuilding the OpenNETCF library) is to pre-install a different implementation of OpenNETCF.IConfigurationSystem before the default implementation is installed. There's no public API for installing your custom IConfigurationSystem; you must install it via reflection calls which bypass the restrictive access modifiers. (Obviously, this is a major hack not suitable for production code.) Here's an example of such an IConfigurationSystem implementation which simply passes configuration requests through to the standard NET PC ConfigurationSettings class:


using System;
using System.Reflection;
using ONETConfig = OpenNETCF.Configuration;
using MSConfig = System.Configuration;

namespace Test.PCUtil
{

public class PassThruConfigSystem : ONETConfig.IConfigurationSystem
{

public static void InstallForOpenNETCF()
{
Type configSettingsType = typeof (ONETConfig.ConfigurationSettings);

lock (configSettingsType)
{
// setting them again will cause an exception
Object existingConfig = configSettingsType.InvokeMember(
"configSystem",
(BindingFlags.Static |
BindingFlags.NonPublic |
BindingFlags.GetField),
null,
null,
null);

if (existingConfig == null)
{
configSettingsType.InvokeMember(
"SetConfigurationSystem",
(BindingFlags.Static |
BindingFlags.NonPublic |
BindingFlags.InvokeMethod),
null,
null,
new object[] {new PassThruConfigSystem()});
}
}
}

public PassThruConfigSystem()
{
}

public object GetConfig(string configKey)
{
return MSConfig.ConfigurationSettings.GetConfig(configKey);
}

public void Init()
{
}
}
}


In my next post I'll show you how to automatically install the PassThruConfigSystem when running unit tests with NUnit, which presents its own unique challenges!

Saturday, August 13, 2005

OpenNETCF 1.3 Reinstall Bug

OpenNETCF 1.3 has an annoying bug that causes it to be reinstalled by Visual Studio every time you run/debug an application. Here's a workaround to stop this from happening (OpenNETCF 1.4 also supposedly fixes the bug):

  1. Open this file: 'C:\Documents and Settings\[user profile]\Application Data\Microsoft\visualstudio\devices\7.1\conman_ds_package.xsl'. Make sure you open the right one as there may be copies under each user profile on your PC as well as under the 'All Users' profile.
  2. Save a backup copy.
  3. Find the following PACKAGE element: '<PACKAGE Name="OpenNETCF.dll" InvariantName="OpenNETCF.dll" ReadOnly="true" Protected="true" xmlns="">'
  4. Delete the entire PACKAGE element and all of its contents. (It's quite long--about 400 lines.) You may also need to restart Visual Studio if it's open.
This will stop the OpenNETCF automatic install/reinstall for all platforms, but you will need to manually install OpenNETCF the first time on any devices you are working with in the future.

Friday, July 29, 2005

PropertyInfo.GetHashCode() Generates an InvalidCastException on NET CF

One of the first tough bugs I ran across when we started our NET PC to NET CF port turned out to be a bug in the NET CF class libraries themselves! Plopping a big pile of code down on another version of an API and trying to get it to run (vs rewriting for another language) can be maddening because you're totally unfamiliar with the codebase. It simply works here but not over there, so you just dig in somewhere and keep digging until you find what's broken. In this case the culprit was the GetHashCode() methods of several of the System.Reflection.MemberInfo subclasses, specifically FieldInfo, PropertyInfo, and MethodInfo. Our application code used PropertyInfo objects as keys into a hashtable like this:

PropertyInfo[] properties = myType.GetProperties();
Hashtable propertyMap = new Hashtable();
foreach (PropertyInfo propInfo in properties)
{
    if (propInfo.GetCustomAttributes(true).Length > 0)
    {
        propertyMap.Add(propInfo, "Something Important");
    }
}

At line 7 where the Hashtable is populated the code was blowing up with an InvalidCastException. With very little NET experience under my belt at that point it was a bit of a headscratcher. I knew that C# allowed extensible type casting and conversion (one of the areas where it improves on Java). Had our initial port changes broken some magic type conversion code that was called from deep inside the Hashtable class?

Type.GetProperties() actually returns an internal subclass of PropertyInfo called RuntimePropertyInfo. I fired up Reflector and started digging around in the mscorlib dll (version 1.0.5000.0 from NET CF 1.1). I figured the best place to start would be the GetHashCode() and Equals() methods since those are all the Hashtable implementation should be calling. (At that point I hadn't yet figured out that I could expand the non-user code part of the stack trace in the debugger to see where the exception originated--and of course NET CF doesn't include programmatic access to stack traces.) Sure enough there was something suspicious in the RuntimePropertyInfo class:

private IntPtr _pData;
private RuntimeType _pRefClass;

<snip>

public override int GetHashCode()
{
    return (((int) this._pData) + ((int) this._pRefClass));
}

Notice that _pRefClass is cast to an int in GetHashCode() even though it's declared as a RuntimeType! I opened up the NET PC version of mscorlib and the RuntimePropertyInfo code was almost exactly the same--except that _pRefClass was declared as an IntPtr. I was still a little suspicious as I've learned that it's usually a bad idea to blame bugs on library code before you take a hard look at your own. So I ran the simplest snippet of code I could think of to test my theory:

PropertyInfo[] propsInfo = "".GetType().GetProperties();
foreach (PropertyInfo propInfo in propsInfo)
{
    propInfo.GetHashCode();
}

Sure enough, the call to propInfo.GetHashCode() threw an InvalidCastException! But now I was even more puzzled. How could RuntimePropertyInfo have been compiled with an invalid cast from RuntimeType to int? RuntimeType does not define an explicit cast to int; moreover the cast actually does fail at runtime as you would expect. The best answer I've come up with so far is this post which points out that there may be some CLR magic going on with RuntimeType as it also throws a NotSupportedException in its only constructor. That's not a very satisfying explanation, but I'll have to let it stand for now.

Wednesday, July 20, 2005

Porting from NET PC to NET CF

I'm porting a large rich client application framework (175,000+ lines of actual code) from the full .NET framework to the .NET Compact Framework (since I'm constantly having to differentiate between the two platforms I've started referring to them as NET PC and NET CF, respectively).

Here's the situation: The NET PC version of the framework was written by a large development team with no consideration for the NET CF port. Just two of us are porting the framework, and we don't have enough time to rewrite it from scratch, so we've developed some processes that will help us merge-in future features and fixes added by the PC team. To make matters worse the CF and PC versions of the framework must reside in separate version control systems (NET PC in CVS, NET CF in Starteam) so we can't even leverage the branching and merging capabilities of those tools to manage the differences between the two products.

Thirty-to-forty percent of the way through the port, we've run into a number of interesting issues and incompatibilities between NET PC and NET CF which I'll begin writing about in my next post. Besides the obvious limitations of the compact framework (many, many of the NET PC classes and methods simply don't exist on NET CF), there are major performance considerations and internal differences (bugs?) between the two .NET API implementations that cause some very hard-to-find bugs (many, many thanks to Lutz Roeder for his .NET decompiler!) when you simply move code from one platform to the other. The two of us doing the port have many years of Java development experience but had just a few months experience with C# and .NET (PC or CF) on one small mobile application, so it has been a quite rapid learning experience.




 
Header photo courtesy of: http://www.flickr.com/photos/tmartin/ / CC BY-NC 2.0