I recently moved the SOSEX project to Visual Studio 2012, and I forgot that the VC11 compiler won’t target Windows XP/2003.  I made a special build of the latest update and added additional links for the Windows XP and Windows Server 2003 builds.  This will likely be the last update that supports Windows XP and Windows Server 2003.

I’ve just posted an update that fixes a problem resolving breakpoints for !mbp and !mbm.  The problem occurs when there are multiple app domains and the debugger is attached after the app domains have been created.

Additionally, I have significantly optimized heap indexing, so !bhi should run much faster now.

Download: 32-bit64-bit

In my latest release, I inadvertently broke compatibility with Windows XP.  This is now fixed.  Additionally, I’ve made a number of performance improvements and minor bug fixes.

Sosex has been updated to work with .NET 4.5.  This version of SOSEX has also been enhanced with DML pretty much everywhere.

The main new feature is the addition of heap indexing.  For large dumps, !bhi (Build Heap Index) will index all of the heap objects and all of the object references in a disk file.  This provides the ability for the new !mroot and existing !refs commands to very quickly locate object roots and references.  With a heap index, !mroot and !refs return nearly instantaneously.  By default, the index is named <DUMP_PATH_AND_NAME>_HeapIndex.bin.  If desired, you can specify your own filename by passing it as a parameter to the !bhi command.  To load a heap index that has already been built, run !lhi (Load Heap Index).  To clear the heap index and remove its resources from memory, run !chi (Clear Heap Index).  For very large dumps, it can take several minutes to build the heap index, but if you need to search for more than a few roots and references, this up-front time is well worth the cost.

As always, PLEASE feel free to offer feedback and/or bug reports to the email address listed by the help command or via the contact link on the site.  I’m generally very responsive to bug reports.  Enjoy!

Download: 32-bit  64-bit

I just released a significant update to SOSEX.  There are many bug fixes as well as several new goodies.

New Commands

• !mdso – Just like !sos.dso, only several switches allow you to filter the output.  No more scrolling through pages of noise to find what you’re looking for.
• !mfrag – Reports on FREE blocks in Gen2, lists objects following the free blocks, and lists pinned GCHandles that reference objects following FREE blocks.
• !mt – Managed trace into.  Cleanly traces into a managed method, skipping runtime goo.
• !mgu – Managed go up.  Cleanly traces out of a managed method, skipping runtime goo.
• !mlocks – Lists all managed locks and unmanaged CriticalSections, along with their owner thread ID.
• !mwaits – Lists all waiting threads and, if it can be determined, the lock objects they are waiting on.
• !mu – Managed Unassemble.  Unassembles around the current IP or the specified IP and lists interleaved source, IL and native code.
• !muf – Managed Unassebmble Function.  Unassembles the current or specified method and lists interleaved source, IL and native code.

Enhanced Commands

• !mx is now AppDomain-aware.
• Breakpoint commands are now AppDomain-aware.

Please consult the help (!sosex.help, !sosex.sosexhelp or !sosex.help <Command>) for more detailed usage information.

Enjoy!  Download 32-bit  Download 64-bit

As Tessmentioned a couple of weeks ago, there was a problem with the way SOSEX displayed CLR thread IDs.  The problem was that I was printing them in decimal format, whereas the SOS/PSSCOR2 !threads command lists them in hexadecimal format.  This caused confusion and extra effort.  This has been corrected, and I now display CLR thread IDs in hex throughout, even using an explicit "0x" prefix to remove all doubt.

As an added goodie, I have implemented support for printing enum fields with their enumeration field names instead of just the numeric value.  For example:

[L0]:0x4004000(DeleteOnClose|Encrypted) (System.IO.FileOptions)

[L8]:0x2 (Green) (ConsoleTestApp.CtaColor)

This way, you can view the backing value as well as the friendly field names.

Enjoy!

I’ve finally completed a new release of SOSEX.  This latest release is compatible with .NET 2.0 and 4.0, and it contains several new goodies as well as some major enhancements to existing functionality.  Here’s an overview:

Enhancements

• .NET 4.0 support for all commands
• !dlk now supports ReaderWriterLock as well as the existing SyncBlock functionality
• !mk now allows you to list locals and parameters inline with the stack trace
• !strings now supports wildcard searches and also now allows you to search user strings in module metadata
• !mdt now supports collection expansion (currently Array, ArrayList, List, Hashtable and Dictionary)
• !mdt now allows you to specify a maximum recursion level for the –r option,
• !mdt is smarter about how it interprets arguments, allowing you to pass arguments in any order
• !mdt now supports field paths, so instead of looking through a complex hierarchy, you can pass a field path explicitly (myLocal.field1.subfield, etc…)
• !mx now provides more information for each item, including the metadata type, MethodDesc, MethodTable and field type, as applicable.
• Many minor bug fixes

New Commands

• !finq – List the contents of the finalization queue
• !frq – List the contents of the f-reachable queue
• !gch – List GCHandles
• !rwlock – List all ReaderWriterLock instances, or list the details of a specific instance by passing an address.

Please be sure to take a look at the !help (or !sosexhelp) command, or open the readme.txt file for details, options and examples for each command.  As always, if you notice any problems or bugs, please contact me to report them.  There are actually four other new commands that I’ve withheld temporarily in order to tighten them up, so I’ll fix any reported bugs while I’m working to complete the new commands.  Expect the new commands within the next month or so.

Enjoy SOSEX v4.0!

Download: 32-bit  64-bit

At long last, I’m pleased to announce the availability of SOSEX version 2.0.  This is a major overhaul of the extension that optimizes and enhances existing functionality, adds many new features, and changes the command naming scheme to be more like the built-in debugger engine commands.  I’ve added support for many of the staple debugger commands by implementing similar functionality and prefixing the commands with “m” (managed).  Included are: mbp, mbm, mbl, mbc, mbe, mbd, mdt, mdv, mframe, mk, mln and mx.

Here’s a quick overview of new things you can do with SOSEX v2:

• Search the heaps for FREE objects by using the new “-free” switch to the “dumpgen” command.  This helps in diagnosing managed heap fragmentation
• Search the heaps for string objects using the new “strings” command.  You can search for strings matching criteria including min length, max length, gen # and string filter
• Specify one-shot, pass count and command for all your managed breakpoints using new breakpoint options
• Set breakpoints on methods that match a wildcard search pattern
• See your full callstack, managed and unmanaged frames, with the “mk” command.  This isn’t straightforward to do with the “clrstack” and “k” commands
• Find type, method and field names in managed assemblies using the “mx” command
• Find out what type of managed data, if any, is located at a given address with the “mln”

Be sure to check out “help” or “sosexhelp” for all commands to ensure you’re aware of all the new goodies.  Stay tuned in the coming days and weeks, as I’ll be posting articles that dive into each individual command in detail.  To start with, though, I’d like to provide an overview of my personal favorite new feature: the “mdt” (Managed Display Type) command.

!mdt allows you to dump the fields of a given type by type name or to examine the fields of an object or value type instance.  Not only can you examine the fields of an object or value type instance, but you can also examine an entire object graph by specifying the “-r” (recursive) switch.  Want to examine a local variable or argument?  No problem.  Just enter “!mdt <Var Name>”, and voila!  The best way to demonstrate the power of this command is with a few examples.

0:000> !mdt System.DateTime 003bef0c
003bef0c (System.DateTime) 2008/01/02 03:04:05.678 VALTYPE (MT=65cd9e98, ADDR=003bef0c)

0:000> !mdt theGuid
003bef1c (System.Guid) {29b9c9c8-3751-42be-8c7a-8b92ff499588} VALTYPE (MT=65cd6c60, ADDR=003bef1c)

0:000> !mdt ConsoleTestApp.TestStruct
TestStruct
   Member1: uint
   Member2: uint

0:000> !mdt ConsoleTestApp.TestStruct 003bef04
003bef04 (ConsoleTestApp.TestStruct) VALTYPE (MT=001b3198, ADDR=003bef04)
   Member1:0x162E (System.UInt32)
   Member2:0x451434E0 (System.UInt32)

0:000> !mdt 6487ce0
06487ce0 (System.Runtime.Remoting.Proxies.__TransparentProxy)
   _rp:06487cb4 (System.Runtime.Remoting.Proxies.RemotingProxy)
   _stubData:06482a5c (BOXED System.IntPtr) VALTYPE (MT=65cfb114, ADDR=06482a60)
   _pMT:65cf902c (System.IntPtr)
   _pInterfaceMT:00000000 (System.IntPtr)
   _stub:68c71e70 (System.IntPtr)

0:000> !mdt 6487ce0 -r
06487ce0 (System.Runtime.Remoting.Proxies.__TransparentProxy)
   _rp:06487cb4 (System.Runtime.Remoting.Proxies.RemotingProxy)
      _tp:06487ce0 (System.Runtime.Remoting.Proxies.__TransparentProxy)
         <RECURSIVE>
      _identity:06487a98 (System.Runtime.Remoting.Identity)
         _flags:0x4 (System.Int32)
         _tpOrObject:06487ce0 (System.Runtime.Remoting.Proxies.__TransparentProxy)
            <RECURSIVE>
         _ObjURI:06484c88 (System.String: "/49cf1154_ea17_4f51_b666_8f91a1263a00/0um3p4he0mnsfugppw1z7gkd_1.rem")
         _URL:NULL (System.String)
         _objRef:06484e10 (System.Runtime.Remoting.ObjRef)
            uri:06484c88 (System.String: "/49cf1154_ea17_4f51_b666_8f91a1263a00/0um3p4he0mnsfugppw1z7gkd_1.rem")
            typeInfo:064850f4 (System.Runtime.Remoting.TypeInfo)
               serverType:064862d4 (System.String: "System.AppDomain, mscorlib, Version=2.0.0.0, Culture=neutral…
               serverHierarchy:NULL (System.Object[])
               interfacesImplemented:064864bc (System.String[], Elements: 2)
            envoyInfo:NULL (System.Runtime.Remoting.IEnvoyInfo)
            channelInfo:06486b5c (System.Runtime.Remoting.ChannelInfo)
               channelData:06486b68 (System.Object[], Elements: 1)
            objrefFlags:0x0 (System.Int32)
            srvIdentity:06484e2c (System.Runtime.InteropServices.GCHandle) VALTYPE (MT=65cfbcb8, ADDR=06484e2c)
               m_handle:007e11ec (System.IntPtr)
            domainID:0x2 (System.Int32)
         _channelSink:06487c90 (System.Runtime.Remoting.Channels.CrossAppDomainSink)
            _xadData:06483834 (System.Runtime.Remoting.Channels.CrossAppDomainData)
               _ContextID:06483854 (BOXED System.Int32) BOXEDVAL=0x87C330
               _DomainID:0x2 (System.Int32)
               _processGuid:06483530 (System.String: "88866410_d263_4d8a_b901_8278132285be")
         _envoyChain:06487ca8 (System.Runtime.Remoting.Messaging.EnvoyTerminatorSink)
         _dph:NULL (System.Runtime.Remoting.Contexts.DynamicPropertyHolder)
         _lease:NULL (System.Runtime.Remoting.Lifetime.Lease)
      _serverObject:NULL (System.MarshalByRefObject)
      _flags:0x3 (System.Runtime.Remoting.Proxies.RealProxyFlags)
      _srvIdentity:06487cd0 (System.Runtime.InteropServices.GCHandle) VALTYPE (MT=65cfbcb8, ADDR=06487cd0)
         m_handle:007e11ec (System.IntPtr)
      _optFlags:0x7000000 (System.Int32)
      _domainID:0x2 (System.Int32)
      _ccm:NULL (System.Runtime.Remoting.Messaging.ConstructorCallMessage)
      _ctorThread:0x0 (System.Int32)
   _stubData:06482a5c (BOXED System.IntPtr) VALTYPE (MT=65cfb114, ADDR=06482a60)
      m_value:ffffffff (System.UIntPtr)
   _pMT:65cf902c (System.IntPtr)
   _pInterfaceMT:00000000 (System.IntPtr)
   _stub:68c71e70 (System.IntPtr)

I hope you enjoy using SOSEX v2.0 as much as I’ve enjoyed writing it!

Download: 32-bit  64-bit

OK, sorry for the OT rant, but may I vent a bit about one of my biggest pet peeves?  OK, thank you for staying with me.

I can't stand online services that don't provide a simple, prominently-displayed cancellation method.  My latest problem is with FreeCreditReport.com.  My wife and I used their services briefly because we needed up-to-the-minute credit score information for a time this Spring and Summer and we found this site to be very convenient for that purpose.  I also want to be clear that we had no problems with the services they provided.  However, tonight I went to cancel the account and was only able to locate this text on the web page:

If for any reason you are not satisfied, you can cancel anytime to discontinue your membership to Triple Advantage Credit Monitoring and stop the monthly billing. However, you will not be eligible for a prorated refund of any portion of your current month's paid membership fee. To cancel, please contact Customer Care at 1-888-829-6560.

Naturally, when I called the number I was informed by the bot that I needed to call back during "normal business hours", which, to them, are 6AM-6PM Pacific.  Forget about it.  I just went and placed a dispute on the recurring charge with my credit card company on the web.

This website so lacks confidence that they can satisfy their customers with the quality of their services that they feel they must resort instead to AOL-like strong-arm tactics to keep subscribers.  Shame on you, Free Credit Report.  If your website is operated in such a manner then please, for the love of all that is good and holy, provide a simple, prominently-displayed cancellation method on your site.

Thank you for listening.  These rants will not be frequent.

Today, a user reported the following error, which occurred when attempting to load the extension.

The call to LoadLibrary(sosex) failed, Win32 error 0n14001 "This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem." Please check your debugger configuration and/or network access.

The short answer to this is that I failed to include the VC9 redist.  You can solve this problem by downloading the latest SOSEX zip file or by downloading the VC9 redist package from Microsoft. The link to the x64 package is also at the bottom of that page.

Here's how I got there, given error number 14001, in case somebody else sees a similar error message in their application.  The description for error 14001 is "The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail."  My search for an explanation of this error led me to this page on Junfeng Zhang's excellent blog.  It was there that I learned that a missing CRT could cause this error.

I've finally gotten around to updating SOSEX.  I incorporated most of the feedback I received and I added a couple of other new goodies.  Here's what's new:

!vars:
Added support for inspecting variables in frames other than the current frame.  By passing a frame index (eg. !vars 3), you can inspect the variables in any single frame in the call stack.  By passing the -w switch (!vars -w), the command will walk the stack all the way up, dumping args/locals for each managed frame.  This was the number one requested improvement for sosex.

!date:
New command.  Pass the address of a System.DateTime Value Type and the date will be displayed in a human readable format (yyyy/mm/dd HH:mm:ss.zzz)
This was another popular feature request.

!dlk:
Added detection capability for cases where a deadlock is caused by a dead thread holding a lock.

!bpmo/!bpsc:
In v1.0, there was a bug that caused only the first instance of a generic method to receive the breakpoint.  This is now fixed so that all generic instances get the breakpoint as expected.

!isf:
Added -c switch to allow viewing of the static variable in the current context only.  Without the switch, the state of the static field is displayed for all AppDomains/Threads.

Value Type display:
I also improved the display of value types.  When !isf or !vars show a value type, SOSEX indicates a value type using a "VALUETYPE" indicator along with a "MT" (method table address) and "ADDR" (value type start address) indicator to facilitate dumping the data with the SOS!dumpvc command.  This provides a big advantage over SOS and v1.0 of SOSEX, because they simply display the first 4 bytes of the value type data, with no VT indication or MT info.

As always, feel free to contact me or leave comments to this post with any bug reports or feature requests.

Enjoy!

Download: x86 | x64

One of the top search phrases I get is "Unable to obtain IXCLRDataProcess".  The reason you're getting this is likely because you are using the !bpsc or !bpmo extensions before the CLR has been initialized.  If you try to run these extensions at the initial process breakpoint, they won't work because the runtime has not been loaded.  I'm hoping to have this corrected for the next release of SOSEX so that !bpsc and !bpmo are saved and then executed when the runtime has initialized.

Over the course of the last few months, I've really put lots of effort into understanding and utilizing WinDbg. As a primarily C# developer, this meant also becoming intimately familiar with the SOS extension. Though a bit tedious, this exercise has already paid rich dividends in my debugging experience. As powerful and handy as SOS is, however, it has some annoying limitations and quirks. My personal peeves with SOS, combined with my desire to learn to write a WinDbg extension, led me to develop SOSEX, a debugging extension for managed code that begins to alleviate some of my frustrations with SOS. SOSEX (available in x86 and x64 versions) provides 8 easy-to-use commands: !dumpgen (dumps the contents of a GC generation), !gcgen (indicates the GC generation of a given object), !refs (lists all references to and from the specified object), !bpsc (breakpoint, source code), !bpmo (breakpoint, method offset), !vars (dump all args and local variables), !isf (inspect static field) and !dlk (deadlock detection).

The rest of this post will provide a bit more detail about each command and how they can save you time. Use the !help command for a list of commands and !help <command name> for the syntax and usage of each command.

!dumpgen and !gcgen

With SOS, you can dump the contents of the heap like so:

0:000> !dumpheap -short
00000642787c7370
00000642787c7388
00000642787c73b0
00000642787c7410
00000642787c7440
00000642787c7498
00000642787c74f0...

The problem with this is that there is no easy way to tell from the output which generation each object belongs to. You can follow up the call to !dumpheap with !eeheap -gc, which will provide the necessary information to determine generations. However, determining the contents of, say, generation 2 using this method is very tedious. Here's the output of !eeheap -gc for a dual-processor system in server GC mode:

0:000> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (0000000002264180)
generation 0 starts at 0x000000007fff0098
generation 1 starts at 0x000000007fff0080
generation 2 starts at 0x000000007fff0068
ephemeral segment allocation context: none
         segment            begin         allocated             size
0000000002271b80 00000642787c7370  0000064278809088 0x0000000000041d18(269592)
000000007fff0000 000000007fff0068  0000000080002fe8 0x0000000000012f80(77696)
Large object heap starts at 0x00000000ffff0068
         segment            begin         allocated             size
00000000ffff0000 00000000ffff0068  00000000ffff80c8 0x0000000000008060(32864)
Heap Size           0x5ccf8(380152)
------------------------------
Heap 1 (0000000002264e00)
generation 0 starts at 0x00000000bfff0098
generation 1 starts at 0x00000000bfff0080
generation 2 starts at 0x00000000bfff0068
ephemeral segment allocation context: none
         segment            begin         allocated             size
00000000bfff0000 00000000bfff0068  00000000bfff00b0 0x0000000000000048(72)
Large object heap starts at 0x000000010fff0068
         segment            begin         allocated             size
000000010fff0000 000000010fff0068  000000010fff0080 0x0000000000000018(24)
Heap Size              0x60(96)
------------------------------
GC Heap Size           0x5cd58(380248)

As you can see, you have a lot of work to do in order to pick through the output of !dumpheap and compare object addresses to the segment addresses provided by !eeheap -gc. Enter SOSEX's !dumpgen command. Using !dumpgen, you can easily determine the contents of a given generation simply by providing the number of the generation you wish to examine:

0:000> !dumpgen 2
00000642787c7370         24    System.Object
00000642787c7388         38    System.String    STRVAL=length
00000642787c73b0         94    System.String    STRVAL=ArgumentOutOfRange_MustBeNonNegNum
00000642787c7410         46    System.String    STRVAL=startIndex
00000642787c7440         84    System.String    STRVAL=ArgumentOutOfRange_StartIndex
00000642787c7498         86    System.String    STRVAL=ArgumentOutOfRange_IndexLength
00000642787c74f0         42    System.String    STRVAL=capacity...

Conversely, if you know an object's address and you wish to know what generation it belongs to, you can simply provide the address of the object to the !gcgen command.

!refs

Though not a replacement, the !refs command supplements SOS's !gcroot command by allowing you to view  the immediate references from and to a given object.

0:000> !refs 0000000080000db8
Objects referenced by 0000000080000db8 (System.Threading.Mutex):
0000000080000ef0         32    Microsoft.Win32.SafeHandles.SafeWaitHandle

Objects referencing 0000000080000db8 (System.Threading.Mutex):
0000000080000e08         72    System.Threading.Mutex+<>c__DisplayClass3
0000000080000e50         64    System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode

The sample output above shows only heap references, but !refs will also list all references from handles, stacks, registers and the freachable queues.

!dlk

The !dlk command allows you to easily spot deadlocks in your application if you suspect deadlock to be the cause of an application hang. If !dlk detects deadlock, the output will list the sync blocks that are held as well as the sync blocks for which each thread is waiting, as well as the type, method, IL offset, and, if symbols are available, the source code and line number at which each thread is waiting:

0:010> !dlk
Deadlock detected:
CLR thread 4 holds sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
             waits sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
CLR thread 5 holds sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
             waits sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
CLR Thread 4 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]
CLR Thread 5 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]

1 deadlock detected.

As you can see, !dlk makes it dead simple to troubleshoot this common kind of deadlock in managed code. One important caveat holds true for !dlk: it only works for deadlocks on "sync blocks". Put more simply, it will only spot locks created by Monitor.Enter (which is used by the C# lock keyword). !dlk will not catch deadlock on other types of synchronization objects, such as mutexes and semaphores.

!bpsc and !bpmo

Although managed debugging with WinDbg has become more supported in later builds, it is still difficult to get managed source code breakpoints to work correctly. With the !bpsc command, it is now simple to set source code breakpoints. !bpsc allows you to set a breakpoint in a source code file at a specified line number and, optionally, column number. I've often wanted to stop the execution of my application either deep inside a method in the BCL or in one of my own methods in a build for which I have no symbols. Though SOS provides the handy !bpmd command, !bpmd only allows you to set a breakpoint on the first instruction of a method. This is often not effective for more difficult debugging scenarios. The !bpmo command now provides a means for you to set a breakpoint at specific IL offset within a given type and method. 

!vars and !isf

Once you are stopped at a breakpoint, you need an easy means to inspect the contents of method arguments, local variables and instance and static fields. !vars dumps the values of arguments and local variables. If symbols are available, !vars will display the names of the local variables. Names of arguments are (with rare exception) provided even without symbols. In order to inspect member variables, first call !vars and then provide the value of "this" to the sos!do command. Using SOS, it is a bit of a cumbersome process to view the contents of static fields. The !isf command makes it simple by allowing you to simply specify the type and field name to inspect.

I hope you enjoy using SOSEX as much as I have enjoyed writing it. Please provide bug reports and feature requests either in the comments for this post or via the "email" link on my home page. I owe many thanks to John Robbins, who provided some key feature ideas for SOSEX as well as some valuable testing time.  John has also written a nice blog post describing how he uses SOSEX in his own debugging.  Thanks, John!

Download: x86 | x64

I just switched over to dasBlog and Windows Live Writer.  This is just a test drive.

HttpConfig has been updated to work correctly with UAC on Vista.

Roger Lipscombe also pointed out some NullReferenceExceptions, which are also now fixed. Thanks, Roger!

Download: exesource

I've updated HttpConfig to fix a couple of 64-bit bugs.

Download: exesource

At long last, I've been able to update the HttpConfig GUI tool. I've fixed many bugs, streamlined the UI, added graphical security editing and gotten the thing to run on Vista.

Thanks to Keith and Dominick for the inspiration to enhance the security stuff. Now that I'm using .NET 2.0 for this app, I was able to get the EditSecurity API working purely in C#.

Download: exe source

As indicated in my last post, I'm posting a wrap-up to the SSL cert saga. Unfortunately, I've waited too long between posts and forgotten some of what I had intended to write here. However, since I included more details in the last post than I had intended, you're not missing much.

One of the more interesting tasks involved in researching SSL cert problems was how to locate the key containers. The manner in which key containers are persisted is not documented anywhere that I know of, and I'm sure that's intentional as key container persistence is an internal implementation detail. I knew from some work several years ago that the MS RSA providers persist the key containers in the file system. I also remembered the approximate location, but I couldn't remember all the details. I fired up filemon and my memory was refreshed. The keys are stored in the common or user profile, depending on whether the container is a machine or user keyset. However, the containers are named in some bizarre guidish/hashy-looking format, so it's not obvious at a glance which files belong to which key containers. I was able to locate older key containers by opening them in TextPad because the name of the container is embedded in ASCII text within the file itself. New containers were easy to locate simply by sorting the files. However, I wanted to know how the filenames are derived. I have a sick mind that way. I always want to know how things work behind the scenes. After a few hours with FileMon, RegMon and the disassembler, I finally think I understand how these filenames are derived. I'm going to document this process here for the curious and so I can find this information the next time I need it. This information is applicable to Win2K, WinXP and Win2K3. I don't yet have a Vista box to test this on, so I don't know if it remains the same in that version of the OS.

First of all, the key containers are persisted in the following locations:
User keys: %PROFILE%\Application Data\Microsoft\Crypto\RSA\[SID string]
Machine keys: %ALLUSERS%\Application Data\Microsoft\Crypto\RSA\MachineKeys

The filenames look like so: d99e964c16600b1681cdfbad2d5d5eac_2f896382-3658-492e-9219-ce847feb6b8a

The filename consists of a MD5-derived prefix and the MachineGuid, separated by an underscore. The MachineGuid is a registry string value, located under the key "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography". The hash prefix is derived by performing the following operations:

1 - Obtain the name of the key container. The private key container name for a cert can be obtained via the CertGetCertificateContextProperty API, specifying the CERT_KEY_PROV_INFO_PROP_ID property. You can also obtain the key container name by using the checkcert utility from my last post.
2 - Convert the key container name to lower case.
3 - Append a null character
4 - Convert the string to ASCII bytes
5 - Perform an MD5 hash of the ASCII bytes
6 - Treat each 4 byte block of the hash as a 32-bit integer, reversing each
7 - Convert the resulting bytes to a hex string

I'm including a sample class that demonstrates the code necessary to calculate the full path and filename of an RSA key container. The class exposes a static method called "GetRsaKeyContainerFilename" as well as methods that will allow you to perform each step of this operation separately.

There may be an API to do this, but I haven't been able to find it. I wasn't as concerned with an easy way to do this as much as understanding the process, anyway.

Download: RsaUtil.zip

One of the joys Ive experienced in my recent dealings with the HTTP API and some lower-level SChannel code is that of trying to generate proper SSL test certificates. During the initial development phase of the product Im currently working on, we used one common test certificate for all our test/dev servers. Later on, we began to generate different certificates for each server so that we could test some cert verification code and also verify that we knew exactly what was required of certificates to work with our product. Needless to say (or I wouldnt have anything to write here), we immediately began to have many problems. Some servers worked just fine, while others failed. On the failing servers, test requests with WebRequest yielded the familiar but unhelpful error "Could not establish secure channel for SSL/TLS". Attempts to connect from IE produced the error "Cannot find server or DNS Error". In order to further troubleshoot, I tried making a connection using a plain SChannel-based SSL socket. The connection was established just fine and the client socket sent the hello message. As soon as the client socket attempted to read the server response, an exception was thrown, indicating that the connection had been closed by the server. Next, I enabled full SChannel logging on the server and restarted. Attempts to connect left 2 messages in the event logs: "Creating an SSL server credential", followed by "The SSL server credentials private key has the following properties..." and NO error. Baffled, I installed Microsofts SSL Diagnostics utility and attempted a handshake simulation. This yielded the error "Unexpected error receiving data (0x80090304)", which error code is defined as SEC_E_INTERNAL_ERROR; again, a most unhelpful description. This led me on a week-long study of X.509 certificates, makecert.exe and the MS Crypto Providers key container implementation. Im going to omit many details here in order to get to the point, which is to enumerate the problems and solutions Ive found for our situation. Ill follow up with another post to describe the gory details of this research in some depth. Anyway, simply stated, the problem is that the certificate's private key is either inaccessible or the keyspec is incorrect.

Problems

Our problems basically boiled down to three root causes:

1 - Improper import of good certificates, which caused them to appear faulty.

2 - Use of a build of makecert.exe that generates key containers improperly

3 - Failure to realize that makecert.exe does not, by default, generate certs that use the correct keyspec for an SSL cert.

Problem #1 - The problem with importing was that I sometimes imported the certificate into the CurrentUser MY store and then, realizing that I'd imported into the wrong store, copied the cert into the LocalMachine MY store. The problem is that once a certificate is imported into a CurrentUser store, the private key container resides in the users profile, where it cannot be found by a system process.

Problem #2 - One of the first things I do when I create a development VM is to add the path to the .NET SDK bin directory to my system path. I always want access to the tools therein from any command prompt. Unfortunately, the makecert.exe in that directory (v5.131.2157.1) wont work for generating SSL certs. The problem is that even if you specify a LocalMachine store for the cert, the private key container it generates is generated as a user key, not a machine key. Therefore, though everything looks good, when you use the cert from a system process, you experience the symptoms enumerated above. In order to use this build for SSL certs, youd have to pre-generate the key container and reference the pre-created container on the makecert command line.

Problem #3 - There are two standard key types for RSA keys: signature and exchange. Which key is used is defined by the keyspec property of the private key context belonging to the certificate. The keyspec property must be set to "Exchange" in order for the cert to work for SSL. This is obvious to me, but I never realized that makecert.exe sets the keyspec to Signature by default, and there isnt a standard interface to view this property of a certificate that Im aware of.

Solutions

Solution #1 - Import certificates intended for SSL usage into a LocalMachine store directly. Dont try to copy a cert from a CurrentUser store into a LocalMachine store. If youve mistakenly imported a cert into a CurrentUser store, you need to delete the cert and the key container (deleting a certificate does not delete the key container), and then import the cert into a LocalMachine store. See my next post for details as to how to locate the key container and delete it.

Solution #2 - Dont use v5.131.2157.1 of makecert.exe (the one in the .NET v1.1 SDK) to generate SSL certificates. It always generates user key containers, which wont work from a system process. Fortunately, the version of makecert.exe (v5.131.3790.0) that ships with v2.0 of the .NET SDK, as well as the latest Platform SDKs, works properly.

Solution #3 - Always use the command line option "-sky exchange" with makecert.exe if youre generating SSL certificates. If you dont specify this, it will default to the signature keyspec, which wont work for SSL.

The difficulties Ive experienced have been compounded by the fact that any one or more of these problems could be combined in a single situation, making it terribly difficult to pinpoint precisely what is going on. To aid in the diagnosis of SSL certificate problems, Im making available a little command line utility that I wrote for this purpose.

checkcert: exe source

To use checkcert.exe, pass a store name and a cert subject search string (such as the common name). Checkcert will display some key information about the cert that is not available in the built-in tools (again, that Im aware of). Most importantly, checkcert will display the private key container location and the keyspec. You want to ensure that the key container location is "Machine" and that the keyspec is "Exchange" for an SSL cert. I hope this information and the utility will save someone the hassle of trying to unravel this mess from scratch.

I've been doing a fair bit of development against the MS HTTP API lately, and I'm weary of the httpcfg.exe utility. First of all, it's a right pain in the neck to find in the first place. Once located and downloaded, it's equally painful to use, IMO. Having read several articles on the net about the woes others were also experiencing in using this tool, I decided to write a simple GUI application against the Http configuration APIs. It's called, quite simply, HttpConfig. It won't win any points for UI flash, but I do hope it will make life easier for some as it already has for me. Enjoy!

Download: exe source