Today, a user reported the following error, which occurred when attempting to load the extension.

The call to LoadLibrary(sosex) failed, Win32 error 0n14001 "This application has failed to start because the application configuration is incorrect. Reinstalling the application may fix this problem." Please check your debugger configuration and/or network access.

The short answer to this is that I failed to include the VC9 redist.  You can solve this problem by downloading the latest SOSEX zip file or by downloading the VC9 redist package from Microsoft. The link to the x64 package is also at the bottom of that page.

Here's how I got there, given error number 14001, in case somebody else sees a similar error message in their application.  The description for error 14001 is "The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log for more detail."  My search for an explanation of this error led me to this page on Junfeng Zhang's excellent blog.  It was there that I learned that a missing CRT could cause this error.

I've finally gotten around to updating SOSEX.  I incorporated most of the feedback I received and I added a couple of other new goodies.  Here's what's new:

!vars:
Added support for inspecting variables in frames other than the current frame.  By passing a frame index (eg. !vars 3), you can inspect the variables in any single frame in the call stack.  By passing the -w switch (!vars -w), the command will walk the stack all the way up, dumping args/locals for each managed frame.  This was the number one requested improvement for sosex.

!date:
New command.  Pass the address of a System.DateTime Value Type and the date will be displayed in a human readable format (yyyy/mm/dd HH:mm:ss.zzz)
This was another popular feature request.

!dlk:
Added detection capability for cases where a deadlock is caused by a dead thread holding a lock.

!bpmo/!bpsc:
In v1.0, there was a bug that caused only the first instance of a generic method to receive the breakpoint.  This is now fixed so that all generic instances get the breakpoint as expected.

!isf:
Added -c switch to allow viewing of the static variable in the current context only.  Without the switch, the state of the static field is displayed for all AppDomains/Threads.

Value Type display:
I also improved the display of value types.  When !isf or !vars show a value type, SOSEX indicates a value type using a "VALUETYPE" indicator along with a "MT" (method table address) and "ADDR" (value type start address) indicator to facilitate dumping the data with the SOS!dumpvc command.  This provides a big advantage over SOS and v1.0 of SOSEX, because they simply display the first 4 bytes of the value type data, with no VT indication or MT info.

As always, feel free to contact me or leave comments to this post with any bug reports or feature requests.

Enjoy!

Download: x86 | x64

One of the top search phrases I get is "Unable to obtain IXCLRDataProcess".  The reason you're getting this is likely because you are using the !bpsc or !bpmo extensions before the CLR has been initialized.  If you try to run these extensions at the initial process breakpoint, they won't work because the runtime has not been loaded.  I'm hoping to have this corrected for the next release of SOSEX so that !bpsc and !bpmo are saved and then executed when the runtime has initialized.

Over the course of the last few months, I've really put lots of effort into understanding and utilizing WinDbg. As a primarily C# developer, this meant also becoming intimately familiar with the SOS extension. Though a bit tedious, this exercise has already paid rich dividends in my debugging experience. As powerful and handy as SOS is, however, it has some annoying limitations and quirks. My personal peeves with SOS, combined with my desire to learn to write a WinDbg extension, led me to develop SOSEX, a debugging extension for managed code that begins to alleviate some of my frustrations with SOS. SOSEX (available in x86 and x64 versions) provides 8 easy-to-use commands: !dumpgen (dumps the contents of a GC generation), !gcgen (indicates the GC generation of a given object), !refs (lists all references to and from the specified object), !bpsc (breakpoint, source code), !bpmo (breakpoint, method offset), !vars (dump all args and local variables), !isf (inspect static field) and !dlk (deadlock detection).

The rest of this post will provide a bit more detail about each command and how they can save you time. Use the !help command for a list of commands and !help <command name> for the syntax and usage of each command.

!dumpgen and !gcgen

With SOS, you can dump the contents of the heap like so:

0:000> !dumpheap -short
00000642787c7370
00000642787c7388
00000642787c73b0
00000642787c7410
00000642787c7440
00000642787c7498
00000642787c74f0...

The problem with this is that there is no easy way to tell from the output which generation each object belongs to. You can follow up the call to !dumpheap with !eeheap -gc, which will provide the necessary information to determine generations. However, determining the contents of, say, generation 2 using this method is very tedious. Here's the output of !eeheap -gc for a dual-processor system in server GC mode:

0:000> !eeheap -gc
Number of GC Heaps: 2
------------------------------
Heap 0 (0000000002264180)
generation 0 starts at 0x000000007fff0098
generation 1 starts at 0x000000007fff0080
generation 2 starts at 0x000000007fff0068
ephemeral segment allocation context: none
         segment            begin         allocated             size
0000000002271b80 00000642787c7370  0000064278809088 0x0000000000041d18(269592)
000000007fff0000 000000007fff0068  0000000080002fe8 0x0000000000012f80(77696)
Large object heap starts at 0x00000000ffff0068
         segment            begin         allocated             size
00000000ffff0000 00000000ffff0068  00000000ffff80c8 0x0000000000008060(32864)
Heap Size           0x5ccf8(380152)
------------------------------
Heap 1 (0000000002264e00)
generation 0 starts at 0x00000000bfff0098
generation 1 starts at 0x00000000bfff0080
generation 2 starts at 0x00000000bfff0068
ephemeral segment allocation context: none
         segment            begin         allocated             size
00000000bfff0000 00000000bfff0068  00000000bfff00b0 0x0000000000000048(72)
Large object heap starts at 0x000000010fff0068
         segment            begin         allocated             size
000000010fff0000 000000010fff0068  000000010fff0080 0x0000000000000018(24)
Heap Size              0x60(96)
------------------------------
GC Heap Size           0x5cd58(380248)

As you can see, you have a lot of work to do in order to pick through the output of !dumpheap and compare object addresses to the segment addresses provided by !eeheap -gc. Enter SOSEX's !dumpgen command. Using !dumpgen, you can easily determine the contents of a given generation simply by providing the number of the generation you wish to examine:

0:000> !dumpgen 2
00000642787c7370         24    System.Object
00000642787c7388         38    System.String    STRVAL=length
00000642787c73b0         94    System.String    STRVAL=ArgumentOutOfRange_MustBeNonNegNum
00000642787c7410         46    System.String    STRVAL=startIndex
00000642787c7440         84    System.String    STRVAL=ArgumentOutOfRange_StartIndex
00000642787c7498         86    System.String    STRVAL=ArgumentOutOfRange_IndexLength
00000642787c74f0         42    System.String    STRVAL=capacity...

Conversely, if you know an object's address and you wish to know what generation it belongs to, you can simply provide the address of the object to the !gcgen command.

!refs

Though not a replacement, the !refs command supplements SOS's !gcroot command by allowing you to view  the immediate references from and to a given object.

0:000> !refs 0000000080000db8
Objects referenced by 0000000080000db8 (System.Threading.Mutex):
0000000080000ef0         32    Microsoft.Win32.SafeHandles.SafeWaitHandle

Objects referencing 0000000080000db8 (System.Threading.Mutex):
0000000080000e08         72    System.Threading.Mutex+<>c__DisplayClass3
0000000080000e50         64    System.Runtime.CompilerServices.RuntimeHelpers+CleanupCode

The sample output above shows only heap references, but !refs will also list all references from handles, stacks, registers and the freachable queues.

!dlk

The !dlk command allows you to easily spot deadlocks in your application if you suspect deadlock to be the cause of an application hang. If !dlk detects deadlock, the output will list the sync blocks that are held as well as the sync blocks for which each thread is waiting, as well as the type, method, IL offset, and, if symbols are available, the source code and line number at which each thread is waiting:

0:010> !dlk
Deadlock detected:
CLR thread 4 holds sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
             waits sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
CLR thread 5 holds sync block 00000000024c6928 OBJ:000000007fff0fa8[System.String] STRVAL=SYNC2
             waits sync block 00000000024c6970 OBJ:000000007fff0f80[System.String] STRVAL=SYNC1
CLR Thread 4 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]
CLR Thread 5 is waiting at ConsoleTestApp.ConsoleTestApp.MonitorDeadlockThreadProc()+0xa4(IL) [C:\dev\ConsoleTestApp\ConsoleTestApp.cs, line 195]

1 deadlock detected.

As you can see, !dlk makes it dead simple to troubleshoot this common kind of deadlock in managed code. One important caveat holds true for !dlk: it only works for deadlocks on "sync blocks". Put more simply, it will only spot locks created by Monitor.Enter (which is used by the C# lock keyword). !dlk will not catch deadlock on other types of synchronization objects, such as mutexes and semaphores.

!bpsc and !bpmo

Although managed debugging with WinDbg has become more supported in later builds, it is still difficult to get managed source code breakpoints to work correctly. With the !bpsc command, it is now simple to set source code breakpoints. !bpsc allows you to set a breakpoint in a source code file at a specified line number and, optionally, column number. I've often wanted to stop the execution of my application either deep inside a method in the BCL or in one of my own methods in a build for which I have no symbols. Though SOS provides the handy !bpmd command, !bpmd only allows you to set a breakpoint on the first instruction of a method. This is often not effective for more difficult debugging scenarios. The !bpmo command now provides a means for you to set a breakpoint at specific IL offset within a given type and method. 

!vars and !isf

Once you are stopped at a breakpoint, you need an easy means to inspect the contents of method arguments, local variables and instance and static fields. !vars dumps the values of arguments and local variables. If symbols are available, !vars will display the names of the local variables. Names of arguments are (with rare exception) provided even without symbols. In order to inspect member variables, first call !vars and then provide the value of "this" to the sos!do command. Using SOS, it is a bit of a cumbersome process to view the contents of static fields. The !isf command makes it simple by allowing you to simply specify the type and field name to inspect.

I hope you enjoy using SOSEX as much as I have enjoyed writing it. Please provide bug reports and feature requests either in the comments for this post or via the "email" link on my home page. I owe many thanks to John Robbins, who provided some key feature ideas for SOSEX as well as some valuable testing time.  John has also written a nice blog post describing how he uses SOSEX in his own debugging.  Thanks, John!

Download: x86 | x64

I just switched over to dasBlog and Windows Live Writer.  This is just a test drive.

HttpConfig has been updated to work correctly with UAC on Vista.

Roger Lipscombe also pointed out some NullReferenceExceptions, which are also now fixed. Thanks, Roger!

Download: exesource

I've updated HttpConfig to fix a couple of 64-bit bugs.

Download: exesource

At long last, I've been able to update the HttpConfig GUI tool. I've fixed many bugs, streamlined the UI, added graphical security editing and gotten the thing to run on Vista.

Thanks to Keith and Dominick for the inspiration to enhance the security stuff. Now that I'm using .NET 2.0 for this app, I was able to get the EditSecurity API working purely in C#.

Download: exe source

As indicated in my last post, I'm posting a wrap-up to the SSL cert saga. Unfortunately, I've waited too long between posts and forgotten some of what I had intended to write here. However, since I included more details in the last post than I had intended, you're not missing much.

One of the more interesting tasks involved in researching SSL cert problems was how to locate the key containers. The manner in which key containers are persisted is not documented anywhere that I know of, and I'm sure that's intentional as key container persistence is an internal implementation detail. I knew from some work several years ago that the MS RSA providers persist the key containers in the file system. I also remembered the approximate location, but I couldn't remember all the details. I fired up filemon and my memory was refreshed. The keys are stored in the common or user profile, depending on whether the container is a machine or user keyset. However, the containers are named in some bizarre guidish/hashy-looking format, so it's not obvious at a glance which files belong to which key containers. I was able to locate older key containers by opening them in TextPad because the name of the container is embedded in ASCII text within the file itself. New containers were easy to locate simply by sorting the files. However, I wanted to know how the filenames are derived. I have a sick mind that way. I always want to know how things work behind the scenes. After a few hours with FileMon, RegMon and the disassembler, I finally think I understand how these filenames are derived. I'm going to document this process here for the curious and so I can find this information the next time I need it. This information is applicable to Win2K, WinXP and Win2K3. I don't yet have a Vista box to test this on, so I don't know if it remains the same in that version of the OS.

First of all, the key containers are persisted in the following locations:
User keys: %PROFILE%\Application Data\Microsoft\Crypto\RSA\[SID string]
Machine keys: %ALLUSERS%\Application Data\Microsoft\Crypto\RSA\MachineKeys

The filenames look like so: d99e964c16600b1681cdfbad2d5d5eac_2f896382-3658-492e-9219-ce847feb6b8a

The filename consists of a MD5-derived prefix and the MachineGuid, separated by an underscore. The MachineGuid is a registry string value, located under the key "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Cryptography". The hash prefix is derived by performing the following operations:

1 - Obtain the name of the key container. The private key container name for a cert can be obtained via the CertGetCertificateContextProperty API, specifying the CERT_KEY_PROV_INFO_PROP_ID property. You can also obtain the key container name by using the checkcert utility from my last post.
2 - Convert the key container name to lower case.
3 - Append a null character
4 - Convert the string to ASCII bytes
5 - Perform an MD5 hash of the ASCII bytes
6 - Treat each 4 byte block of the hash as a 32-bit integer, reversing each
7 - Convert the resulting bytes to a hex string

I'm including a sample class that demonstrates the code necessary to calculate the full path and filename of an RSA key container. The class exposes a static method called "GetRsaKeyContainerFilename" as well as methods that will allow you to perform each step of this operation separately.

There may be an API to do this, but I haven't been able to find it. I wasn't as concerned with an easy way to do this as much as understanding the process, anyway.

Download: RsaUtil.zip

One of the joys Ive experienced in my recent dealings with the HTTP API and some lower-level SChannel code is that of trying to generate proper SSL test certificates. During the initial development phase of the product Im currently working on, we used one common test certificate for all our test/dev servers. Later on, we began to generate different certificates for each server so that we could test some cert verification code and also verify that we knew exactly what was required of certificates to work with our product. Needless to say (or I wouldnt have anything to write here), we immediately began to have many problems. Some servers worked just fine, while others failed. On the failing servers, test requests with WebRequest yielded the familiar but unhelpful error "Could not establish secure channel for SSL/TLS". Attempts to connect from IE produced the error "Cannot find server or DNS Error". In order to further troubleshoot, I tried making a connection using a plain SChannel-based SSL socket. The connection was established just fine and the client socket sent the hello message. As soon as the client socket attempted to read the server response, an exception was thrown, indicating that the connection had been closed by the server. Next, I enabled full SChannel logging on the server and restarted. Attempts to connect left 2 messages in the event logs: "Creating an SSL server credential", followed by "The SSL server credentials private key has the following properties..." and NO error. Baffled, I installed Microsofts SSL Diagnostics utility and attempted a handshake simulation. This yielded the error "Unexpected error receiving data (0x80090304)", which error code is defined as SEC_E_INTERNAL_ERROR; again, a most unhelpful description. This led me on a week-long study of X.509 certificates, makecert.exe and the MS Crypto Providers key container implementation. Im going to omit many details here in order to get to the point, which is to enumerate the problems and solutions Ive found for our situation. Ill follow up with another post to describe the gory details of this research in some depth. Anyway, simply stated, the problem is that the certificate's private key is either inaccessible or the keyspec is incorrect.

Problems

Our problems basically boiled down to three root causes:

1 - Improper import of good certificates, which caused them to appear faulty.

2 - Use of a build of makecert.exe that generates key containers improperly

3 - Failure to realize that makecert.exe does not, by default, generate certs that use the correct keyspec for an SSL cert.

Problem #1 - The problem with importing was that I sometimes imported the certificate into the CurrentUser MY store and then, realizing that I'd imported into the wrong store, copied the cert into the LocalMachine MY store. The problem is that once a certificate is imported into a CurrentUser store, the private key container resides in the users profile, where it cannot be found by a system process.

Problem #2 - One of the first things I do when I create a development VM is to add the path to the .NET SDK bin directory to my system path. I always want access to the tools therein from any command prompt. Unfortunately, the makecert.exe in that directory (v5.131.2157.1) wont work for generating SSL certs. The problem is that even if you specify a LocalMachine store for the cert, the private key container it generates is generated as a user key, not a machine key. Therefore, though everything looks good, when you use the cert from a system process, you experience the symptoms enumerated above. In order to use this build for SSL certs, youd have to pre-generate the key container and reference the pre-created container on the makecert command line.

Problem #3 - There are two standard key types for RSA keys: signature and exchange. Which key is used is defined by the keyspec property of the private key context belonging to the certificate. The keyspec property must be set to "Exchange" in order for the cert to work for SSL. This is obvious to me, but I never realized that makecert.exe sets the keyspec to Signature by default, and there isnt a standard interface to view this property of a certificate that Im aware of.

Solutions

Solution #1 - Import certificates intended for SSL usage into a LocalMachine store directly. Dont try to copy a cert from a CurrentUser store into a LocalMachine store. If youve mistakenly imported a cert into a CurrentUser store, you need to delete the cert and the key container (deleting a certificate does not delete the key container), and then import the cert into a LocalMachine store. See my next post for details as to how to locate the key container and delete it.

Solution #2 - Dont use v5.131.2157.1 of makecert.exe (the one in the .NET v1.1 SDK) to generate SSL certificates. It always generates user key containers, which wont work from a system process. Fortunately, the version of makecert.exe (v5.131.3790.0) that ships with v2.0 of the .NET SDK, as well as the latest Platform SDKs, works properly.

Solution #3 - Always use the command line option "-sky exchange" with makecert.exe if youre generating SSL certificates. If you dont specify this, it will default to the signature keyspec, which wont work for SSL.

The difficulties Ive experienced have been compounded by the fact that any one or more of these problems could be combined in a single situation, making it terribly difficult to pinpoint precisely what is going on. To aid in the diagnosis of SSL certificate problems, Im making available a little command line utility that I wrote for this purpose.

checkcert: exe source

To use checkcert.exe, pass a store name and a cert subject search string (such as the common name). Checkcert will display some key information about the cert that is not available in the built-in tools (again, that Im aware of). Most importantly, checkcert will display the private key container location and the keyspec. You want to ensure that the key container location is "Machine" and that the keyspec is "Exchange" for an SSL cert. I hope this information and the utility will save someone the hassle of trying to unravel this mess from scratch.

I've been doing a fair bit of development against the MS HTTP API lately, and I'm weary of the httpcfg.exe utility. First of all, it's a right pain in the neck to find in the first place. Once located and downloaded, it's equally painful to use, IMO. Having read several articles on the net about the woes others were also experiencing in using this tool, I decided to write a simple GUI application against the Http configuration APIs. It's called, quite simply, HttpConfig. It won't win any points for UI flash, but I do hope it will make life easier for some as it already has for me. Enjoy!

Download: exe source