Monday, November 18, 2013

Accessing 64-bit Registry Keys from 32-bit Powershell

On my current project we’re using SolutionScripts to perform automated initialization of the development environment.  One of our steps restores a SQL Server database from a provided backup filename.  However, the Powershell script that performs the restore looks in the registry to determine the default SQL Server locations for Data and Log files (this script was previously used on a build server where we had put the data and log files on separate physical drives for performance reasons). 

The problem is that the Package Manager Console window in Visual Studio runs in 32-bit mode, while SQL Server is a 64-bit application and thus its registry entries are not accessible.  Thus, the following command fails to read the desired value:

$dataPath = gi "HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\$instanceName\MSSQLServer" | %{$_.GetValue("DefaultData")}

One possible solution would be to use Invoke-Command to effectively execute the statement in a 64-bit process, like so:

$dataPath = Invoke-Command -ComputerName localhost -ScriptBlock {gi "HKLM:\SOFTWARE\Microsoft\Microsoft SQL Server\$instanceName\MSSQLServer" | %{$_.GetValue("DefaultData")}}
The problem with this approach is that it is using Powershell Remoting and requires that each developer execute the following command as Administrator:
Enable-PsRemoting -Force

Anytime I have to execute something as an Administrator and use the -Force flag (or confirm through a bunch of prompts to explicitly change system settings), a shiver goes up my spine. 


Powershell 2.0 Solution


I pieced together a solution that works with Powershell 2.0 using the following tidbits:


  • While the out-of-the-box APIs prior to .NET 4.0 do not give you the ability to read the 64-bit registry from a 32-bit process, the Windows API does support it.
  • With a little help from pinvoke.net you can piece together the import definitions in C# to make the necessary Windows API calls.
  • With a little help from LINQPad (and plenty of tinkering and process crashes), you can get the allocation and releasing of memory buffers just right to successfully read 64-bit registry values from a 32-bit process.
  • With the use of the Add-Type commandlet in Powershell, you can actually make an embedded C# class available to your scripts.


Here’s the C# source:

using Microsoft.Win32;
using System;
using System.Text;
using System.Runtime.InteropServices;

namespace Dlp
{
public static class Registry
{
static UIntPtr HKEY_LOCAL_MACHINE = new UIntPtr(0x80000002u);
const int KEY_WOW64_64KEY = 0x100;
const int KEY_ALL_ACCESS = 0xf003f;
const int KEY_QUERY_VALUE = 0x0001;

const int RegSz = 0x00000002;
const int RegMultiSz = 7;

// Define other methods and classes here
[DllImport("advapi32.dll")]
static extern int RegOpenKeyEx(
UIntPtr hKey,
string subKey,
int options,
int sam,
out UIntPtr phkResult );

/* Retrieves the type and data for the specified registry value. */
[DllImport("Advapi32.dll", EntryPoint = "RegGetValueW", CharSet = CharSet.Unicode, SetLastError = true)]
internal static extern long RegGetValue(
UIntPtr hkey,
string lpSubKey,
string lpValue,
int dwFlags,
out int pdwType,
IntPtr pvData,
ref uint pcbData);

public static string GetRegistryStringValue(string key, string subkey, string valueName)
{
UIntPtr hkey = UIntPtr.Zero;
RegOpenKeyEx(HKEY_LOCAL_MACHINE, key, 0, KEY_WOW64_64KEY | KEY_ALL_ACCESS, out hkey);

int pdwType = 0;

IntPtr pvData = IntPtr.Zero;
uint pcbData = 0;

// Determine necessary buffer length
RegGetValue(hkey, subkey, valueName, RegSz, out pdwType, pvData, ref pcbData);

// If there's no data available...
if (pcbData == 0)
return null;

// Allocate the correctly sized buffer
pvData = Marshal.AllocHGlobal((int) pcbData);
RegGetValue(hkey, subkey, valueName, RegSz, out pdwType, pvData, ref pcbData);

// Copy unmanaged buffer to a byte array
byte[] ar = new byte[pcbData-2]; // Trim trailing null character from buffer
Marshal.Copy(pvData, ar, 0, ar.Length);

// Free the unmanaged buffer
Marshal.FreeHGlobal(pvData);

// Convert the unicode bytes to a string
string value = Encoding.Unicode.GetString(ar);

return value;
}
}
}

The code makes use of the RegOpenKeyEx and RegGetValue Windows API functions, along with the ever so important KEY_WOW64_64KEY option flag which is what actually provides the 64-bit registry access.  Beyond that, there’s some boring (but very dangerous) buffer management and unicode string encoding (boy, if I had a nickel for every time I crashed LINQPad while…)


With the C# function written and nicely packaged up in a namespace and a static class, it’s time to make it available to Powershell environment.  I simply added a function to my database management Powershell module that looks like this:

Function Define-GetRegistryStringValue () {
$sourceCode = @"
using Microsoft.Win32;
using System;
using System.Text;
using System.Runtime.InteropServices;

namespace Dlp
{
public static class Registry
{
static UIntPtr HKEY_LOCAL_MACHINE = new UIntPtr(0x80000002u);

....

public static string GetRegistryStringValue(string key, string subkey, string valueName)
{
UIntPtr hkey = UIntPtr.Zero;
RegOpenKeyEx(HKEY_LOCAL_MACHINE, key, 0, KEY_WOW64_64KEY | KEY_ALL_ACCESS, out hkey);

....

// Convert the unicode bytes to a string
string value = Encoding.Unicode.GetString(ar);

return value;
}
}
}
"@

Add-Type -TypeDefinition $sourceCode
}

Define-GetRegistryStringValue

I probably could have defined this more succinctly with just a string variable and an inline call to Add-Type, but this worked and I had spent enough time on the problem. With the code now available to Powershell, my updated call to read the registry (which works in both 64-bit and 32-bit modes) looks like this:
$dataPath = [Dlp.Registry]::GetRegistryStringValue("SOFTWARE\Microsoft\Microsoft SQL Server\$instanceName\MSSQLServer", $null, "DefaultData")

Powershell 3.0+ Solution


In .NET 4.0, Microsoft introduced the RegistryView enumeration.  This enables 32-bit processes to read 64-bit values from .NET, and so it significantly simplifies the solution.  Here is a Powershell function that provides the same behavior:

Function Get-RegistryValue64([string] $keyPath, [string] $valueName) {
$hklm64 = [Microsoft.Win32.RegistryKey]::OpenBaseKey([Microsoft.Win32.RegistryHive]::LocalMachine, [Microsoft.Win32.RegistryView]::Registry64);
$key = $hklm64.OpenSubKey($keyPath);

if ($key) {
return $key.GetValue($valueName)
} else {
return $null
}
}

Sunday, November 10, 2013

Determine the Exact Position of an XmlReader

I was dealing with a scenario today where I wanted to read through an XML file capturing the locations of various elements so that I could come back on a second pass and process the file using random access.  To construct the XmlReader, I first created a StreamReader on the file and then passed that to the XmlReader.Create method to get my reader.

Initially I was looking at the StreamReader’s BaseStream property, but that stream reports the Position in increments of 1024 bytes due to the buffering behavior of the StreamReader.  I poked around a bit looking for a solution and when I couldn’t find one I decided to try and roll my own.

Here is what I came up with, but be warned that your mileage may vary because I’m doing non-future-proof things like reflecting on private properties of the .NET Framework classes.

The key to the approach is that the XmlReader is backed by a StreamReader which itself is backed by a FileStream.  Having access to the underlying FileStream gives us visibility into how much of the file has been read into the internal buffers of the XmlReader and the StreamReader .

Since the XmlReader uses an internal buffer of 4096 bytes, when it is first initialized from the StreamReader, it will read in the first 4096 bytes to fill its buffer.  Since the StreamReader uses an internal buffer of 1024 bytes, the XmlReader’s initialization activities will force it to retrieve four chunks of 1024 bytes.  With its own internal buffer exhausted, it will then read ahead in the FileStream another 1024 bytes.

image

The difficulty comes in determining where in the original file the XmlReader is positioned at any given moment since there are no public properties that report that information.  As it turns out, we can calculate it by reading a few additional private fields on the StreamReader and XmlReader implementation classes.  The basic formula looks (almost) like this: 

Actual XmlReader Position =
   
FileStream Position – StreamReader Buffer Size – XmlReader Buffer Size
       + XmlReader Buffer Position + StreamReader Buffer Position

Here’s the code for the XmlReader extension method:


public static class XmlReaderExtensions
{
private const long DefaultStreamReaderBufferSize = 1024;

public static long GetPosition(this XmlReader xr, StreamReader underlyingStreamReader)
{
// Get the position of the FileStream
long fileStreamPos = underlyingStreamReader.BaseStream.Position;

// Get current XmlReader state
long xmlReaderBufferLength = GetXmlReaderBufferLength(xr);
long xmlReaderBufferPos = GetXmlReaderBufferPosition(xr);

// Get current StreamReader state
long streamReaderBufferLength = GetStreamReaderBufferLength(underlyingStreamReader);
int streamReaderBufferPos = GetStreamReaderBufferPos(underlyingStreamReader);
long preambleSize = GetStreamReaderPreambleSize(underlyingStreamReader);

// Calculate the actual file position
long pos = fileStreamPos
- (streamReaderBufferLength == DefaultStreamReaderBufferSize ? DefaultStreamReaderBufferSize : 0)
- xmlReaderBufferLength
+ xmlReaderBufferPos + streamReaderBufferPos - preambleSize;

return pos;
}

#region Supporting methods

private static PropertyInfo _xmlReaderBufferSizeProperty;

private static long GetXmlReaderBufferLength(XmlReader xr)
{
if (_xmlReaderBufferSizeProperty == null)
{
_xmlReaderBufferSizeProperty = xr.GetType()
.GetProperty("DtdParserProxy_ParsingBufferLength",
BindingFlags.Instance | BindingFlags.NonPublic);
}

return (int) _xmlReaderBufferSizeProperty.GetValue(xr);
}

private static PropertyInfo _xmlReaderBufferPositionProperty;

private static int GetXmlReaderBufferPosition(XmlReader xr)
{
if (_xmlReaderBufferPositionProperty == null)
{
_xmlReaderBufferPositionProperty = xr.GetType()
.GetProperty("DtdParserProxy_CurrentPosition",
BindingFlags.Instance | BindingFlags.NonPublic);
}

return (int) _xmlReaderBufferPositionProperty.GetValue(xr);
}

private static PropertyInfo _streamReaderPreambleProperty;

private static long GetStreamReaderPreambleSize(StreamReader sr)
{
if (_streamReaderPreambleProperty == null)
{
_streamReaderPreambleProperty = sr.GetType()
.GetProperty("Preamble_Prop",
BindingFlags.Instance | BindingFlags.NonPublic);
}

return ((byte[]) _streamReaderPreambleProperty.GetValue(sr)).Length;
}

private static PropertyInfo _streamReaderByteLenProperty;

private static long GetStreamReaderBufferLength(StreamReader sr)
{
if (_streamReaderByteLenProperty == null)
{
_streamReaderByteLenProperty = sr.GetType()
.GetProperty("ByteLen_Prop",
BindingFlags.Instance | BindingFlags.NonPublic);
}

return (int) _streamReaderByteLenProperty.GetValue(sr);
}

private static PropertyInfo _streamReaderBufferPositionProperty;

private static int GetStreamReaderBufferPos(StreamReader sr)
{
if (_streamReaderBufferPositionProperty == null)
{
_streamReaderBufferPositionProperty = sr.GetType()
.GetProperty("CharPos_Prop",
BindingFlags.Instance | BindingFlags.NonPublic);
}

return (int) _streamReaderBufferPositionProperty.GetValue(sr);
}

#endregion
}

Saturday, May 4, 2013

A Performance Comparison of the .NET BinaryFormatter and the ServiceStack TypeSerializer

I have recently been doing some performance profiling on our web application and one of the areas that showed up as somewhat significant was our data serialization.  Our application uses a caching mechanism where on every request, every view model is either serialized to cache after it is built, or it is deserialized from cache in lieu of rebuilding it.  Our initial implementation used the .NET BinaryFormatter.  Given the amount of runtime execution traffic that goes through the serializer, I decided to take a closer look at optimizing it.

I first attempted to use protobuf-net, but due to the the use of dynamics and “object” types in some of our models, I decided the process to get it to work was too invasive to pursue.  Instead, I converted our serialization to use ServiceStack’s TypeSerializer.  On their wiki, they show the following results:

Serializer Size Performance
Microsoft DataContractSerializer 4.68x 6.72x
Microsoft JsonDataContractSerializer 2.24x 10.18x
Microsoft BinaryFormatter 5.62x 9.06x
NewtonSoft.Json 2.30x 8.15x
ProtoBuf.net 1x 1x
ServiceStack TypeSerializer 1.78x 1.92x

 

Their table baselines the performance of all the serializers against protobuf-net, however since I am only comparing BinaryFormatter to ServiceStack TypeSerializer, you can translate their numbers to look like this:

Serializer Size Performance
Microsoft BinaryFormatter 3.16x 4.72x
ServiceStack TypeSerializer 1x 1x

 

I ran all of our web application view models (178 different classes) through the two serializers.  Each timing in the chart below represents the total time to just serialize and deserialize each model “n” times (where n={1, 2, 3, 4, 5, 10, 25, 50, 100, 250, 500}), and were performed using “Release” builds of all classes.

The first chart shows how ServiceStack’s serializer gets off to a slow start, but surpasses the BinaryFormatter in performance at a certain point.

image

Focusing in on the performance for the first 100 serializations of each model, you can see the break even point on my data set is after about 40th serialization.

image

As the number of repeat serializations increase, the performance of ServiceStack’s serializer settles in at taking about 42% of the time, and is consistently (within my test model data) producing byte arrays that are 24% of the size of those produced by the BinaryFormatter.

To state my results in the same format as seen on ServiceStack’s wiki, you would see the following:

Serializer Size Performance
Microsoft BinaryFormatter 4.22x (vs. 3.16x) 2.27x (vs. 4.72x)
ServiceStack TypeSerializer 1x 1x

 

As you can see, I am not seeing nearly the performance increase I had been hoping for, based on their published results.  I decided to run some additional tests where I pre-warmed the serializers (serialized one of each of the models before starting the timings).  The results were certainly more favorable to ServiceStack, and it serialized my data about 2.62x faster than the BinaryFormatter over the 500 serializations, but still nowhere near their performance numbers.

image

 

image

 

image

Based on these results, I think it is indeed worth using the ServiceStack TypeSerializer if you anticipate that you’re going to serialize and deserialize the same model types many times over.  For even better performance, prepare the serializer by processing each serializable type as part of your application startup activities, but temper your expectations.  Performance is good, but not quite what you might be hoping for.