Colourblind

So here's a good one. Say we have some data in a table:

Id | Name
---+-------
 1 | Steve
 2 | Claire
 3 | Sarah 
 4 | Dave
 5 | Katie

And we run a LINQ-to-SQL query against it.

var results = dataContext.Persons.Where(x => x.Name.Contains("dave"));

It does what you'd expect it to do. Our database is case insensitive, so it matches as such.

Func<Persons, bool> predicate = x => x.Name.Contains("dave"));
var results = dataContext.Persons.Where(predicate);

Just when you think you have it licked, the above returns no rows. The problem is that the fact that we're using the predicate through a variable means that the data is grabbed from the DB before the Where() and then is filtered on the C#-side of the fence, which means your string comparisons are now case sensitive.

It's a slightly odd example (just a specific one that bit me on the arse), but it could have more serious ramifications. While this one fails pretty obviously, how about this: rather than getting 10 rows from the database you could be getting a million and then processing the lot of them in code.

Between enigmatic stuff like this, DataContext lifetime, detached objects and the eternally fun connection string pinata game, I still don't trust Linq to SQL completely.

NPOI is a .Net port of the Apache POI project for reading and writing Microsoft Office files. I've been using it recently to generate reports in Excel format so the accounting department don't burst into tears, and while I'm not exactly pushing the boundaries of XLS so far it's acquitted itself admirably.

The naming is pretty zany. SS is the namespace which contains all of the interfaces for Excel files. HSSF is for XLS files and XSSF is the OOXML version. Since the Workbook class is the means for constructing all of the other objects you can simply make a decision between XLS and OOXML right at the start and then access everything else through the interface. So once you've done the initial:

   1:  Workbook workbook = new HSSFWorkbook();

your Workbook becomes your one stop shop for creating the other necessary objects: sheets, rows and cells.

   1:  Sheet sheet = workbook.CreateSheet("SheetName");

   2:  Row row = sheet.CreateRow(rowIndex);

   3:  Cell cell = row.CreateCell(columnIndex);

   4:  cell.SetCellValue("foo");

I didn't find any gotchas in their object model, so it was pretty simple to produce code for creating sheets based on DataTables:

   1:  public static Sheet Write(Workbook workbook, string sheetName, DataTable data)

   2:  {

   3:      sheetName = String.IsNullOrEmpty(sheetName) ? data.TableName : sheetName;

   4:      sheetName = String.IsNullOrEmpty(sheetName) ? "Sheet" : sheetName;

   5:      Sheet sheet = workbook.CreateSheet(sheetName);

   6:   

   7:      Row header = sheet.CreateRow(0);

   8:      for (int i = 0; i < data.Columns.Count; i ++)

   9:          header.CreateCell(i).SetCellValue(data.Columns[i].ColumnName);

  10:   

  11:      int rowIndex = 0;

  12:      foreach (DataRow dataRow in data.Rows)

  13:      {

  14:          Row row = sheet.CreateRow(rowIndex + 1);

  15:          for (int i = 0; i < data.Columns.Count; i ++)

  16:              row.CreateCell(i).SetCellValue(data.Rows[rowIndex][i].ToString());

  17:          rowIndex++;

  18:      }

  19:   

  20:      return sheet;

  21:  }

And also for objects, using a little reflection:

   1:  public static Sheet Write(Workbook workbook, string sheetName, IEnumerable data)

   2:  {

   3:      IEnumerator foo = data.GetEnumerator();

   4:      foo.MoveNext();

   5:      Type t = foo.Current.GetType();

   6:   

   7:      Sheet sheet = workbook.CreateSheet(sheetName);

   8:   

   9:      Row header = sheet.CreateRow(0);

  10:      PropertyInfo[] properties = t.GetProperties();

  11:      for (int i = 0; i < properties.Length; i++)

  12:          header.CreateCell(i).SetCellValue(properties[i].Name);

  13:   

  14:      int rowIndex = 0;

  15:      foreach (object o in data)

  16:      {

  17:          Row row = sheet.CreateRow(rowIndex + 1);

  18:          for (int i = 0; i < properties.Length; i++)

  19:              row.CreateCell(i).SetCellValue(properties[i].GetValue(o, null).ToString());

  20:          rowIndex++;

  21:      }

  22:   

  23:      return sheet;

  24:  }

The above code is ripe for improvement. It's all currently stringly typed, although NPOI will allow us to specify types for each cell. It would also be simple enough to write up a property attribute to exclude a property in the same vein as XmlIgnore and ScriptIgnore to reduce some of the cruft in the second example. Or just use anonymous types.

We've got a client who is doing IP-based geolocation when a user first hits to the site via a third-party service. They've got a very limited number of lookups they can make per day, and it turns out the hosting company's heartbeat service is eating up a fair amount of these. What we wanted is to filter out calls made from private IP addresses. I was expecting to come across something in .NET to do this, but I'm damned if I can find it.

According to RFC 1918 we have 3 blocks of IPs in the private address space:

• 10.0.0.0/24
• 172.16.0.0/20
• 192.168.0.0/16

Since the easiest way to perform the comparison is going to be using the IP in integer format, first we need a method for converting an IP from the more widely used dotted octet. I'm pretty much rewriting the System.Net.IPAddress.Address property, which has been deprecated since the dawn of time.

   1:  using System;

   2:   

   3:  public static class IpUtils

   4:  {

   5:      private static readonly long[] _privateBlocks;

   6:   

   7:      static IpUtils()

   8:      {

   9:          _privateBlocks = new long[]

  10:          { 

  11:              IpToInteger("10.0.0.0"), 

  12:              IpToInteger("172.16.0.0"), 

  13:              IpToInteger("192.168.0.0") 

  14:          };

  15:      }

  16:   

  17:      public static long IpToInteger(string ip)

  18:      {

  19:          long result = 0;

  20:          string [] octets = ip.ToString().Split(new char [] { '.' });

  21:          result = Convert.ToInt64(

  22:                    Int32.Parse(octets[0]) * Math.Pow(2, 24)

  23:                  + Int32.Parse(octets[1]) * Math.Pow(2, 16)

  24:                  + Int32.Parse(octets[2]) * Math.Pow(2, 8)

  25:                  + Int32.Parse(octets[3]) * Math.Pow(2, 0));

  26:          return result;

  27:      }

  28:   

  29:      public static bool IsPrivateIp(string ip)

  30:      {

  31:          long ipInt = IpToInteger(ip);

  32:          return (ipInt >= _privateBlocks[0] && ipInt < _privateBlocks[0] + Math.Pow(2, 24))

  33:              || (ipInt >= _privateBlocks[1] && ipInt < _privateBlocks[1] + Math.Pow(2, 20))

  34:              || (ipInt >= _privateBlocks[2] && ipInt < _privateBlocks[2] + Math.Pow(2, 16));

  35:      }

  36:  }

This is coded for the sake of readability and is very inefficient. All of the Math.Pow calls are constants (does the compiler work this out? Or the JIT? Answers on a postcard) and we could swap out all of that _privateBlocks stuff in favour of 167772160, 2886729728 and 3232235520 if you want. And using bit-shifts rather than multiplication would probably save you a few micro-seconds . . .

So what with my work on Oh Teh Noes I was wondering if I could co-opt it to do performance monitoring. As it happens, I couldn't figure out how to do it without doing some serious square-pegging, but it did get me thinking about how I would like to attack the problem. And as is so often that case that eventually turned into me actually attacking the problem. That happens a lot. I nerd snipe myself daily.

And thus Regarding Your Performance was born. It consists of a Windows service which sits on the host machine and farms basic performance data, and a HTML page which collects that data from a list of hosts via AJAX and drops it into some graphs. It became an excuse for me to learn some new things, so I thought I'd share the experience. This is a fairly high-speed tour so you might want to saunter over to Github and grab the source

InstallUtil Can Die in a Fire

I wanted the agent (the side of the system which resides on the target host and harvests delicious, delicious performance information) to run as a Windows service. But there's one particular element of Windows services that I find especially uncomfortable . . .

InstallUtil.exe can piss right off.

I've been packaging that dowdy little app with all of my Windows Services for years now and enough is enough. It's time to find a more elegant solution. Fire up Reflector and it quickly becomes apparent that InstallUtil doesn't actually do a lot. In essence it simply creates a ManagedInstaller, which itself is a thin wrapper around a TransactedInstaller. By taking that code and then adding on the contents of the usual Service designer files you can roll your own pretty easily. I ended up using command line arguments to install and uninstall the service, resulting in a Main method like this:

   1:  static void Main(string[] args)

   2:  {

   3:      if (args.Length > 0)

   4:      {

   5:          if (args[0] == "-i" || args[0] == "-u")

   6:          {

   7:              ServiceInstaller serviceInstaller = new ServiceInstaller();

   8:              serviceInstaller.ServiceName = "Performance.Agent.Service";

   9:              serviceInstaller.StartType = ServiceStartMode.Automatic;

  10:              serviceInstaller.DisplayName = "Colourblind Performance Agent";

  11:              serviceInstaller.Description = "Agent for the Colourblind performance monitor";

  12:   

  13:              ServiceProcessInstaller processInstaller = new ServiceProcessInstaller();

  14:              processInstaller.Account = ServiceAccount.LocalSystem;

  15:              processInstaller.Username = null;

  16:              processInstaller.Password = null;

  17:   

  18:              TransactedInstaller installer = new TransactedInstaller();

  19:              installer.Installers.Add(processInstaller);

  20:              installer.Installers.Add(serviceInstaller);

  21:              installer.Context = new InstallContext("install.log", null);

  22:              installer.Context.Parameters.Add("assemblypath", Assembly.GetCallingAssembly().Location);

  23:   

  24:              if (args[0] == "-i")

  25:                  installer.Install(new Hashtable());

  26:              else if (args[0] == "-u")

  27:                  installer.Uninstall(null);

  28:          }

  29:      }

  30:      else

  31:      {

  32:          ServiceBase[] ServicesToRun;

  33:          ServicesToRun = new ServiceBase[] 

  34:          { 

  35:              new Service() 

  36:          };

  37:          ServiceBase.Run(ServicesToRun);

  38:      }

  39:  }

Next up, how to poll our host system for performance information.

Real-time System Information Voyeurism

PerformanceCounter is the .NETified way of querying the Windows for performance metrics.

   1:  PerformanceCounter cpu = new PerformanceCounter("Processor", "% Processor Time", "_Total");

   2:  PerformanceCounter memory = new PerformanceCounter("Memory", "Available MBytes");

   3:  PerformanceCounter requestsPerSecond = new PerformanceCounter("ASP.NET Applications", "Requests/Sec", "__Total__");

The strings you use to instantiate PerformanceCounters come firmly in the realm of magic. They're not terribly well documented and in all honesty you're better off just opening up Windows' Performance Monitor (WinKey-R 'perfmon') and seeing which counters you can add through that. It'll even give you a convenient description of quite what some of the more esoteric ones mean. Failing that you can enumerate them yourself

It would make a lot of sense to create our performance counters in the constructor, but it turns out they can spend a lot of time sitting around doing nothing which then causes net start to report a timeout. This is not the case if we move the object instantiation to an OnStart event of the service though, so that's what I ended up doing. The last thing on the startup todo list is to get the total amount of physical memory for the host, and for this we take a brief forray into the nightmare realm of WMI.

   1:  ManagementObjectSearcher wmi = new ManagementObjectSearcher("select * from Win32_ComputerSystem");

   2:  foreach (ManagementObject o in wmi.Get())

   3:      TotalPhysicalMemory += Convert.ToSingle(o["TotalPhysicalMemory"]) / (1024 * 1024);

Now we simply poll the counters every second and store the results in a dictionary.

   1:  Status["CpuUsage"] = cpu.NextValue().ToString();

   2:  Status["MemoryUsage"] = ((TotalPhysicalMemory - memory.NextValue()) * 100 / TotalPhysicalMemory).ToString();

   3:  Status["RequestsPerSecond"] = requestsPerSecond.NextValue().ToString();

All well and good, but now our service has shut-in syndrome, and that frankly will not fly.

Bare Bones HTTP

At this point I had only vague ideas about the form that my client would take. With that in mind I wanted something flexible enough that my later decisions wouldn't be affected by my earlier design choices. With that in mind I decided to go for HTTP as my application protocol, and JSON as my information format.

.NET's HttpListener provides an itty-bitty HTTP server, capable of asynchronous responses and not a lot else. That said, it's perfect for small HTTP-enabled apps like this.

   1:  string prefix = String.Format("http://+:{0}/", ConfigurationManager.AppSettings["ServerPort"]);

   2:  HttpListener httpListener = new HttpListener();

   3:  httpListener.Prefixes.Add(prefix);

   4:  httpListener.Start();

   5:   

   6:  // 10 worker threads

   7:  for (int i = 0; i < 10; i ++)

   8:      httpListener.BeginGetContext(new AsyncCallback(RequestCallback), null);

Besides that, you're on your own. You handle each request as you see fit in your request handler, and once you've done what you need to you spool up another listener to replace the one you've just 'used', then wait for more calls to come rolling in.

   1:  private void RequestCallback(IAsyncResult result)

   2:  {

   3:      HttpListenerContext context = null;

   4:      try

   5:      {

   6:          context = HttpListener.EndGetContext(result);

   7:   

   8:          string output = "{";

   9:          bool foo = false;

  10:          foreach (string key in Status.Keys)

  11:          {

  12:              output += String.Format("{2}\n\t{0} : '{1}'", key, Status[key], foo ? "," : "");

  13:              foo = true;

  14:          }

  15:          output += "\n}";

  16:   

  17:          Encoding encoding = new UTF8Encoding(false); // Whenever I have UTF8 problems it's BOM's fault

  18:          byte[] outputBytes = encoding.GetBytes(output);

  19:   

  20:          context.Response.OutputStream.Write(outputBytes, 0, outputBytes.Length);

  21:      }

  22:      finally

  23:      {

  24:          if (context != null && context.Response != null)

  25:              context.Response.Close();

  26:          HttpListener.BeginGetContext(new AsyncCallback(RequestCallback), null);

  27:      }

  28:  }

Now we can consume the one-trick web service via AJAX requests spawned from an HTML page!

That's Not a Client - It's an HTML Page

Damn right it's an HTML page. Why bother with apps and servers and all that cruft when all you need is a few lines of Javascript? The 'client' is a simple HTML page containing a list of active agents which are polled every second and the results are used as the data for a set of sparklines graphs care of a jQuery plugin. The only gotcha here is the scale on the requests/sec graph. We can treat CPU and memory usage as percentages, but requests/sec is an absolute value so you may need to tweak the y scale accordingly.

Hmmm. Hang on a minute . . .

Oh crap, I forgot about the Same Origin Policy

Every time I think of something cool to do with a web client the Same Origin Policy comes and bites me on the arse. SOP states that an XmlHttpRequest can only be sent to the same domain as the Javascript file which initiates it, and therefore prevents the kind of cross-domain AJAX shenanigans we're attempting here.

JSONP is a cheeky little hack designed to circumvent this very restriction. Rather than being just Javascript object literal notation, as is the case with JSON, JSONP actually returns a javascript source file which is fetched by the host page using a script block, dodging the SOP. When making a call to a JSONP-aware service you provide the name of the function that is ready and waiting to process the response, and the service wraps it's JSON response in a call to that function so that when the file is fetched by the client the response handler is executed. Conceptually it feels weird as you're essentially pulling a pushing, or maybe pushing a pull, but getting down to the brass tacks this means we have to add two lines to our request handler:

   1:  string callback = context.Request.QueryString["callback"];

   2:  output = String.Format("{0}({1})", callback, output);

On the client side there is a handy jQuery JSONP plugin which makes for a similarly smooth transition.

~fin

Those are the bits that interested me, at least. Feel free to grab the code and have an explore. It's a testament to the .NET framework that it packs so much into so few lines of working code and made for a fun train-journey-home-from-work project over a few days.

I scrawled about Oh Teh Noes last month and said I'd post more, only to get distracted by fractals (it's a long story) like the technical butterfly that I am, but I return to you here to deliver on my promise. So let's talk about Oh Teh Noes' simple-enough-to-work plugin system.

To recap: Oh Teh Noes is a host program for a set of plugins that perform simple checks on the local machine. These checks are called tasks, and each plugin takes the form of a .NET assembly containing one or more tasks. Tasks all inherit from an abstract base class called, imaginatively enough, Task. This is not a direct copy of the code used in Oh Teh Noes - I've stripped out some irrelevancies.

   1:  public abstract class Task

   2:  {

   3:      public abstract void Run();

   4:  }

A (slightly simplified) task to check the remaining space left on the machine's physical drives would look like this:

   1:  class DiskSpaceTask : Task

   2:  {

   3:      public override void Run()

   4:      {

   5:          foreach (DriveInfo drive in DriveInfo.GetDrives())

   6:          {

   7:              if (drive.DriveType == DriveType.Fixed)

   8:              {

   9:                  if (drive.AvailableFreeSpace < 1024 * 1024 * 1024)

  10:                  {

  11:                      // Less than a GB remains! Alert the authorities!

  12:                  }

  13:              }

  14:          }

  15:      }

  16:  }

And now, for a brief asside . . .

Best Bad Solution Available

At this point what I want to do is define a property in the abstract class that must be overridden in the derived classes, but is part of the class, not the objects. What I need is an abstract const, which is a real shame as they don't exist. Or an abstract static property, which doesn't exist either. Hmmm.

Using an attribute here is a pretty clear abuse of the concept. An attribute should be a class agnostic property, but here we only want to be able to apply is to classes which inherit from Task. We can check that at runtime but it's hardly ideal.

The other alternative is to have a static property on the Task class which is also defined in concrete Task classes and then explicitly 'overriden' using the new keyword. Again, no compile-time cover and you need to know about it in advance. Alternatively have the property in base class throw an exception, which is a pretty serious runtime middle finger. And in using properties we will have to instantiate an object based on the Type, simply to get the value of the property when we're enumerating.

Plan D is an abstract property. And it's really not a property of an object. It's a property of a class. And the instantiation stuff still stands.

So attributes it is. Lacking a better alternative it's the lesser of evils. If anyone has any alternatives please let me know, as frankly this kind of thing sets my teeth on edge, but my pragmatic side just about outweights my semantic purity OCD side.

What was I on about?

Oh that's right. Plugins.

The Already-Maligned Attribute

So now I've grudgingly decided to go the attribute route we'd better bite the bullet and hop to it.

   1:  public class TaskAttribute : Attribute

   2:  {

   3:      // A friendly name for the task

   4:      public string TypeName

   5:      {

   6:          get;

   7:          set;

   8:      }

   9:   

  10:      public TaskAttribute(string typeName)

  11:      {

  12:          TypeName = typeName;

  13:      }

  14:  }

We'll use this attribute to decorate the Tasks and allow our host program to fetch meaningful names for them. In the future it will also be on-hand in case we ever need somewhere to stuff more class-level metadata.

Plugin Discovery

When Oh Teh Noes is run it collects a list of all of the DLL files in the program's plugin subdirectory. It then gets a list of types from each of the assemblies and iterates through those, looking for classes which are subclasses of our Task class and are decorated with TaskAttribute. When we've found one we add the class' Type to a dictionary, using the TaskName of the TaskAttribute as a key. The code looks a lot like this:

   1:  Dictionary<string, Type> plugins = new Dictionary<string, Type>();

   2:   

   3:  DirectoryInfo pluginDirectory = new DirectoryInfo(Path.Combine(

   4:                                      AppDomain.CurrentDomain.SetupInformation.ApplicationBase, 

   5:                                      "plugins"));

   6:                                      

   7:  foreach (FileInfo file in pluginDirectory.GetFiles("*.dll"))

   8:  {

   9:      Assembly assembly = Assembly.LoadFile(file.FullName);

  10:      foreach (Type type in assembly.GetTypes())

  11:      {

  12:          object[] attributes = type.GetCustomAttributes(typeof(TaskAttribute), true);

  13:          if (type.IsSubclassOf(typeof(Task)) && attributes != null)

  14:          {

  15:              foreach (object attribute in attributes)

  16:              {

  17:                  if (attribute is TaskAttribute)

  18:                  {

  19:                      TaskAttribute taskAttribute = (TaskAttribute)attributes[0];

  20:                      plugins[taskAttribute.TypeName] = type;

  21:                  }

  22:              }

  23:          }

  24:      }

  25:  }

Cocked, locked and ready to rock

So we've loaded our assemblies, enumerated their types, loaded the relevant classes (grumble grumble grumble) and are ready to roll. Next step is to create instances of the plugins themselves. We can use the dictionary we built up in the last method to look up Types by their friendly name, and then create them using the Activator class.

   1:  Type pluginType = plugins["TaskName"];

   2:  object[] args = new object[];

   3:  Task task = (Task)Activator.CreateInstance(pluginType, args);

Hopefully that's covered enough to get people started. If you want to see it in situ then please drop by the OhTehNoes GitHub repository.