Colourblind

Welcome to Colourblind.

This is the personal web space of Tom Milsom. As much as possible everything is free (as in speech and as in beer).


Make text: Smaller Bigger

Mapping Dependencies with Mono Cecil

Posted by Tom on 15/04/2012 10:12:13

So a while back I got curious about how a .NET assembly would look if you could map the internal structure. There is code around there to create maps of references at an assembly level, but I wanted something more fine-grained. If a class A contains a reference to class B then we could say that A depends on B. Now we look at B and see that it references C and D and eventually we can build a directed graph of all these dependencies.

I decided to take Mono Cecil for a spin this time, simply in order to try something new. With hindsight I don't think it would have been half as easy to catch the unassigned local variable case using .NET reflection since it doesn't have any real abstraction at the IL level. But I'm getting ahead of myself. Let's start from the top.

Enumerating Dependencies

First of all: what form can these references take? At the class level:

  • Inheritance and Implementation - Each class can inherit once and implement to its heart content, so we need to look for both of these when we check each class.
  • Properties and fields - Pretty obvious here. We'll need to check the types on each property and field.

At the method level we've got a few things to check:

  • Parameters
  • Local variables
  • Unassigned local variables

I'm going to expound on that last one because it doesn't make any sense, but I've spent the last five minutes wracking my brains for a better term and I'm now bored and stumped, so it's example time.

   1:  class TestClass
   2:  {
   3:      void TestCase()
   4:      {
   5:          var foo = new HereIsMyClass();
   6:          var bar = new HereIsMyReader(foo);
   7:      }
   8:  }

This acts as expected. However . . .

   1:  class TestClass
   2:  {
   3:      void TestCase()
   4:      {
   5:          var boo = new HereIsMyReader(new HereIsMyClass());
   6:      }
   7:  }

In the first example the HereIsMyClass dependency will be picked up due to the local variable, but in the second it won't. The HereIsMyClass class itself will still be processed since it's on the parameter list of the HereIsMyReader constructor, but we will miss the association between TestClass and HereIsMyClass if we only check for local variables. Curses!

So instead we're grinding through all of the instructions in every method. Nasty, but I can't find a neater alternative. And to be honest once we're doing that we don't need to check parameters and local variables any more either since they'll be picked up while looking for the last case.

And now for some cases best categorised under 'other' . . .

Attributes

Attributes can be specified at global scope (to specify attributes on the containing assembly or module) and for type-declarations (Section 9.5), class-member-declarations (Section 10.2), interface-member-declarations (Section 13.2), struct-member-declarations (Section 11.2), enum-member-declarations (Section 14.3), accessor-declarations for properties (Section 10.6.2), event-accessor-declarations (Section 10.7.1), and formal-parameter-lists (Section 10.5.1).

So, basically, anywhere. Less facetiously, we're going to have to check for attributes on class, properties, fields and methods.

Generics

The next irritating corner case that needs handling: generics. Checking the GenericParameters property of the class or method will give us back a list of constructed generic types and that is all well and good, but say one of those parameters is akin to the deliciously convoluted example below.

IDictionary<string, IList<Tuple<int, Tuple<MyClass, string, decimal>, SomeOtherClass>>>>

We're going to have to walk a tree in order to get back all of the relevant types. Once we get a generic parameter we're going to need to recurse into each of its own generic parameters to make sure we've caught them all.

Compiler Generated Types

In some situations (anonymous types and LINQ spring to mind) the compiler itself will create types. These don't have any relevence to the job at hand, so let's filter them out. This is simply a case of checking to see if a Type has the CompilerGeneratedAttribute assigned. Please note that I've not actually done this yet, but it's the next job on the list!

The Code!

As usual all of this is on GitHub under the charming name Spaghetti Detector, which given an assembly it will enumberate the classes in it and check them for references to other types using the above strategy, until it builds up a map of the module as a whole. It'll happily jump into other assemblies, and by default it filters out the System and Microsoft namespaces, or we'd be here all day. It can also be limited to only chase references to a certain depth.

Finally, if the JSON output of the sample application seems a little unusual it's because I'm serialising into a format designed to be loaded by another piece of code I wrote around the same time: a force-based graph renderer in Javascript. Next post (once I stop playing Tribes Ascend ;-) will probably be exploring some of the implementation details of that, since it kept me amused for a while and got me started with the Raphael SVG Javascript library.

Thoughts on Cecil

Unless you really get down into the IL of your target code then I don't think you'll see a lot of difference between .NET Reflection and Mono Cecil. They're both well equipped for code inspection. Conceptually I still find Cecil's distinction between a TypeDefinition and a TypeReference a little hard to grasp, but it started to make more sense once I began to think of it as similar to the relationship between classes and objects. I dunno. Maybe I'm just being fick.

There doesn't seem to any documentation, which is a bit hairy for a project of this size and scope, but the community is very active and most of the big questions have already been answered on the forums.

The only genuine problem I ran into was using NUnit with Cecil on 64 bit Win7, which eventually led me to the linked Gist and was all fixed in short order. The VS solution files are right there in the source downloads, so rebuilding using the Microsoft .NET compiler is crazy simple.

To be continued . . .

If anyone knows of a slicker alternative (what I really want is a TypeDefinition.GetReferencedTypes or something) then please drop me a line. In the mean time, hopefully this will save someone else some work. I'm sure there are some more zany cases that I haven't covered. It's bounded by my imagination, and my knowledge of IL is woefully inadaquate.

That said, this project has piqued my interest in code analysis (and it has some interesting visualisation opportunities on the side) so expect more!

Comments (0)

Pesky Harmonics - Spectral Visualisation of Audio

Posted by Tom on 01/03/2012 22:17:38

Whoops. I got a bit distracted and I forgot I was sitting on this one.

A few weeks back I was thinking of ways to visually represent music and the most obvious one was frequency over time. Pesky Harmonics is the result and it takes an MP3 file as input and spits out a PNG showing the time along the X axis, freqency along the Y and with amplitude as the pixel brightness.

Here's a few samples (click the image to listen to the tune on YouTube):


Guitar intro to Better Living Through Chemistry by Queens of the Stone Age


Only by Nine Inch Nails (about 2:15 in)


Cosmonaut by At the Drive In (around 1:33 - the two dark patches at the bottom are where the guitar and bass plays rests)

It's a bit noisy and there are quite a few restrictions on the input format (a bit-depth of 16, 2 channels - one of which is discards completely). The result is also ripe for some improvement (the brightness needs tweaking, the resolution is pretty crappy at low frequencies, I'd rather have a logarithmic frequency scale) but it's possible to make out some sonic features and I imagine I'll keep chipping away at the idea.

As usual, the smart parts are provided by better people than I. I'm using libav to load the audio file. FFT is provided by KissFFT. Image writing is handled my libpng.

And also as usual - it's available on GitHub.

Comments (0)

Linq to SQL Deferred Loading Gotcha

Posted by Tom on 17/01/2012 09:10:32

So here's a good one. Say we have some data in a table:

Id | Name
---+-------
 1 | Steve
 2 | Claire
 3 | Sarah 
 4 | Dave
 5 | Katie

And we run a LINQ-to-SQL query against it.

var results = dataContext.Persons.Where(x => x.Name.Contains("dave"));

It does what you'd expect it to do. Our database is case insensitive, so it matches as such.

Func<Persons, bool> predicate = x => x.Name.Contains("dave"));
var results = dataContext.Persons.Where(predicate);

Just when you think you have it licked, the above returns no rows. The problem is that the fact that we're using the predicate through a variable means that the data is grabbed from the DB before the Where() and then is filtered on the C#-side of the fence, which means your string comparisons are now case sensitive.

It's a slightly odd example (just a specific one that bit me on the arse), but it could have more serious ramifications. While this one fails pretty obviously, how about this: rather than getting 10 rows from the database you could be getting a million and then processing the lot of them in code.

Between enigmatic stuff like this, DataContext lifetime, detached objects and the eternally fun connection string pinata game, I still don't trust Linq to SQL completely.

Tags: dotNet

Comments (0)

Making MSBuild in NANT act like MSBuild in Visual Studio

Posted by Tom on 21/12/2011 09:45:37

So I was making some NANT scripts for an ASP.NET MVC project.

Rather than going through the rigmarole of maintaining references and dependencies in two places (the VS project file and the NANT scripts), this time round I started using the <msbuild> task from nantcontrib instead of invoking the compiler directly using the plain old <csc> task. After making sure I set my OutDir I run NANT and get a failed build while copying a file, which is odd because I don't want MSBuild doing anything of the sort. It builds fine from Visual Studio, but after a little investigation it turns out that MSBuild is attempting to copy a file which is in the project but is missing from the file system. It's trying to drop it, along with the rest of the project contents, into _PublishedWebsites which has attaching itself to my output directory, like a warty blemish or tumour on my otherwise pristine build.

By setting the verbosity level of MSBuild to Diagnostic in both NANT and Visual Studio I could diff the two outputs and see where the processes started to diverge. The output of the NANT ran to a svelte 26k lines, but a quick search on '_PublishedWebsites' turns up the part of the log which includes the failing build. The copy operation is happening within a _CopyWebApplication target. Time to do a quick search in the Visual Studio MSBuild log . . .

Aha! Here's the slippery little devil:

Target "_CopyWebApplication" skipped, due to false condition; (!$(Disable_CopyWebApplication) And '$(OutDir)' != '$(OutputPath)') was evaluated as (!False And 'bin\' != 'bin\').

But why? Why would you do that? What is the OutputPath even supposed to be? According to MSBuild\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets . . .

<!--
============================================================
_CopyWebApplication

This target will copy the build outputs along with the content files into a 
_PublishedWebsites folder.

This Task is only necessary when $(OutDir) has been redirected to a folder 
other than ~\bin such as is the case with Team Build.

The original _CopyWebApplication is now a Legacy, you can still use it by 
setting $(UseWPP_CopyWebApplication) to true. By default, it now change to 
use _WPPCopyWebApplication target in Microsoft.Web.Publish.targets. It allow 
to leverage the web.config trsnaformation.
============================================================
-->

Fine. Whatever. Regardless of the rationale, if you set your OutDir and OutputPath properties to the same value in the NANT <msbuild> task:

<msbuild project="${dir.src}/${project::get-name()}/${project::get-name()}.csproj"
      target="Rebuild" verbosity="Minimal">
   <property name="Configuration" value="Release" />
   <property name="OutDir" value="../../${dir.build}/${project::get-name()}/bin/" />
   <property name="OutputPath" value="../../${dir.build}/${project::get-name()}/bin/" />
</msbuild>

Then your case of _PublishedWebsites should clear up in no time. (Alternatively you could set the Disable_CopyWebApplication property, but I'm attempting to get our NANT builds as close to our VS builds as possible.) As an addendum, once I knew what I was looking for I found quite a bit of confusion related to OutputPut, including a blog post from Mark Needham who also ran into problems related to OutputPath when tweaking OutDir and who turned up this nugget from MSDN:

OutputPath: This property is typically specified in the project file and resembles OutDir. OutputPath has been deprecated and OutDir should be used instead whenever possible.

At this point I stopped caring and reversed out of the rabbit hole.

Tags: ASP, NAnt

Comments (0)

Client-side templating with Handlebars.js

Posted by Tom on 30/11/2011 22:43:38

So back to data molestation in Javascript.

Now we have our post-processed data it's time to display it to the user.

Templating in Javascript seems to be in vogue at the moment, so there's a pretty decent selection to choose from. Originally I was using jquery-tmpl, but since that project got abandoned in favour of something more awesomer it was time to have a look at alternatives. I initially fell upon Mustache (to be honest, mainly because of the name) but founds myself butting up against the stringently 'logic-less' approach. I guess if you are stricter with your view-models it's less of an issue, but I just found it a little restrictive. Handlebars is an evolution of Mustache from Yehuda Katz (of Rails fame), and appears to scratch the itches that Mustache leaves me with in addition to being a superset of the Mustache syntax. Also, the name is still cool.

A quick note, before we get started. On my Mustache-based travels I stumbled across ICanHaz.js, which simplifies the process and has some really nice ideas. For example, on page load it grabs all of the templates and adds them to a globally available collection for ease of use. This becomes even more useful in Handlebars since it requires an additional compilation step. I ended up writing something very similar but considerably more primitive, and here it is:

   1:  var BandleHars = {
   2:      templateCache : {},
   3:      init : function() {
   4:          var templateList = $('script[type="text/html"]');
   5:          templateList.each(function(index, value) { BandleHars.templateCache[value.id] = Handlebars.compile($(value).html()); });
   6:          templateList.remove();
   7:      },
   8:      render : function(templateName, data) {
   9:          return this.templateCache[templateName](data)
  10:      }
  11:  };
  12:   
  13:  $(function() { BandleHars.init(); });

Right, let's get back to templating. First we need our data. I used a grab of the first few posts of this very site, pretty much as it's dumped out by System.Web.Script.Serialization.JavaScriptSerializer.

   1:  var data = [ 
   2:    {
   3:      "Comments" : [
   4:          { "Body" : ". . . the comment system.",
   5:            "CommentId" : 1,
   6:            "DateCreated" : "/Date(1248627151247)/",
   7:            "GravatarUrl" : "http://www.gravatar.com/avatar/d41d8cd98f00b204e9800998ecf8427e?d=identicon&s=64",
   8:            "Heading" : "Testing . . .",
   9:            "Poster" : "Tom",
  10:            "Website" : "http://monochromacy.net/"
  11:          },
  12:          { "Body" : "n/t",
  13:            "CommentId" : 2,
  14:            "DateCreated" : "/Date(1254127922440)/",
  15:            "GravatarUrl" : "http://www.gravatar.com/avatar/d41d8cd98f00b204e9800998ecf8427e?d=identicon&s=64",
  16:            "Heading" : "Notification Test",
  17:            "Poster" : "Test",
  18:            "Website" : ""
  19:          }
  20:        ],
  21:      "DateCreated" : "/Date(1248624603550)/",
  22:      "DateLastTouched" : "/Date(1248624603550)/",
  23:      "Heading" : "Oh hai!",
  24:      "PostId" : 1,
  25:      "Slug" : "Oh-hai",
  26:      "Summary" : "This weblog will be largely concerned with ASP.NET, C# and ...",
  27:      "Tags" : [ "Waffle" ]
  28:    },
  29:    /* SNIP */
  30:    { 
  31:      "Comments" : [  ],
  32:      "DateCreated" : "/Date(1261657386630)/",
  33:      "DateLastTouched" : "/Date(1261657386630)/",
  34:      "Heading" : "Introducing Compares Favourably",
  35:      "PostId" : 8,
  36:      "Slug" : "Introducing-Compares-Favourably",
  37:      "Summary" : "So I made a database schema comparison tool for MSSQL 2005.\nCompares Favourably\n<p ...",
  38:      "Tags" : [ "ComparesFavourably", "dotNet", "Projects" ]
  39:    },
  40:  ];
  41:   
  42:  var dataSet = new DataContainer(data, 3);
  43:  dataSet.update = function() 
  44:  {
  45:      // We'll come back to this . . .
  46:  };
  47:  dataSet.sortProperty = 'DateCreated';
  48:  $(function() { dataSet.update(); });

Here I'm setting up the DataContainer with three items to a page, sorted on the created data and then running the (currently empty) update function once the DOM has been loaded. Now it's time for some markup to display and manipulate our data.

   1:  <label>Paging</label>
   2:  <ul class="horizontal">
   3:      <li><a href="#" onclick="return dataSet.pageDown();">Page down</a></li>
   4:      <li><a href="#" onclick="return dataSet.pageUp();">Page up</a></li>
   5:  </ul>
   6:   
   7:  <label>Sort by</label>
   8:  <ul class="horizontal">
   9:      <li><a href="#" onclick="return dataSet.setSortProperty('Heading');">Heading</a></li>
  10:      <li><a href="#" onclick="return dataSet.setSortProperty('Summary');">Summary</a></li>
  11:      <li><a href="#" onclick="return dataSet.setSortProperty('Slug');">Slug</a></li>
  12:      <li><a href="#" onclick="return dataSet.setSortProperty('DateCreated');">Date Created</a></li>
  13:  </ul>

Nice and simple. This will page the data up and down, allow us to change the sort criteria, and includes some bonus hey-that's-not-what-label-elements-are-for abuse. Sue me. I figure a tag filter would make sense in this context and since this is dependant on the data it will need to be templated. First we need a container for our filter buttons:

   1:  <label>Tags</label>
   2:  <ul class="horizontal" id="tagsFilter"></ul>

And now a template for creating the filter links:

   1:  <script id="tagsFilterTemplate" type="text/html">
   2:  {{#each data}}<li><a href="#" onclick="return dataSet.addFilter('Tags', '{{this}}');">{{this}}</a></li>{{/each}}
   3:  <li><a href="#" onclick="return dataSet.removeFilter('Tags');">remove</a></li>
   4:  </script>

The above Bandlehars code (I hate that name already) will fetch and compile this template on body load, so now we just need to add this to our DataContainer update function to include the template rendering:

   1:  dataSet.update = function() 
   2:  {
   3:      // There's still more to come!
   4:      $('#tagsFilter').html(Bandlehars.render('tagsFilterTemplate', { data : dataSet.getPropertyValues('Tags') }));
   5:  };

Now, once the page is loaded and then whenever it is filtered, the list of available tags will be updated. This means we can drill down further using additional filters. Worth noting is that we're wrapping our data in another object, since Handlebars doesn't accept raw arrays as a data context. So that's the filters sorted. But we still can't see the data itself. It's time for another container and another template.

   1:  <div id="data"></div>
   2:   
   3:  <script id="dataTemplate" type="text/html">
   4:  {{#each data}}
   5:  <div>
   6:      <h2><a href="http://monochromacy.net/Post/{{Slug}}.aspx">{{Heading}}</a></h2>
   7:      <p><em>{{DateCreated}}</em></p>
   8:      <p>{{Summary}}</p>
   9:      <label>Tags</label>
  10:      <ul class="horizontal">{{#each Tags}}<li><a href="#" onclick="return dataSet.addFilter('Tags', '{{this}}');">{{this}}</a></li>{{/each}}</ul>
  11:  </div>
  12:  {{/each}}
  13:  </script>

Now to add it to our DataContainer update function.

   1:  dataSet.update = function() 
   2:  {
   3:      $('#data').html(Bandlehars.render('dataTemplate', { data : dataSet.getData() }));
   4:      $('#tagsFilter').html(Bandlehars.render('tagsFilterTemplate', { data : dataSet.getPropertyValues('Tags') }));
   5:  };

Well it's a start. What's next? How about comments.

   1:  <script id="dataTemplate" type="text/html">
   2:  {{#each data}}
   3:  <div>
   4:      <h2><a href="http://monochromacy.net/Post/{{Slug}}.aspx">{{Heading}}</a></h2>
   5:      <p><em>{{DateCreated}}</em></p>
   6:      <p>{{Summary}}</p>
   7:      <label>Tags</label>
   8:      <ul class="horizontal">{{#each Tags}}<li><a href="#" onclick="return dataSet.addFilter('Tags', '{{this}}');">{{this}}</a></li>{{/each}}</ul>
   9:  </div>
  10:  <br />
  11:  {{#if Comments.length}}
  12:  <div class="comments">
  13:      <h4>Comments</h4>
  14:      <ul>
  15:  {{#each Comments}}
  16:          <li>{{Heading}} - {{Body}} - {{Poster}}</li>
  17:  {{/each}}
  18:      </ul>
  19:  </div>
  20:  {{/if}}
  21:  <hr />
  22:  {{/each}}
  23:  </script>

As you can see, we've got a conditional part of the template in there. This is the kind of thing that Mustache consciously avoids, but I personally find quite useful.

A note about line 11 above. The Handlebars site says that the if block helper evaluates an empty array to false, and I also uncovered a pull request on Github mentioning this very thing that was merged on June 27. However, the latest download is the v1 beta-3 which was packaged on June 2, hence resorting to the .length trick. It looks like this won't be necessary after the next release.

Finally, .NET has done something weird and unseemly to our date values, so we need some way to tweak those back into shape. Thankfully another feature of Handlebars is the ability to add custom helpers.

   1:  Handlebars.registerHelper("dotNetDate", function(context) {
   2:      var m = context.match(/^\/Date\((\d+)\)\/$/);
   3:      var date = null;
   4:      if (m)
   5:          date = new Date(parseInt(m[1]));
   6:      return date.toLocaleDateString();
   7:  });

And we can invoke it using this:

{{dotNetDate DateCreated}}

Giving us our final version, which you can see running here:

DataContainer + Handlebars.js Example

The downsides? Unfortunately this is an SEO black hole since all of the real content is injected in via Javascript at page load. You'll need to render this on the server-side in order to receive any search engine attention at all. That said, there's Mustache implementations in a tonne of viable server-side languages and while the few changes to Handlebars almost certainly increase the complexity . . . hmmm. Time for another weekend project?

Comments (0)

 
Older