Colourblind

Linq to SQL Deferred Loading Gotcha

2012-01-17T09:10:32Z

So here's a good one. Say we have some data in a table:

Id | Name
---+-------
 1 | Steve
 2 | Claire
 3 | Sarah 
 4 | Dave
 5 | Katie

And we run a LINQ-to-SQL query against it.

var results = dataContext.Persons.Where(x => x.Name.Contains("dave"));

It does what you'd expect it to do. Our database is case insensitive, so it matches as such.

Func<Persons, bool> predicate = x => x.Name.Contains("dave"));
var results = dataContext.Persons.Where(predicate);

Just when you think you have it licked, the above returns no rows. The problem is that the fact that we're using the predicate through a variable means that the data is grabbed from the DB before the Where() and then is filtered on the C#-side of the fence, which means your string comparisons are now case sensitive.

It's a slightly odd example (just a specific one that bit me on the arse), but it could have more serious ramifications. While this one fails pretty obviously, how about this: rather than getting 10 rows from the database you could be getting a million and then processing the lot of them in code.

Between enigmatic stuff like this, DataContext lifetime, detached objects and the eternally fun connection string pinata game, I still don't trust Linq to SQL completely.

Making MSBuild in NANT act like MSBuild in Visual Studio

2011-12-21T09:45:37Z

So I was making some NANT scripts for an ASP.NET MVC project.

Rather than going through the rigmarole of maintaining references and dependencies in two places (the VS project file and the NANT scripts), this time round I started using the <msbuild> task from nantcontrib instead of invoking the compiler directly using the plain old <csc> task. After making sure I set my OutDir I run NANT and get a failed build while copying a file, which is odd because I don't want MSBuild doing anything of the sort. It builds fine from Visual Studio, but after a little investigation it turns out that MSBuild is attempting to copy a file which is in the project but is missing from the file system. It's trying to drop it, along with the rest of the project contents, into _PublishedWebsites which has attaching itself to my output directory, like a warty blemish or tumour on my otherwise pristine build.

By setting the verbosity level of MSBuild to Diagnostic in both NANT and Visual Studio I could diff the two outputs and see where the processes started to diverge. The output of the NANT ran to a svelte 26k lines, but a quick search on '_PublishedWebsites' turns up the part of the log which includes the failing build. The copy operation is happening within a _CopyWebApplication target. Time to do a quick search in the Visual Studio MSBuild log . . .

Aha! Here's the slippery little devil:

Target "_CopyWebApplication" skipped, due to false condition; (!$(Disable_CopyWebApplication) And '$(OutDir)' != '$(OutputPath)') was evaluated as (!False And 'bin\' != 'bin\').

But why? Why would you do that? What is the OutputPath even supposed to be? According to MSBuild\Microsoft\VisualStudio\v10.0\WebApplications\Microsoft.WebApplication.targets . . .

<!--
============================================================
_CopyWebApplication

This target will copy the build outputs along with the content files into a 
_PublishedWebsites folder.

This Task is only necessary when $(OutDir) has been redirected to a folder 
other than ~\bin such as is the case with Team Build.

The original _CopyWebApplication is now a Legacy, you can still use it by 
setting $(UseWPP_CopyWebApplication) to true. By default, it now change to 
use _WPPCopyWebApplication target in Microsoft.Web.Publish.targets. It allow 
to leverage the web.config trsnaformation.
============================================================
-->

Fine. Whatever. Regardless of the rationale, if you set your OutDir and OutputPath properties to the same value in the NANT <msbuild> task:

<msbuild project="${dir.src}/${project::get-name()}/${project::get-name()}.csproj"
      target="Rebuild" verbosity="Minimal">
   <property name="Configuration" value="Release" />
   <property name="OutDir" value="../../${dir.build}/${project::get-name()}/bin/" />
   <property name="OutputPath" value="../../${dir.build}/${project::get-name()}/bin/" />
</msbuild>

Then your case of _PublishedWebsites should clear up in no time. (Alternatively you could set the Disable_CopyWebApplication property, but I'm attempting to get our NANT builds as close to our VS builds as possible.) As an addendum, once I knew what I was looking for I found quite a bit of confusion related to OutputPut, including a blog post from Mark Needham who also ran into problems related to OutputPath when tweaking OutDir and who turned up this nugget from MSDN:

OutputPath: This property is typically specified in the project file and resembles OutDir. OutputPath has been deprecated and OutDir should be used instead whenever possible.

At this point I stopped caring and reversed out of the rabbit hole.

Client-side templating with Handlebars.js

2011-11-30T22:43:38Z

So back to data molestation in Javascript.

Now we have our post-processed data it's time to display it to the user.

Templating in Javascript seems to be in vogue at the moment, so there's a pretty decent selection to choose from. Originally I was using jquery-tmpl, but since that project got abandoned in favour of something more awesomer it was time to have a look at alternatives. I initially fell upon Mustache (to be honest, mainly because of the name) but founds myself butting up against the stringently 'logic-less' approach. I guess if you are stricter with your view-models it's less of an issue, but I just found it a little restrictive. Handlebars is an evolution of Mustache from Yehuda Katz (of Rails fame), and appears to scratch the itches that Mustache leaves me with in addition to being a superset of the Mustache syntax. Also, the name is still cool.

A quick note, before we get started. On my Mustache-based travels I stumbled across ICanHaz.js, which simplifies the process and has some really nice ideas. For example, on page load it grabs all of the templates and adds them to a globally available collection for ease of use. This becomes even more useful in Handlebars since it requires an additional compilation step. I ended up writing something very similar but considerably more primitive, and here it is:

   1:  var BandleHars = {

   2:      templateCache : {},

   3:      init : function() {

   4:          var templateList = $('script[type="text/html"]');

   5:          templateList.each(function(index, value) { BandleHars.templateCache[value.id] = Handlebars.compile($(value).html()); });

   6:          templateList.remove();

   7:      },

   8:      render : function(templateName, data) {

   9:          return this.templateCache[templateName](data)

  10:      }

  11:  };

12:

  13:  $(function() { BandleHars.init(); });

Right, let's get back to templating. First we need our data. I used a grab of the first few posts of this very site, pretty much as it's dumped out by System.Web.Script.Serialization.JavaScriptSerializer.

   1:  var data = [

   2:    {

   3:      "Comments" : [

   4:          { "Body" : ". . . the comment system.",

   5:            "CommentId" : 1,

   6:            "DateCreated" : "/Date(1248627151247)/",

   7:            "GravatarUrl" : "http://www.gravatar.com/avatar/d41d8cd98f00b204e9800998ecf8427e?d=identicon&s=64",

   8:            "Heading" : "Testing . . .",

   9:            "Poster" : "Tom",

  10:            "Website" : "http://monochromacy.net/"

  11:          },

  12:          { "Body" : "n/t",

  13:            "CommentId" : 2,

  14:            "DateCreated" : "/Date(1254127922440)/",

  15:            "GravatarUrl" : "http://www.gravatar.com/avatar/d41d8cd98f00b204e9800998ecf8427e?d=identicon&s=64",

  16:            "Heading" : "Notification Test",

  17:            "Poster" : "Test",

  18:            "Website" : ""

  19:          }

  20:        ],

  21:      "DateCreated" : "/Date(1248624603550)/",

  22:      "DateLastTouched" : "/Date(1248624603550)/",

  23:      "Heading" : "Oh hai!",

  24:      "PostId" : 1,

  25:      "Slug" : "Oh-hai",

  26:      "Summary" : "This weblog will be largely concerned with ASP.NET, C# and ...",

  27:      "Tags" : [ "Waffle" ]

  28:    },

  29:    /* SNIP */

  30:    {

  31:      "Comments" : [  ],

  32:      "DateCreated" : "/Date(1261657386630)/",

  33:      "DateLastTouched" : "/Date(1261657386630)/",

  34:      "Heading" : "Introducing Compares Favourably",

  35:      "PostId" : 8,

  36:      "Slug" : "Introducing-Compares-Favourably",

  37:      "Summary" : "So I made a database schema comparison tool for MSSQL 2005.\nCompares Favourably\n<p ...",

  38:      "Tags" : [ "ComparesFavourably", "dotNet", "Projects" ]

  39:    },

  40:  ];

41:

  42:  var dataSet = new DataContainer(data, 3);

  43:  dataSet.update = function()

  44:  {

  45:      // We'll come back to this . . .

  46:  };

  47:  dataSet.sortProperty = 'DateCreated';

  48:  $(function() { dataSet.update(); });

Here I'm setting up the DataContainer with three items to a page, sorted on the created data and then running the (currently empty) update function once the DOM has been loaded. Now it's time for some markup to display and manipulate our data.

   1:  <label>Paging</label>

   2:  <ul class="horizontal">

   3:      <li><a href="#" onclick="return dataSet.pageDown();">Page down</a></li>

   4:      <li><a href="#" onclick="return dataSet.pageUp();">Page up</a></li>

   5:  </ul>

6:

   7:  <label>Sort by</label>

   8:  <ul class="horizontal">

   9:      <li><a href="#" onclick="return dataSet.setSortProperty('Heading');">Heading</a></li>

  10:      <li><a href="#" onclick="return dataSet.setSortProperty('Summary');">Summary</a></li>

  11:      <li><a href="#" onclick="return dataSet.setSortProperty('Slug');">Slug</a></li>

  12:      <li><a href="#" onclick="return dataSet.setSortProperty('DateCreated');">Date Created</a></li>

  13:  </ul>

Nice and simple. This will page the data up and down, allow us to change the sort criteria, and includes some bonus hey-that's-not-what-label-elements-are-for abuse. Sue me. I figure a tag filter would make sense in this context and since this is dependant on the data it will need to be templated. First we need a container for our filter buttons:

   1:  <label>Tags</label>

   2:  <ul class="horizontal" id="tagsFilter"></ul>

And now a template for creating the filter links:

   1:  <script id="tagsFilterTemplate" type="text/html">

   2:  {{#each data}}<li><a href="#" onclick="return dataSet.addFilter('Tags', '{{this}}');">{{this}}</a></li>{{/each}}

   3:  <li><a href="#" onclick="return dataSet.removeFilter('Tags');">remove</a></li>

   4:  </script>

The above Bandlehars code (I hate that name already) will fetch and compile this template on body load, so now we just need to add this to our DataContainer update function to include the template rendering:

   1:  dataSet.update = function()

   2:  {

   3:      // There's still more to come!

   4:      $('#tagsFilter').html(Bandlehars.render('tagsFilterTemplate', { data : dataSet.getPropertyValues('Tags') }));

   5:  };

Now, once the page is loaded and then whenever it is filtered, the list of available tags will be updated. This means we can drill down further using additional filters. Worth noting is that we're wrapping our data in another object, since Handlebars doesn't accept raw arrays as a data context. So that's the filters sorted. But we still can't see the data itself. It's time for another container and another template.

   1:  <div id="data"></div>

2:

   3:  <script id="dataTemplate" type="text/html">

   4:  {{#each data}}

   5:  <div>

   6:      <h2><a href="http://monochromacy.net/Post/{{Slug}}.aspx">{{Heading}}</a></h2>

   7:      <p><em>{{DateCreated}}</em></p>

   8:      <p>{{Summary}}</p>

   9:      <label>Tags</label>

  10:      <ul class="horizontal">{{#each Tags}}<li><a href="#" onclick="return dataSet.addFilter('Tags', '{{this}}');">{{this}}</a></li>{{/each}}</ul>

  11:  </div>

  12:  {{/each}}

  13:  </script>

Now to add it to our DataContainer update function.

   1:  dataSet.update = function()

   2:  {

   3:      $('#data').html(Bandlehars.render('dataTemplate', { data : dataSet.getData() }));

   4:      $('#tagsFilter').html(Bandlehars.render('tagsFilterTemplate', { data : dataSet.getPropertyValues('Tags') }));

   5:  };

Well it's a start. What's next? How about comments.

   1:  <script id="dataTemplate" type="text/html">

   2:  {{#each data}}

   3:  <div>

   4:      <h2><a href="http://monochromacy.net/Post/{{Slug}}.aspx">{{Heading}}</a></h2>

   5:      <p><em>{{DateCreated}}</em></p>

   6:      <p>{{Summary}}</p>

   7:      <label>Tags</label>

   8:      <ul class="horizontal">{{#each Tags}}<li><a href="#" onclick="return dataSet.addFilter('Tags', '{{this}}');">{{this}}</a></li>{{/each}}</ul>

   9:  </div>

  10:  <br />

  11:  {{#if Comments.length}}

  12:  <div class="comments">

  13:      <h4>Comments</h4>

  14:      <ul>

  15:  {{#each Comments}}

  16:          <li>{{Heading}} - {{Body}} - {{Poster}}</li>

  17:  {{/each}}

  18:      </ul>

  19:  </div>

  20:  {{/if}}

  21:  <hr />

  22:  {{/each}}

  23:  </script>

As you can see, we've got a conditional part of the template in there. This is the kind of thing that Mustache consciously avoids, but I personally find quite useful.

A note about line 11 above. The Handlebars site says that the if block helper evaluates an empty array to false, and I also uncovered a pull request on Github mentioning this very thing that was merged on June 27. However, the latest download is the v1 beta-3 which was packaged on June 2, hence resorting to the .length trick. It looks like this won't be necessary after the next release.

Finally, .NET has done something weird and unseemly to our date values, so we need some way to tweak those back into shape. Thankfully another feature of Handlebars is the ability to add custom helpers.

   1:  Handlebars.registerHelper("dotNetDate", function(context) {

   2:      var m = context.match(/^\/Date\((\d+)\)\/$/);

   3:      var date = null;

   4:      if (m)

   5:          date = new Date(parseInt(m[1]));

   6:      return date.toLocaleDateString();

   7:  });

And we can invoke it using this:

{{dotNetDate DateCreated}}

Giving us our final version, which you can see running here:

DataContainer + Handlebars.js Example

The downsides? Unfortunately this is an SEO black hole since all of the real content is injected in via Javascript at page load. You'll need to render this on the server-side in order to receive any search engine attention at all. That said, there's Mustache implementations in a tonne of viable server-side languages and while the few changes to Handlebars almost certainly increase the complexity . . . hmmm. Time for another weekend project?

Data Wrangling on the Client with Javascript

2011-10-24T20:20:46Z

Filtering, sorting and paging. These three operations are the bread and butter of data display. And it really is a display concern, so why not push it as far as possible into the presentation layer? When it comes to webapps that means the browser. So I wrote a data wrapper in Javascript as part of one of my side projects. It goes by the highly uninspired name of DataContainer, because I was having an unoriginal day.

Setting up

Everything is contained in the DataContainer object, and the constructor takes the data itself as an array of objects, and the number of objects on each page.

var data = [
    {
        color: "red",
        value: "#f00",
        prime: true,
        rand: 1
    },
    {
        color: "green",
        value: "#0f0",
        prime: true,
        rand: 3
    },
    {
        color: "blue",
        value: "#00f",
        prime: true,
        rand: 1
    },
    {
        color: "cyan",
        value: "#0ff",
        prime: false,
        rand: 2
    },
    {
        color: "magenta",
        value: "#f0f",
        prime: false,
        rand: 1
    },
    {
        color: "yellow",
        value: "#ff0",
        prime: false,
        rand: 1
    },
    {
        color: "black",
        value: "#000",
        prime: false,
        rand: 3
    }
];

var pageSize = 10;
var dataSet = new DataContainer(data, pageSize);

As intimated above, there are three types of operations available in DataContainer - filtering, sorting and paging - and they are always performed in the following order.

Filtering
Sorting
Paging

A call to getData takes the original dataset and then applies the filters, performs a sort and then splits the result onto the correct page and returns it. The most practical way of fetching data is through the use of callbacks. Whenever any operations are performed on the DataContainer which result in the resulting dataset changing the DataContainer's update function is called. If you want to update UI elements whenever the data is changed (and what else would you be using it for?) then simple pass the DataContainer a new update method with your own code. I'll be covering this in more detail in the next post - this is more about how to use the various DataContainer methods available.

Controls

Here's a brief rundown of the methods available on each DataContainer object.

Filtering

dataSet.addFilter(property, value);
dataSet.removeFilter(property);

Filtering is done based on the properties of each object in the initial array. Given the above data, if you make the call dataSet.addFilter("primary", true) then subsequent calls to getData will return on only those objects with the primary property set to true. If the initial data contains complex objects, then you can drill down into the object hierarchy using dot notation, the same way you would in code.

You can add multiple filters, allowing you filter the data more specifically. Also of note is the getPropertyValues method. When you pass this the name of a property it will apply the current filters to the dataset and then build a list of values of that property. The intent here is make building a drill-down UI nice and easy.

Sorting

dataSet.toggleSortDir();
dataSet.setSortProperty(property);

You can only sort by one property at a time. If you call the setSortProperty method a subsequent time with the same property name it will toggle the sort between ascending and descending.

Paging

dataSet.pageUp();
dataSet.pageDown();
dataSet.setPage(pageNum);

All pretty self explanatory. The pages are zero-indexed. You can also use the maxPage method to get the index of the final page.

Yay

There's real advantages to doing this stuff in the browser.

It means that your backend infrastructure can be swapped out easily. Wish to port from ASP.NET to Rails? It's not exactly free, but it's less work than it would be otherwise
No more sorting, filtering and paging gunk in your controllers

However . . .

Doesn't scale so good. For large datasets or crappy hosts you're going to kill the browser
If you want a non-Javascript fallback you'll need to implement everything ye olde way anyway

This is going to get long and boring so I'll move the second half of this over to second post in a couple of weeks. The next post will show how to create a dynamic UI over the top of DataContainer using Javascript templating.

Random burbling follows:

In my professional life we need to target everything when it comes to browsers. Sales ride on it and you can't just clam the site up because users are a bunch of dirty, IE6-using, tech-ignorant peasants. While we can create sites that rely heavily on Javascript we still need a Javascript-free fallback, and so we end up doing the work twice, in two different languages and on two different sides of the client-server divide. Usually this means we shy away from too much Javascript due to our time budget. Since this was part of an intranet project it was nice to be able to get down and write some real JS code, safe in the knowledge that if someone isn't using a supporting browser I can just walk over to the other side of the office and slap them. That said I did a trawl through some sites recently and a good proportion of them in our target market sector (fashion), simply don't work without Javascript. Maybe the days of non-Javascript fallbacks are past. Man, that would be great.

I gone done a Rainmeter skin

2011-09-17T12:39:05Z

I've been playing around with Rainmeter recently and have nailed together my own skin.

Click on the image above to download the skin files. It contains a clock, CPU and memory graphs, network in and out, disk usage, RSS reader, and some system info. And yes, the look is partially inspired by Deus Ex: Human Revolution, which may or may not be responsible for the slow rate of updates recently. Ahem.

One thing to note is that the snapping will be buggy against the clock, since the automatic width and height of string meters doesn't take into account the transformation matrix. I found a link to one of the Rainmeter devs confirming this, but my Google-fu is weak and I can't find it now. Also, thanks to Oscar Del Ben for this Github feeds and Eidos forum user f0xhound for his Deus Ex font.

In other news:

addons.mozilla.org appears to be sucking the addons right out of my browser. That's just mean.