The build is an MSBUILD script that compiles all of our projects and dependencies, and then copies the results to a staging directory from which we can run unit tests or build distribution packages for our internal customers (see below).  To this point, everything can be done with a minimal batch file script, the MSBUILD program supplied with the .NET development tools, and of course the subversion command line client.  We have one other tool called sendEmail that notifies me of the build status.

I’d like to notify everybody when the build fails, but doing so requires that I tell them why it failed.  And the generated build log is very large:  about 120 kilobytes, most of which is irrelevant.  The important information is typically the last 10 or so lines of the file, and that’s what I’d like to send to people when the build fails.  Those lines say, in effect, “The build failed for these reasons.”  A programmer who receives that message can quickly determine if it’s his responsibility, and take steps to fix the problem.

The only trouble I have is that there is no simple way with Windows-supplied tools to extract those pertinent lines from the file.  At least, I can’t think of a way.  But the GNU awk (gawk) can do it trivially.

When the build fails, the last thing that MSBUILD outputs is a line that says, “Build FAILED”, followed by some lines that describe the error or errors.  So all I need is a program that will go through the file, locate the “Build FAILED” line, and then output that line and all following lines to the end of the file.  It’s been 20 years since I did any awk programming, but this script was simple:

gawk "{ if (/^Build FAILED/) { doit=1 } if (doit) print $0 }" < buildlog.txt

Done and done.

The only problem I have now is deciding whether I want to install the full GNU Tools for Windows package on my server, or if I should just grab Gawk for Windows.  The full package is probably the right way to go because I suspect I’ll be needing some other tools in the future.

Either way, I’m annoyed that Windows doesn’t include these simple text processing tools.  I can perhaps understand why they don’t exist in desktop versions, but we do these types of things on servers all the time, and the standard server install should include a more robust toolset.

Above I mentioned “internal customers.”  In reality, we are our own customers.  There are only five of us here, and one of us doesn’t use the tools that the build creates.  In light of that, it’d be easy to take a more cavalier attitude towards our build process.  I’ve found, though, that things run smoother if, as a programmer, I think of the users of my software (in this case, the crawler subsystem and the tools that process the collected data) in much the same way as I would an external customer.  Even though the primary user of the crawler is me.  I wear a number of different hats around here (as does everybody else–we’re a startup, after all), and it’s useful to think of Jim the SysAdmin as a separate person from Jim the Programmer.  That way, when we can afford to hire a system administrator to take those duties from me, the systems will already be in place for him to step right in.

Like source code version control, a formal build process is one of those things that you don’t need to implement until the size of your project team exceeds zero people.

http://www.example.com/videos/?id=23&format=hd

Will be encoded so that “&” becomes “&”.  Any character that is “special” in XML will end up being entity encoded like that.  Oops.

A quick search for “xml grep” led me to XMLStarlet:  a command line XML toolkit that lets you examine, query, fold, spindle, and mutilate XML files from the command line.  I don’t know nearly as much as I should about XPath, XSLT, and XML in general, but after a few minutes of looking at examples and struggling with the syntax, I managed to pull those URLs out of the XML file and send them off to the crawler.

Granted, I spent a heck of a lot more time on this than I would have just writing a quick C# program to extract that one element from the file in question.  But my C# program would have worked for this situation only.  I already have other plans for XMLStarlet.

Highly recommended.  If you ever find yourself having to manipulate XML files outside of your application, you need this tool.

grep -o "http://www.example.com/feeds/rss/[^.]\+.rss" feedIndex.html
  | sort -u | xargs wget

What makes that possible is the GNU tools–a standard set of tools that mimic and extend the standard tools that have been available for Unix-based systems for decades.

Although the Windows command line supports piping, it doesn’t include a comprehensive set tools that were designed to work together the way the GNU tools were designed.  The Windows toolset is primitive, and not up to solving this simple task.  I used GNU grep for Windows to extract the URLs and save them to a file, TextPad to sort and then manually remove duplicates, and finally GNU wget for Windows to download the files.

This isn’t the first time I’ve had to resort to a hodgepodge of tools to solve a problem that I could solve without trouble if I had the GNU tools.  But in the past, downloading and installing all the GNU tools for Windows was a giant pain in the neck with version conflicts and such.  That’s not a problem any longer.  Today I discovered the getgnuwin32 project, which automates the process of downloading, installing, and maintaining a full set of GNU tools for Windows. 

The few tools I’ve used so far work exactly as expected.  Time (and some effort:  it’s been a while since I used the GNU tools) will tell if this is as useful as I hope it is.

Update (later the same day):
There is one slight problem:  some of the GNU tools have name conflicts with the Windows tools.  sort is a good example.  If I tried the above command line on a Windows machine, it would try to invoke the brain damaged Windows SORT utility, which is so bad that whoever wrote it should die from embarrassment.  It depends on where in your path you put the GnuWin32\bin directory.  Either way you go, name conflicts are going to give you some headaches.

I’m thinking that, since most programmers don’t even know that the Windows command line exists, I’ll put GnuWin32\bin ahead of the Windows directory that contains the standard tools.  Or maybe I should just delete or rename SORT and any other tools that have conflicting names.  It’s not like I ever run batch files that I get from other people.

Disclaimer:

The examples below mention particular artists whose content appears legitimately on YouTube and other media sites, and can be legally obtained with the blessing of the copyright holders.

Although it’s possible that content from these artists can also be obtained illegally from other sites, I do not advocate that practice.  I do not support the use of any Internet search technology to obtain music, video, or other electronic media illegally. 

Companies that operate search engines do not knowingly index such illegal content.  Reputable companies remove links to illegal content as required by the Digital Millennium Copyright Act (DMCA), when the existence of that content is made known in accordance with the DMCA’s notification procedures.

Except it turns out that britney and spears are pretty common spam terms in metadata (the keywords and description fields of YouTube videos, for example).  People will upload all manner of stuff to YouTube and put bogus terms in the description in an attempt to get people to watch the video.  To reduce the number of irrelevant or inappropriate results returned (it’s probably impossible to eliminate irrelevant content), you decide to index the metadata by field and allow the user to say which fields are searched.  So, if they want just those videos that have “Britney” and “Spears” in the title field, they would type britney spears IN Title.  That doesn’t eliminate all of the spam, but it reduces it quite a bit.

It turns out that you have to make the IN case sensitive.  Otherwise you’d never be able to search for the word “in” in any metadata.  The same is true for any word that you use in your query language.  For example, if wanted all the videos that contain “Britney” or “Spears”, we’d write britney OR spears IN Title.

Still, not too hard, right?  But what if you want to search the Title field and the Description field?  At first you’d think you could write:  britney spears IN Title OR Description.  You could make that work until you take into account the possibility of more complex query expressions.  For example, let’s say you wanted a list of all videos that claim to be a Led Zeppelin song, or some version of Stairway to Heaven.  One possible query would be:

led zeppelin IN Artist OR Description OR stairway heaven IN Title

Whereas that query might look reasonable to a non-programmer, writing a computer program to properly handle the general case of queries like that is non-trivial.  The query can be parsed in several different ways.  Three of which are:

(led zeppelin IN Artist OR Description) OR (stairway heaven IN Title)
(led zeppelin IN Artist) OR )(Description OR stairway) heaven IN Title)
(led zeppelin IN Artist) OR (description OR (stairway heaven) IN Title)

All three of those interpretations are perfectly valid.  Applying rules of operator precedence can disambiguate some of the cases, but if you go through the exercise you’ll find out that IN has to have lower precedence than OR, and if you do that, then you end up with:

(led zeppelin IN Artist OR (Description OR stairway heaven)) IN Title

You end up having to either decorate the field names (i.e. “@Artist”) or group them with brackets or parentheses (i.e IN [Artist or Description]).

All of this is doable, and not especially heavy lifting as far as parsing is concerned.  But then you have to explain it to a non-technical user and make it easy for the non-technical user to use.  Otherwise, only programmers will want to (or even be able to) use it.

I’ve heard many a programmer (myself included, come to think of it) complain about a search facility that doesn’t allow complex queries.  We look at it from a programmer’s perspective and think it’d be trivial to implement a comprehensive query facility.  And in most cases they’re probably right.  You could develop a query system that anybody with a couple years’ of programming experience could use without trouble and get exact results.  And when you flipped the switch to turn it on, you’d hear crickets.  Most users don’t understand Boolean algebra or the difference in precedence between AND and OR.  Trust me, people will go somewhere else to get their information rather than have to think of how to ask for it.

What users really want is a DWIM mode:  Do What I Mean.  They want to type word soup into the search and get back exactly what they were looking for, with no false hits (i.e. asking for beatles the music group and getting back something about dung beetles because somebody misspelled “beetle”).

But DWIM doesn’t exist.  Not today, and not for a long time (perhaps ever) in the future.  As a result, we have to restrict what the user can type and very carefully specify how things will be interpreted.  We have to make it easy for the most common cases, but able to do moderately complex and powerful things.  That balance is difficult to achieve, and no matter what you come up with, somebody will complain.  You can only hope that the number of users you delight will vastly outweigh those whom you annoy.

Other sites are much worse in that they provide no metadata in the HTML.  For example, one site’s video page is very code-heavy.  Requiring that the page be reloaded every time you request a new video would require a lot of network traffic.  Their design instead uses JavaScript to request a particular video’s metadata from a server.  Loading a new video involves downloading just a few kilobytes of data.

I spent some time this afternoon searching through the a video page HTML and the associated JavaScript, looking for the magic incantation that would get me the data I’m looking for.  The amount of code involved is staggering, and I quickly went crosseyed trying to decipher it before I hit on the idea of hooking up a sniffer to see if I could identify the HTTP request that gets the data.

It took me all of five minutes to download and install Free Http Sniffer, request a video from the site in question, and locate the magic line in the 230 or so requests that the page makes when it loads.  Problem solved.  Now all I have to do is write code that’ll transform a video page url into a request for the metadata, and I’m set.

I have no idea why I didn’t think of the sniffer earlier.  I’d used one before for a similar purpose.  I suspect I’ll be making heavy use of it in the near future as I expand the number of sites that we crawl for media.

After 30 years, you’d think that I would have learned that there’s no such thing as a small change that you can push into production code without having to test.  You might get away with it from time to time, but eventually that arrogance is going to cost you.

But, hey, it’s a simple change!  What could go wrong?

When you hear yourself say that, think about what you’re saying.  And then spend the few minutes it will take to test your assumption.  If nothing else, you’ll save yourself the embarrassment of explaining to your business partner that you made the kind of mistake that you’d reprimand an employee for.

Fortunately, all it cost me was a little embassassment, a few hours’ lost sleep, and an additional hour of down time for the crawler.  I got off easy.

So I fired up Clonezilla, fought through the user interface to tell it what I wanted backed up and where, and then pressed the any key (really!  There was a prompt that said, “Press the any key”) to start the copy.  Clonezilla promptly told me that my network card wasn’t supported.  It would have been nice if it would have checked that when I first started the program.

Slightly discouraged but not yet willing to give up, I decided to try PING.  Another cryptic user interface, but I won’t complain too much considering the price.  This time my network card was supported and after a couple of house it had created a copy of my partition.  So I fired up the next machine, ran PING, told it to copy the partition image to the disk.  That went well, too.  Except that after I was done, the machine wouldn’t boot.  The BIOS doesn’t see a bootable image on the disk.

At that point I gave up.  I’d already spent almost a full day futzing with the things.  In that time I could have installed and configured all of the machines.  (Or so I thought.)  In any case, my experiments with free drive cloning software left me disappointed.

There’s a good overview of Ghost alternatives over at pack rat studios, but I haven’t had the opportunity to try any of the others mentioned.  Clonezilla didn’t support my hardware, and PING failed for reasons unknown.  Anybody know of a package that actually works?

By the way, telling a potential user, “if your network card isn’t supported, download it and compile it into the Clonezilla package” is not likely to be met with smiles and thanks.  More likely, users—even technically competent users like me who are capable of downloading and building—are more likely to say, “no thanks,” and move on to something else.

I learned long ago.  In a little more than 30 years of programming computers (about 25 years doing it full time), I can think of only a handful of cases in which a bug in one of my programs was caused by anything other than my own error.  I know that there are bugs in operating systems and runtime libraries, but it’s not often that they cause problems for me.

So imagine my surprise when, within the space of one week, I’ve identified two previously unknown (or unreported, as far as I can tell) genuine bugs in the Microsoft .NET Framework runtime libraries.  I wrote about the first one the other day.  Microsoft has acknowledged that it is a bug and is currently reviewing it.

I originally thought that the new bug was in the Uri.TryCreate method, because it throws an exception in some circumstances even though the documentation says that it won’t throw an exception but rather will return False if it’s unable to create a valid Uri from the input parameters. And although throwing an exception in this case is (maybe) a bug, the cause of the bug is something else: the Uri constructor allows you to construct invalid Uri instances.

In my particular case, my Web crawler crashed because Uri.TryCreate threw an exception. That was very unexpected, and whatever parameters caused it are lost to me. But it’s pretty rare. I pass somewhere upwards of 250 million urls through that function every day. Unable to reconstruct the exact parameters that caused the problem, I used what I learned from poking around in the disassembled code to come up with a string that illustrates the problem.

The Uri constructor creates a Uri instance from a passed string. Uri is pretty cool in that it supports many types of resources, not just HTTP.  One such type is a mailto: URI of the form mailto:jim@mischel.com.

But the Uri constructor will succeed when it should fail. For example:

Uri mailUri = new Uri("mailto:jim@mischel.comtest@mischel.com");
// trying to access mailUri.AbsoluteUri at this point will throw UriFormatException
// with the message "The host name cannot be parsed"

Passing that invalid Uri as the baseUri parameter to Uri.TryCreate throws the exception that I encountered in the crawler.

What I find most curious about all this is that the Uri class appears to have two different methods for parsing strings.  There appears to be one parser for constructing Uri instances from strings (as in the constructor), and a separate parser used internally for various things.   I know from experience that parsing URIs is a horrendously difficult problem and hard to do correctly.   Why anybody would want to write, test, and maintain two different versions of such difficult code is beyond me.

Bug reported.  Awaiting response.

System.OverflowException: Negating the minimum value of a twos complement number is invalid.
   at System.Math.AbsHelper(Int32 value)
   at System.Random..ctor(Int32 Seed)
   at System.Threading.Collections.ConcurrentQueue`1.TryDequeueCore(T& result)
   at System.Threading.Collections.ConcurrentQueue`1.TryDequeue(T& result)
   at MyProgram.ThreadProc() in c:\MyProgram\Main.cs:line 118
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

The documentation for the Random constructor says that this exception will be thrown if the Seed parameter is equal to Int32.MinValue.  Curious, I thought I’d disassemble the ConcurrentQueue class to see what’s going on.  No problem there, as it’s calling the default (parameterless) Random constructor.  So I took a look at that.

.method public hidebysig specialname rtspecialname
        instance void  .ctor() cil managed
{
  // Code size       12 (0xc)
  .maxstack  8
  IL_0000:  ldarg.0
  IL_0001:  call       int32 System.Environment::get_TickCount()
  IL_0006:  call       instance void System.Random::.ctor(int32)
  IL_000b:  ret
} // end of method Random::.ctor

That code gets the current tick count and then passes that value as a seed to the constructor.  So, what’s wrong with this?  Take a look at the documentation for System.Environment.TickCount:

The value of this property is derived from the system timer and is stored as a 32-bit signed integer. Consequently, if the system runs continuously, TickCount will increment from zero to Int32.MaxValue for approximately 24.9 days, then jump to Int32.MinValue, which is a negative number, then increment back to zero during the next 24.9 days.

What all this means is that if your program calls the Random constructor during that one millisecond (after the system has been up for Int.MaxValue milliseconds), the value returned by System.Environment.TickCount is going to be equal to Int.MinValue, and passing that value as the seed will result in an exception.

I’ll be the first to admit that encountering this bug is highly unlikely.  Your computer has to be up and running for almost 25 days, and there’s a one-millisecond window when it’s vulnerable.  But the fact that my program crashed is proof that it is possible for this error to cause a problem.

This is a huge oversight on the part of Microsoft’s runtime library team.  I’ve reported the bug and hope they manage to get it fixed before they release .NET 4.0.

Update 2009/07/21:  Microsoft’s Base Class Library team has said that the issue has been resolved for the next major release.

When Borland, Microsoft, and other development tool companies moved their online technical support to the Internet, their support was mostly done through newsgroups, and I found the signal-to-noise ratio there almost unbearable.  Except for the moderated newsgroups, which were few and far between, asking a question was like talking to a wall.  Worse, even, because a wall won’t give you wrong answers or call you stupid for doing something different.  Even with the advent of forums rather than newsgroups, online technical help was virtually non-existent for a number of years and I just stopped trying.

Enter Stack Overflow, a free programming Q&A site where you can ask questions, share your expertise, or just browse for nuggets of programming wisdom.  Stack Overflow works.  In many ways it works better than the old Borland Compuserve forums that I enjoyed so much.

Why it works is simple: they’ve found a way to reward people for supplying good answers and, to a lesser extent, asking good questions.  It all has to do with reputation:  ego.  You gain reputation points for supplying good answers, and asking good questions.  “Good” is determined by a simple up- or down-votes by site users.  As you gain reputation points, you gain the ability to help moderate the site: re-tag questions, vote to close, edit questions, etc.  And your current reputation is prominently displayed beside your name.  There are also awards (“Badges”) given for a number of different things.

If you don’t care about reputation, that’s fine.  You can use the site anonymously and still ask, answer, and comment on questions.  But Stack Overflow works because a whole lot of people there do care about their reputations.  Giving more experienced users the ability to help moderate keeps the flaming and other invective to a minimum, and the constant peer review ensures that (in general) the higher-rated answers really are the best.

My only real complaint with Stack Overflow (and it’s not huge) is that the format doesn’t encourage an ongoing threaded discussion as was available on the Compuserve forums.  That’s not a problem in most cases, but there are times when arriving at a satisfactory answer requires much back-and-forth, and it’d be nice to see questions and answers displayed in threaded newsgroup fashion.  The ability to see answers ordered by date helps a lot, as does the comments feature, and I suspect that adding a threaded view would be of only limited additional help.

Despite a few nitpicks, I’m seriously impressed with Stack Overflow.  If you have a programming question on any topic, you should search for the answer there.  And if you don’t find it, ask.  You’ll probably be surprised at the speed and the quality of the answers you get.