Monthly Archives: July 2009

Downloading with wget

Posted on by

GNU wget is a free program for retrieving files from the Web.  I freely admit that I’m just a casual user and don’t know all the things that it can do.  But it does one thing that I use quite often:  download a list of files.  It’s incredibly convenient.  All you have to do is create a text file that contains the URLs of the files you want to download, one file per line.  Then fire up wget.  Assuming the file that contains the URLs is called DownloadList.txt, then this command line will download all the files, one at a time:

wget -i DownloadList.txt

wget is available on just about any Linux distribution.  There’s also a Windows version.

Documentation is at http://www.gnu.org/software/wget/manual/.

Another .NET Framework bug?

Posted on by

When faced with inexplicable program behavior, inexperienced programmers often blame the operating system, the runtime library, the compiler, or some other external force for the error.  Even when they discover that the bug is in their code, these programmers often will try to blame it on something else.  A sure sign of maturity in a programmer is how he approaches bugs.  If, when faced with a bug, he concentrates his initial efforts on looking for the problem in his own code, you know that he’s learned.

I learned long ago.  In a little more than 30 years of programming computers (about 25 years doing it full time), I can think of only a handful of cases in which a bug in one of my programs was caused by anything other than my own error.  I know that there are bugs in operating systems and runtime libraries, but it’s not often that they cause problems for me.

So imagine my surprise when, within the space of one week, I’ve identified two previously unknown (or unreported, as far as I can tell) genuine bugs in the Microsoft .NET Framework runtime libraries.  I wrote about the first one the other day.  Microsoft has acknowledged that it is a bug and is currently reviewing it.

I originally thought that the new bug was in the Uri.TryCreate method, because it throws an exception in some circumstances even though the documentation says that it won’t throw an exception but rather will return False if it’s unable to create a valid Uri from the input parameters. And although throwing an exception in this case is (maybe) a bug, the cause of the bug is something else: the Uri constructor allows you to construct invalid Uri instances.

In my particular case, my Web crawler crashed because Uri.TryCreate threw an exception. That was very unexpected, and whatever parameters caused it are lost to me. But it’s pretty rare. I pass somewhere upwards of 250 million urls through that function every day. Unable to reconstruct the exact parameters that caused the problem, I used what I learned from poking around in the disassembled code to come up with a string that illustrates the problem.

The Uri constructor creates a Uri instance from a passed string. Uri is pretty cool in that it supports many types of resources, not just HTTP.  One such type is a mailto: URI of the form mailto:jim@mischel.com.

But the Uri constructor will succeed when it should fail. For example:

Uri mailUri = new Uri("mailto:jim@mischel.comtest@mischel.com");
// trying to access mailUri.AbsoluteUri at this point will throw UriFormatException
// with the message "The host name cannot be parsed"

Passing that invalid Uri as the baseUri parameter to Uri.TryCreate throws the exception that I encountered in the crawler.

What I find most curious about all this is that the Uri class appears to have two different methods for parsing strings.  There appears to be one parser for constructing Uri instances from strings (as in the constructor), and a separate parser used internally for various things. I know from experience that parsing URIs is a horrendously difficult problem and hard to do correctly.   Why anybody would want to write, test, and maintain two different versions of such difficult code is beyond me.

Bug reported.  Awaiting response.

“Highly unlikely” is not the same as “Impossible”

Posted on by

One of my programs crashed the other day in a very unexpected place:  inside the runtime library.  The exception stack trace is pretty clear on where the error occurred:

System.OverflowException: Negating the minimum value of a twos complement number is invalid.
   at System.Math.AbsHelper(Int32 value)
   at System.Random..ctor(Int32 Seed)
   at System.Threading.Collections.ConcurrentQueue`1.TryDequeueCore(T& result)
   at System.Threading.Collections.ConcurrentQueue`1.TryDequeue(T& result)
   at MyProgram.ThreadProc() in c:\MyProgram\Main.cs:line 118
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

The documentation for the Random constructor says that this exception will be thrown if the Seed parameter is equal to Int32.MinValue.  Curious, I thought I’d disassemble the ConcurrentQueue class to see what’s going on.  No problem there, as it’s calling the default (parameterless) Random constructor.  So I took a look at that.

.method public hidebysig specialname rtspecialname
        instance void  .ctor() cil managed
{
  // Code size       12 (0xc)
  .maxstack  8
  IL_0000:  ldarg.0
  IL_0001:  call       int32 System.Environment::get_TickCount()
  IL_0006:  call       instance void System.Random::.ctor(int32)
  IL_000b:  ret
} // end of method Random::.ctor

That code gets the current tick count and then passes that value as a seed to the constructor.  So, what’s wrong with this?  Take a look at the documentation for System.Environment.TickCount:

The value of this property is derived from the system timer and is stored as a 32-bit signed integer. Consequently, if the system runs continuously, TickCount will increment from zero to Int32.MaxValue for approximately 24.9 days, then jump to Int32.MinValue, which is a negative number, then increment back to zero during the next 24.9 days.

What all this means is that if your program calls the Random constructor during that one millisecond (after the system has been up for Int.MaxValue milliseconds), the value returned by System.Environment.TickCount is going to be equal to Int.MinValue, and passing that value as the seed will result in an exception.

I’ll be the first to admit that encountering this bug is highly unlikely.  Your computer has to be up and running for almost 25 days, and there’s a one-millisecond window when it’s vulnerable.  But the fact that my program crashed is proof that it is possible for this error to cause a problem.

This is a huge oversight on the part of Microsoft’s runtime library team.  I’ve reported the bug and hope they manage to get it fixed before they release .NET 4.0.

Update 2009/07/21:  Microsoft’s Base Class Library team said that the issue has been resolved for the next major release.

Getting attention the new way

Posted on by

So let’s say that you’re a musician on your way by airplane from Point A to Point C with a stop at Point B. Trusting the airlines to handle your luggage, you check your guitar. While sitting in the airplane at Point B you see the luggage handlers treating your guitar roughly, and when you arrive at Point C you learn that the guitar has been broken.

So you spend a year trying to convince the airline that they should make things right.  When your efforts fail and the airline says that their final response is “No,” you decide on a different plan of action.

United Breaks Guitars

Dave Carroll posted that video on July 6.  CNN reported on it two days later.  Since then, it’s been reported on several other major news networks and countless blogs.  Today, five days after the video was posted, it has over 2 million views.

As a coworker said, “never piss off a musician.”  I’m betting United Airlines wishes they had handled this differently.

Tribal drum beating

Posted on by

To summarize a long reply I sent to a mailing list message ranting about the evils of the Obama administration:

Don’t get me wrong. I’m no more an Obama supporter than I was a Bush supporter. The game is rigged, folks. Wake up and stop wasting your energy railing against the current President or passing around partisan screeds.  Take off your blinders and notice that both parties have one thing in common that should frighten any thinking adult:  their continued existence requires that you actively participate in the continued erosion of your own freedom.  The only way we’ll return to a more rational world, where the freedoms guaranteed by the Constitution and the limits placed on government are actually adhered to, is to actively campaign to throw the bums out.  Get rid of every incumbent. Refuse to vote for anybody who claims membership in any of the major political parties.  Hell, get rid of party politics altogether.

Yes, I know.  I’m asking too much.  I’m not expecting it to happen.  But I’d just like you to know that until we get rid of the parties, all your ranting and carrying on is just pissing into the wind.

So go ahead with your tribal drum beating.  But please leave me out of it.

“Cash for clunkers” is a wreck

Posted on by

President Obama and those members of Congress who approved the cash for clunkers program as part of the most recent war funding bill are celebrating today and telling us how they knew that it “would work.”  What’s their definition of “working?”  Why, that people actually took advantage of it.

Imagine that.  You’re going to give somebody up to $4,500 for their old worthless gas guzzler if they trade it in on a new, fuel efficient car.  And people take advantage of it?  Shocking!  In one week, the $1 billion allocated for the program is gone.  Congress today is expected to approve an additional $2 billion.

The clunkers collected through this program are supposed to be sent to a facility where they can be crushed and recycled.  However, there’s some wiggle room in the language, allowing transmissions and other parts to be salvaged.  And I doubt that there are sufficient controls in place to prevent the cars themselves from being resold and showing up on the roads again, either here in the U.S. or in other parts of the world.

Proponents of the program say that it is good for consumers, good for the auto industry, and good for the environment.  One would also expect that salvage yards and such would get a boost from the junked cars.

The truth is that this is a blatant handout to the auto industry and auto workers unions, paid for with money the government doesn’t have and will have to extort from financially responsible citizens in the future, and taken advantage of in large part by people who probably shouldn’t be buying new cars anyway.  It has the curious effect of artifically lowering new car prices to the point where it’s cheaper to buy a new car than to buy one that’s a year or two old.  I’ll bet used car dealers aren’t too happy about this one.

Congress and the President, of course, are falling all over themselves trying to scrape up more deficit spending so they can continue the program.  Damn the long term consequences, (some of) the people love this program!  Ain’t nothing like buying votes, is there.

Don’t you just love the way our political system works?