Jim’s Random Notes

Musings on technology and life

October 6th, 2008

Hardware Problems

I’ve mentioned before that we use removable drives to transfer data between the data center and the office. Some of those files are very large—50 gigabytes or larger. The other day we discovered an error in one of the files that we had here at the office. The original copy at the data center was okay. Somewhere between when it was created at the data center and when we read it here, an error crept in. There is plenty of room for corruption. The file is copied to the removable, then copied from the removable, transferred across the network, and stored on the repository machine.

The quick solution to that problem is to copy with verify. That takes a little longer, but it should at least let us know if a bit gets flipped.

Saturday we ran into another error when copying a file from the removable drive to its destination on the network:

F: is the removable drive. The machine it was connected to disappeared from the network. I’m still trying to decipher that error message. I can’t decide if we got a disk error or if there was a network error. Did the disk error cause the network error? Or perhaps Windows considers a USB storage device to be a network drive. We removed the drive from that machine, connected it directly to the repository machine, and the copy went just fine. The file checked out okay, leaving me to think that the first machine is flaky.

About a year ago we purchased a Netgear ProSafe 16 port Gigabit Switch (model GS116 v1). It’s been a reliable performer, although it does get a little warm to the touch. Still, we ran it pretty hard and it never had a glitch. We bought another about 6 months ago. Last month, the first one flaked out and started running at 100 Mbps. Not good when you’re trying to copy multi-gigabyte files. This morning, the other one gave up the ghost completely and wouldn’t pass traffic at all.

I suspect that excess heat caused both switch failures. The units were operating in a normal office environment where the ambient temperature is between 75 and 80 degrees. There was no special cooling and we ran the units pretty hard what with the Web crawler and all. As I said, the switch did get very warm to the touch. In a normal office configuration where the switch doesn’t get a lot of traffic, it probably will hold up fine. But I would not recommend this switch for high duty cycles unless you have special cooling for it.

October 2nd, 2008

Bailout, Again

Congress never ceases to amaze me.  Following the unexpected defeat of the measure on Monday, the Senate decided to take a crack at writing something that could pass.  The Senate’s technique was nothing short of brilliant.  Rather than debating the need for the measure and perhaps rewriting provisions to make it more palatable, they did what any good politician would do:  they added bribes to sweeten the deal for those Representatives who voted against it. 

Because the House is responsible for introducing legislation that deals with budgetary matters, the Senate didn’t introduce this as new legislation. Instead, the Senate greatly amended existing legislation, H.R.1424, which apparently has been bouncing around between the House and the Senate for 18 months. The Senate bill now contains the original enacting clause:

To amend section 712 of the Employee Retirement Income Security Act of 1974, section 2705 of the Public Health Service Act, section 9812 of the Internal Revenue Code of 1986 to require equity in the provision of mental health and substance-related disorder benefits under group health plans, to prohibit discrimination on the basis of genetic information with respect to health insurance and employment, and for other purposes.

And then:

Strike all after the enacting clause and insert the following:

And there follows the 110 pages of the House’s Emergency Economic Stabilization Act of 2008 and 350 more pages of additional legislation.  The bill, now 451 pages long, contains three divisions:

Division A: The Emergency Economic Stabilization Act of 2008.

This is pretty much the same legislation that failed to pass in the House on Monday.  A notable addition is Section 136, which temporarily raises the FDIC and National Credit Union Share Insurance Fund deposit insurance limits from $100,000 per account to $250,000 per account.  The idea behind this move is to reassure businesses and prevent them from moving their deposits from small banks to larger banks that are viewed as more secure.

Division B: Energy Improvement and Extension Act of 2008

Provides tax incentives and credits for renewable energy, investments in cleaner coal technology and biofuels, plug-in electric cars, energy conservation, and many other things.  It’s a scattershot energy bill that’s been working its way through the legislature as H.R.6049 since May of this year. 

DIvision C: Alternative Minimum Tax Relief Act of 2008

This part began life in June 2008 as H.R.6275, to provide relief from the alternative minimum tax for middle- and low-income taxpayers.  The original version was a few dozen pages long.  The new version is almost 200 pages in length and includes all manner of personal and business tax cuts, credits, and deductions.  And then there are the miscellaneous provisions, among them:

  • The Paul Wellstone and Pete Domenici Mental Health Parity and Addiction Equity Act of 2008 states that health insurance companies have to provide the same level of coverage for mental health benefits as they do for medical and surgical benefits.
  • A reauthorization of the Secure Schools and Community Self-Determination Act of 2000,
  • Disaster relief legislation.

If you read the Table of Contents for each of the divisions, it becomes very clear that the Senate in its infinite wisdom targeted certain provisions to individual parties and even individual states in an attempt to win over those who voted against the original bailout plan.  In doing so, they had to be careful not to add something that would cause members who originally voted for the plan to vote against it.  Honestly, though, I don’t think there was much chance of somebody changing his vote from “aye” to “nay.”

Estimates place the cost of new provisions at about $105 billion, a very large part of which would fit under the category of “pork barrel projects.”  Both McCain and Obama voted for the measure, despite recent public statements about the evils of pork barrel spending.

It’s now five minutes to noon in Washington.  Party whips have been making the rounds in the House, trying to get members’ commitments to support the bill.  The House is set to convene at noon, and will be voting on this bill.  I’d like to think that those who voted against it on Monday will stand their ground, but I fear that politics will once again trump good sense.

October 1st, 2008

What is the problem?

Since Paulson, Bernanke, and Company aren’t forthcoming with layman’s explanations , I thought I’d take a little closer look at the problem to see if I could understand what’s happening here. What I found is that the problem is very large—potentially catastrophic. I’m also astounded by the hubris that got us into this mess, but that’s a topic for another time.

The fundamental problem today is that there is a lack of confidence in the financial system. Not so much by you and me, but by banks that lend money and large institutions that buy the banks’ debt for investment purposes. Investors demand more documentation and higher interest rates, which causes the banks in turn to charge higher interest rates and to be more careful in selecting borrowers. Ultimately, fewer individuals and companies will qualify for loans, and those who do will end up paying higher interest rates. Credit hasn’t dried up, but it’s a lot less available than it was previously. Creditors and investors got burned (in large part by their own greed), and now they’re being very (perhaps overly) cautious.

A key factor in the lack of confidence is that financial institutions’ portfolios have large numbers of assets that are difficult or impossible to value. A simple example would be mortgage backed securities. A mortgage backed security is, in essence, a bundle of loans that a lender has packaged up and sold as a single investment instrument. The price the bank asks for the package is based on the quality of the mortgages that make it up: their face value, the interest rates they pay, their maturities, the quality of the collateral backing them, and the credit worthiness of the borrowers. Industry rating agencies such as Moodys and Standard & Poor’s assign ratings that give some indication as to the risk level of such securities. Investors demand higher interest rates (or, sometimes, lower prices) for more risk.

But there’s a problem. It’s exceedingly difficult to know the real value of even one mortgage. Imagine the difficulty involved in computing the real value of a box that contains 100 mortgages. If you buy the box directly from the originating bank then you can have some confidence in the box’s value. Assuming, of course, that you trust the bank and the rating agency. But if you as an investor want to sell that box on the open market, you’re subject to the whim of the market. When things are going well, that’s a good thing. People are willing to spend $1,000,000 on that box of low-risk loans that pay an average of 5% interest because they have confidence that they’ll get their payments and that they can re-sell the box if they need the cash.

But the market can go sour. If a bank starts reporting higher levels of loan defaults, then every box of loans that the bank sold becomes less valuable because the probability of the box actually paying the stated interest or being sold at face value is lower. You could find that the market value (the price people are willing to pay for) your million-dollar box of loans is only $900,000.

Because of the difficulty in valuing mortgage backed securities (and many other investment vehicles), a very commonly used method of determining their value is mark to market. The box’s value is what people are willing to pay for it. Generally accepted accounting principles state that financial institutions use mark to market in determining their assets’ values. And transparency laws on the books insist that banks and other regulated financial institutions calculate the value of their assets on a daily basis.

Now, imagine that you’re a small bank. You take deposits from customers and lend money to individuals and local companies. You also buy high-quality (i.e. low-risk) mortgage backed securities to invest the depositor’s money. You’re very conservative with your investments, always get good documentation from your borrowers, and everything’s going great. Whenever you need a little cash to meet depositors’ demands, you sell one of those securities on the open market. This is how the financial system works when people have confidence in it.

But if the bank from whom you bought those mortgage backed securities starts seeing a higher than usual number of loan defaults, the value of the security you’re holding in your vault declines. As that bank’s troubles deepen, so do yours because it becomes increasingly difficult to sell those securities–not only at face value, but at all. If the originating bank defaults, you could very well end up unable to find a buyer for your box of loans. At any price.

You know that if you could open the box and show the individual loans to potential buyers, you could at least salvage some value from your investment, but that’s not the way it works. Your investment is essentially worthless. And since you have to report your financial condition on a daily basis, it doesn’t take long for the public (your depositors) to find out and begin demanding their money. You’re wiped out, and you didn’t do anything wrong.

I’m not just spinning a fantasy here, by the way. Such things do happen, and they have a cascade effect. As more securities become worthless, investors become less inclined to buy any securities. Institutions who hold those securities are unable to sell them in order to meet other obligations, and the financial system grinds to a halt.

That’s it in a nutshell. I’m sure there are economists who will say that the problem is more complex than that. In some ways, that’s true. I haven’t mentioned the more creative investment vehicles (like tranches) and the mania that helped get us where we are. I’ve tried to concentrate on what the problem is, rather than what caused it.

The problem, then, is that financial institutions are holding securities that nobody wants to buy. The securities’ values, based on mark to market, are essentially zero because there is no market for them. The truth is that some of those securities are very valuable and some really are worthless. But right now there’s no way for buyers to know which is which. The point behind the bailout is to create a market—to buy securities and by doing so restore some investor confidence so that they, too, will begin buying.

So that’s the problem and the rationale behind the proposed solution. Whether the proposed solution is right or even necessary is still a matter of some debate among economists, and also in my own mind.

September 30th, 2008

No bailout. Yet?

I was pleasantly surprised yesterday when the House failed to pass the Emergency Economic Stabilization Act of 2008.  (Be patient.  That site is getting hit pretty hard right now.  You might be better off visiting your favorite news site for the full text.)  From news reports over the weekend and yesterday, I was pretty sure that it was going to pass.  It’s interesting to note that approximately 40% of Democrats and about two-thirds of Republicans voted against the bill.

What would be much more interesting to me is why the bill didn’t pass.  News reports contain nothing but a bunch of partisan sniping.  Some say that Speaker Pelosi’s comments before the vote angered Republicans.  If true, that doesn’t say much for those representatives whose feelings were hurt, or for Rep. Roy Blunt, who was dumb enough to put forward the suggestion.

I wonder how many voted against the bill because it was unnecessary, or because it was bad legislation.  I also wonder how many members’ votes were cast in response to constituents’ support or opposition.  According to an Associated Press analysis, 13 of the 19 most vulnerable (in the upcoming election) Republicans and Democrats voted against the bill.

And that brings me to the real point I’ve been struggling with:  where should a Congressman’s loyalty lie?  Should he be more concerned with the good of the country, or should he be more concerned with satisfying the whim of his constituents so he can get re-elected?  On the same note, do we elect our Congressmen because we trust them to do what’s best for the country while keeping our interests in mind, or do we elect them to bring home the bacon?

I need to think more about that one.

In all the hype surrounding this mess, I’ve heard a lot of alarmism, but very little hard fact.  Paulson and Bernanke warn us of “dire consequences” if we don’t provide some assistance.  I’d like to trust them, but I trusted President Bush and his advisers when they told us that we needed to invade Iraq.  I trusted that they had credible evidence of nuclear or biological weapons development and that they actually had a plan to transition Iraq into a stable, functioning democracy.  It turns out that they had no credible evidence and their “plan” amounted to “kill the bad guys and everybody else will join hands singing Kumbaya.”

So excuse me if I’m skeptical when the administration all of sudden decides that the economy is in grave danger and the only way out is to spend 700 billion dollars.  This is the same administration (and largely the same Congress) that has been ignoring the problem for the last few years, insisting that there is no cause for concern.  Bernanke and Paulson may very well be smart guys.  But their association with the current administration makes them suspect in my eyes.  They’ll have to provide me with a whole lot more evidence than their vague warnings of “dire consequences” before I’ll believe what they have to say.

September 25th, 2008

Just say “No” to the bailout

Have you read the text of the Bush Administration’s proposed 700 billion dollar bailout of financial institutions?  If you don’t want to wade through the three-page proposal (although it is written in reasonably clear English), this summary will tell you all you need to know.  (Thanks to David Stafford for the link.)

When that proposal was presented, Congress was given the opportunity to exercise its most important obligation under the Constitution:  to serve as a check on the Executive branch.  Were Congress seriously interested in doing what’s best for the economy, for the taxpayers, and for the country, they would have just said, “No.”  Instead, they view the proposal as a starting position and are taking this opportunity to ingratiate themselves with their constituents and extend government’s control over private lending.  You doubt that?  Consider:

  • Democrats are pushing for legislation that allows bankruptcy judges to rewrite mortgages to “ease the burden” on homeowners who are facing foreclosure.  This contentious issue probably will be dropped in favor of getting a bill passed.
  • Democrats want any proceeds of the bailout to go into a fund designed to pay for housing for poor families.  This is the old shell game.  On one hand, they’re telling us that we’ll “get back” much of that $700 billion when the assets are sold.  The reality is that any proceeds will go into the general fund, which Congress can squander at their whim.
  • Lawmakers on both sides have agreed in principal to limit pay packages for executives whose companies benefit.  This is largely a symbolic move, as executive pay is a drop in the bucket compared to the amount of money we’re discussing.  But it looks good to the voters.  “I voted to limit executive pay!”
  • Absent the “No” that they should give, Congress is right in insisting that it be given more control over the bailout than what the proposal allows.  But I doubt that they’ll exercise restraint.  I fear that they’ll make a serious power grab.  For example, Representative Barney Frank has said: “we’re now the biggest mortgate holder in town, and we can do serious foreclosure avoidance.”  That frightens me, as I think it should frighten anybody.

I’m not convinced that this bailout is at all required.  Were Congress to do the right thing—nothing—markets would take an immediate tumble, rebound a bit, and then financial institutions and others affected would get back to business.  Sure, it’d be a struggle.  But the relatively short-term pain involved will be much less than the long-term pain that this bailout legislation will undoubtedly cause.

Needed or not, I’m certain that it doesn’t have to happen within the next week, as Secretary Paulson and Federal Reserve Chariman Bernanke insist.  I’m always nervous when Congress acts at all, and I get very, very scared whenever Congress rushes through legislation to “address a serious problem.”  Eight years of an administration and a Congress that have both lost all concept of the term “fiscal restraint” has taught me that much.

September 24th, 2008

The Last Sucker Theory

Join me in a little thought experiment.

Seal a 100 dollar bill in an envelope, affix a price tag to the envelope, and write $110 on it.  Then put it up for sale, telling potential buyers that if they spend $110 on this envelope, they can turn around and sell it for more.  Don’t worry about what’s in it.

Somebody buys it, affixes a new price tag that says $125, and sells it to somebody else who also increases the price and turns it over.

This goes on for some time, with each new buyer swapping out the price tag.  Somewhere along the way, somebody gets the bright idea of putting 10 envelopes into a larger, fancier-looking envelope.  Why not make 10 times the profit in a single transaction, right?

And the party goes on.  The fancy envelopes beget pretty printed shoeboxes and the prices go up again.  Nevermind the party poopers screaming, “But what’s in the box?”  Nobody cares what’s in the boxes.  They must be valuable, right?  People keep paying more for them.

One day, the holder of a refrigerator-sized package wrapped in gold paper and sporting all manner of ribbons and bows tries to sell it for $100,000,000, and fails.  The lender who floated him the loan to buy the thing takes it back and decides to sell it at a loss just to get it off the books.

But the lender finds out that he can’t sell it at any price.  A lot of people have been having trouble selling their pretty boxes and envelopes.  Not only that, but lenders have a lot of money tied up in those pretty boxes, meaning they don’t have any money to lend for other purposes.

Desperate, the lenders start trying to sell their assets at ever-lower prices, trying to get something out of them.  But nobody’s buying.  Nobody wants a pretty box that he can’t resell at a higher price.

Finally, the lender finds a buyer who says that he’ll be happy to buy the box, provided he can open it beforehand to see what’s inside.  The lender reluctantly agrees and looks on as the potential buyer opens the box and pulls out 100 pretty shoeboxes.  Inside each shoebox there are 10 fancy envelopes, each of which contains 10 plain white envelopes holding a single hundred dollar bill each.  The lender’s $100,000,000 “asset” is worth $1,000,000.

My dad used that little parable (also known as the last sucker theory or, more commonly, the greater fool theory) to explain the events leading up to the savings and loan crisis in the late 1980s.  It’s equally apt in explaining much of the current financial meltdown, what with the mortgage backed securities that were “backed” by worthless mortgages, investment firms that were leveraging their investments 35-to-1, and all the while knowing that they were just riding the wave—hoping they weren’t the ones holding the box when somebody demanded that it be opened.

September 22nd, 2008

Removable Drives

Although we’ve moved the crawlers and a large part of our workflow to a co-location facility, we still do some of our processing here at the office.  So on a daily basis I hop on my bike and ride down the road (a little over a mile) to pick up the daily data dump.  We do have a VPN to the data center, but the 100 gigabytes in a daily dump is more than we could download.

We have two USB hard drives that we use for this shuttle service.  It didn’t take us long to learn that it’s best to have two of the same type of drive.  That way all you need to carry back and forth is the drive itself.  You can leave a USB cable and power supply at each location.

In the short time we’ve been doing this, we’ve tested a number of different drives.  My favorite is the 500 GB Maxtor Personal Storage 3200.  It has a very conveinent form factor (just a brick), the power supply is not a wall wart, and it has a standard USB 2.0 connector.  We have used several of these around the office, and I have one at home.  It’s been a reliable drive and a good performer.  Unfortunately, you can’t get them anymore.

We picked up a couple of Iomega eGo 1TB Desktop Hard Drives on sale at Fry’s for about $160 each.  Hard to argue with the price, and the form factor is nice.  However, the drive made a lot of clicking noises, got very hot (uncomfortable to touch), and one of them failed outright.  The other started giving errors and we ended up taking them both back.  Considering my experience with the eGo, and also the experience I’ve had with the 160 GB unit I bought several years ago (slow, noisy, and hot), I would not recommend Iomega removable hard drives.

About two weeks ago we picked up two Seagate FreeAgent 1 TB (it’s a PDF) drives on sale at Fry’s for around $150 each.  I’m not real wild about the form factor and the wall wart power supply, nor do I much care for the mini-USB connector.  But the drives are quiet, fast, and reliable.  They do get significantly hotter than the 500 GB Maxtors that I prefer, but not near as hot as the Iomega drives.  We also have two of the 750 GB FreeAgent drives in the office for backups and haven’t had any problems with them.

So far, I haven’t encountered the mysterious drive removal problem with the FreeAgent drives.  I’m beginning to think that permissions might have something to do with it.  Let me explain why.

I edited the security properties on the FreeAgent drives (right-click on the drive, select Properties, and then click on the Security tab) to give Everyone full control.  This allows me to attach the drive to any of our machines at the office or at the data center, and be able to add, copy, rename, delete, or otherwise manipulate files without trouble.  Since I did that, I haven’t had any trouble removing the drive.  I’ve yet to try this on the other drives I was having trouble with.

September 15th, 2008

Hurricane Rescue

Almost every year during monsoon season in the Phoenix area, some idiot will drive around a barricade and attempt to cross a flooded low water crossing. In most cases, search and rescue workers are successful in plucking the occupants from the stranded car. Invariably, the driver will claim ignorance, despite signs warning of the danger, barricades across the road, and many years’ experience living in the area. It’s impossible to live in the Phoenix area for any length of time and not know of the dangers inherent in driving through flood waters.

Things got so bad in Arizona that they finally passed what is termed the Stupid Motorist Law which, when translated to simple English, says that a motorist who drives around barricades to enter a flooded stretch of roadway may be charged for the cost of his rescue. I don’t know if the law has actually prevented anybody from trying to drive through a flooded area. It seems to me that if the threat of being swept downriver and drowned doesn’t deter somebody, the prospect of having to pay for rescue won’t raise a red flag either. In any event, I support the law simply because I believe that people should have to pay for their own stupidity–especially when said stupidity puts others’ lives at stake.

As hurricane Ike approached last week, officials in Galveston and other coastal areas urged citizens to evacuate, warning of a possible 25-foot storm surge and “certain death” if they stayed behind. By all reports, most people heeded the warnings and got out before Friday at noon. But somewhere between 100,000 and 150,000 people decided that they knew better, and stayed behind.

I was in the Williamson County Emergency Operations Center from midnight until 6:00 AM on Saturday. The eye of Ike made landfall at about 2:00 AM. The entire time I was at the EOC, I heard reports of 911 calls from people who had elected to stay behind, begging for somebody to come help them. Of course, nobody was going to send rescue workers out in the middle of a hurricane. Those people who elected to stay truly were on their own–just as officials had said they would be.

Today, 48 hours after Ike came roaring through, we’re still in the middle of what Governor Rick Perry is calling the largest search and rescue operation in Texas history. Over 1,500 rescue workers are searching Galveston and surrounding areas for people who are stranded in their houses, still surrounded by floodwaters. So far, every person I’ve seen interviewed after being rescued said pretty much the same thing: “I never thought it would be so bad. I was wrong to stay.” I’ve yet to hear anybody say they didn’t know that the storm was coming, or they didn’t hear the warnings to evacuate.

Those who stayed and survived were very fortunate that the projected 25-foot storm surge never materialized. The estimated 13-foot surge did a very good job of devastating the area. I imagine that nothing would be left had there been twice as much water, and it’s doubtful that any of the holdouts would have survived.

Those who did survive (and we may never know how many got swept away by the storm) are now stranded in the attic or on the roof, with no services, no food or water, and no way to get out except being rescued. It’s unfortunate that Texas doesn’t have a Stupid Homeowner Law that allows us to bill those people for the cost of their rescue. Whereas I fully support a citizen’s right to stay even in the face of “mandatory” evacuations, I also believe that they should bear the consequences, including paying the cost of pulling them out of an area they were advised to evacuate three or four days ago.

Here in the Austin area, Ike had almost no effect. We got a little bit of wind Saturday morning. Some parts of the area might have received some rain. We didn’t get a drop at our place.

September 10th, 2008

Hurricane Ike

The latest projection of hurricane Ike’s path puts a tropical storm right over our house sometime early Sunday morning.  Of course, that’s just a projection based on current conditions and input from a half dozen climate models.  The storm could still move considerably north or south of its projected track.

I typically rely on the National Hurricane Center for information about hurricanes and tropical storms, and the Navy/NRL Tropical Cyclone Page for satellite pictures.  I recently learned about this Tropical Cyclones page, which has a whole bunch of graphs and images from many different places.  Not only does it have the graphs, it also has links to the pages the graphs came from.  I didn’t realize there were so many different hurricane tracking sites out there.

I especially liked this image, which shows Ike’s historical track and its projected path.  I’ve cropped it to show my area.  Click on it to get the full-sized image.

Hurricane Ike projected path

A tropical storm here shouldn’t be too big a deal.  Although Ike is expected to be a major hurricane when it makes landfall, we’re about 200 miles inland.  The storm will lose a lot of its intensity in the 24 hours between when it hits the coast and when we begin to enjoy it.  I expect some high winds, but not enough to do major damage.  And we certainly could use some rain.  I suspect, though, that we’ll get too much too soon and there will be some flash flooding.

I wouldn’t want to be in the grocery store between Thursday and Sunday.  People around here tend to panic when they see a storm coming, even though it’s highly unlikely that we’ll lose power or suffer other storm related inconvenience.  You do not want to be the only thing between the last case of bottled water and a woman who has visions of hurricane Katrina in her mind.  I wouldn’t be surprised if there are more injuries resulting from the pre-storm supermarket rush than from the storm itself.

All told, it should be an interesting weekend.

September 8th, 2008

An Hour A Day

I hear a lot of people say that they want to change their lives: lose weight, make more money, learn something new, take up a new hobby, accomplish some physical challenge, etc. Sadly, most of those people then go on to say that they wish they could do that, but they can’t. And most of the time the reason they can’t ends up being a variation on one of these two:

  1. I don’t have enough time.
  2. I’ll be too old before I complete it.

Excuse me, but both of those reasons are bullshit. The second one, especially. How old will you be if you don’t complete it? You can sit there and wish all you like, and complain about how you’re not getting any younger and the world is passing you by, or you can decide that you want to do something. Either way, time will pass. But after five or twenty years, the person who actually tries will have something to show for it. The person who spent that time complaining about how it’s too hard or will take too long will have nothing but bitterness.

As for the “I don’t have enough time” complaint: that’s crap, too. You might not have enough time to dedicate your life to a new pursuit, but you most definitely have time to improve your life. All it takes is an hour a day, and most people spend way more than an hour every day watching TV, surfing the Web, or doing other things that are neither relaxing nor productive.

Let me give an example. I’ve mentioned a time or two that I used to run marathons when I was younger. I since stopped running and took up bicycling, but lately I’ve wanted to get back into running. So I set myself a goal of running a 10K race (6.2 miles). I can’t run 6 miles today. I’m hard pressed to run even one mile without stopping. But I can run a bit, walk some, run a bit more, and so on for an hour every day. I don’t know yet how long it’ll take me to build up to 6 miles, but every day I go out I find that I can run a little bit further. And in a few months I’ll be up to 6 miles.

Another example is education. A lot of people think they need school in order to get educated. It’s true that if you want a degree or a certification, you need to attend formal classes. But if you just want to learn about a particular topic, you have all the educational resources you need on the Web, in your public library, bookstores, and Amazon.com. All you need to do is start reading. Try reading on your topic for an hour a day.

Want to learn the piano? Spend an hour a day practicing. Build strength? Get some weights and a beginner’s book and spend an hour a day lifting. Learn to write better? Practice writing an hour every day. Pretty much whatever you want to do, you can get a very good start on it by allocating one hour per day, and I don’t know anybody who doesn’t have at least an hour per day to spend on self improvement.

Try it. Rather than complaining about how something is too hard or will take too long, sit down and plan how you can accomplish that thing you’ve always dreamed about. Spend an hour per day working toward your goal. You’ll be surprised at how much progress you make in just a few weeks. The longer you work at it, the more you’ll learn and the better you’ll become, and the more you’ll want to continue. All you have to lose is time, and the rewards are potentially limitless.

September 1st, 2008

Disposing of a refrigerator

Some friends gave us a refrigerator about 10 years ago when they were moving to a new house.  It really wasn’t much of a refrigerator, even back then, but it fit nicely in the laundry room.  We used it mostly for beer and sodas, and the top freezer helped with the overflow from the main fridge in the kitchen.

A few weeks ago it started getting very noisy and then Debra noticed that the freezer wasn’t keeping things frozen.  It got to making more noise and the louder it got the warmer it got until it wasn’t keeping my sodas very cool, either.  Debra emptied it and I started looking for a way to get rid of the thing.

My first thought was to recycle the thing.  The Williamson County recycling facility takes appliances, but refrigerators and other appliances that contain or contained freon must be accompanied by a certificate showing that the freon has been recovered.  So I’d have to take the thing to a shop, pay somebody to recover the freon, and then haul it to the recycling center.

I was about to break down and pay to have the thing hauled off when somebody suggested that I post it on craigslist.  That worked great.  Sunday afternoon I posted a note in the Free Stuff section, describing the refrigerator’s condition and offering it free to the first person who came to haul it off.  I got a call within an hour, and 30 minutes after that the refrigerator was gone.  Works for me.

I’m sold.  Next time I need to get rid of something, I’ll know to try craigslist first thing.

August 25th, 2008

Remove Hardware Mystery

Updated

Two different times now, when trying to “safely remove” a USB hard drive, I’ve had this error message pop up when I’m not actually accessing the drive.

I closed all open applications, even logged out to make sure that no user applications were open.  When I logged back in and tried to remove the device, I got the same error message.

The Windows 2008 Resource Monitor tells me that the System process (process ID 4) has two files open on the device:  F:\$LogFile (NTFS Volume Log) and F:\$Mft (NTFS Master File Table).  Why it’s holding those files open when I told it to remove the device is beyond me.  And I have absolutely no idea how to tell Windows to let go of the device.

I just realized that both times this happened, I was logged in to the machine via Remote Desktop.  That shouldn’t be an issue, but it’s probably worth looking into.  I know that I’ve removed drives via Remote Desktop before.

If anybody has a clue why this is happening, I’d sure like to hear about it.

Update Wednesday, August 27

It happened again yesterday, so I downloaded Process Explorer to see if I could get a little more information.  Searching for handles to “F:\” produces these search results:

That tells me what handles are open, but it sure doesn’t give me much in the way of useful information.  It seems like the remove hardware action should tell those services to let go of the handles.

There is a way within Process Explorer to force-close those handles, but attempting to do so results in a dire warning about possibly causing a crash or instability, and I wasn’t prepared at the time to crash my server.  Closing the Remote Desktop window and logging in as Administrator at the machine’s console didn’t allow me to remove the drive.  So I just pulled the plug.  No ill effect.

After disconnecting the drive yesterday, I took it to the data center, copied some stuff to it, and brought it back here.  I connected the drive, ran the program that copies data off the drive, and attempted to disconnect it again.  That time it worked.  As far as I recall, I performed the same steps as I always do.  Why it worked this last time when it hasn’t worked previously is beyond me.

August 20th, 2008

Should VPN be this hard?

Last week we moved the crawlers from our office to a real data center where we can get more, and more reliable, bandwidth.  Getting everything installed and working wasn’t too much trouble, although the next time I have to do something like that I’m going to do a lot more pre-installation work here at the office before taking the machines to the data center.  Installing and configuring 10 machines while standing in the cold, noisy data center isn’t my idea of a good time.

Having machines at the data center means that we need some way to log in and check on them.  Not a problem, as the Cisco security appliance we bought supports VPN.  And configuring the Cisco IPSec VPN was quite simple.  I was pretty happy when, with just an hour of looking at the documentation and fiddling with the configuration, I was able to log in to the VPN from my laptop.  I packed up my stuff and headed back here to get everybody set up to use the VPN.

And then I found out that Cisco’s IPSec VPN client won’t run on 64-bit versions of Windows.  Nor does Cisco have any plans to upgrade it.  Since I’m not willing to create a 32-bit virtual machine just for running the VPN client, that leaves me with the option of configuring the router for some other type of VPN.  And there things get difficult.  The documentation that came with the router doesn’t discuss any type of VPN configuration other than IPSec, and the online documentation I’ve seen makes the assumption that I understand everything there is to know about VPN.  It gets confusing in a real hurry.

There are VPN standards.  There are so many, in fact, that no mere mortal can begin to understand them.  It might as well be a free for all with all those competing protocols.  Just the acronyms are enough to push a questionably sane person such as myself over the edge into babbling lunacy.  I’ve yet to find a document that explains, in terms a reasonably bright person who hasn’t passed Cisco’s certification can understand, how to configure the VPN.  I can’t even find a good discussion of the benefits and drawbacks of the different VPN technologies:  IPSec, L2TP, or SSL.

I also need to configure VPN on our pfSense box here at the office.  That looks almost as daunting as the Cisco’s configuration and the documentation is, if you can imagine, even worse.

I realize that much of my frustration stems from my lack of expertise in this area.  I’m a programmer, not a network admin.  But I have to think that VPN just doesn’t need to be this hard.

I can find lots of “how VPN works” types of discussions online, but they’re presented at a very high level.  There also is plenty of detailed documentation about VPN configurations for very specific situations.  But I’ve found nothing in the middle.  Something like “Simple VPN configuration for people who don’t live and breathe this stuff.”

Pointers to good discussions of the different types of VPN, and good tutorials about configuring VPN on the Cisco ASA or pfSense would be greatly appreciated…

August 13th, 2008

The Government Rant

The best thing about our government is that it never ceases to amuse me. It’s also continuously annoying, but I guess you have to take the bad with the good. It’s not the government itself that amuses me so much, but rather the absurd things that our illustrious Congresscritters do and say in an attempt to garner votes. The most amusing (and also the most frustrating) thing is that constituents continue to be taken in. Rather than making an effort to come up with a solution ourselves, we argue over which totally unworkable plan our elected representatives should vote on. This gives the leeches in Washington Congress incredible leeway to do anything, and then spin their positions to best advantage.

Examples abound. Let’s look at some of the more recent.

Dependence on foreign oil

Our country’s dependence on foreign oil has been a major problem since the Arab oil embargo of 1973. In the 35 ensuing years, Congress has put forth all manner of proposals to “fix” the problem. We’ve funded research into solar, geothermal, tidal, and other natural energy sources, provided incentives and subsidies for domestic oil exploration, coal, ethanol, and all manner of questionable energy saving technologies. Today our government has much more control over energy policy than it did in 1973 and yet we’re more dependent on foreign oil than we were back then.

Seven administrations and countless members of Congress have been “doing something about the problem” for 35 years, and the problem has gotten worse. And yet the vast majority of Americans look to Congress and the President for a solution to high gas prices, all the while cheering for or ridiculing the laughably simple minded, short term proposals that are put forth. Our representatives, of course, couldn’t care less. All they have to do is make themselves look good to their own constituents. As long as they can keep the voting public believing that government is the solution, their jobs are secure.

Every thinking American (and, sadly, I’m beginning to believe that the number is falling fast) knows that the solution to our energy problems requires conservation, domestic oil and gas production, development of nuclear plants, exploitation of wind, thermal, solar, and other natural sources, and research into more energy efficient transportation and buildings. We won’t solve anything unless we address all of those areas. And it’s going to take time. Government has proven that it’s incapable of formulating and implementing a workable energy policy. It’s time to get government out of the picture. No more subsidies, incentives, or preferential treatment. Let the market decide.

Tax Rebates

This is one of the dumber things I’ve seen Congress do. And, yes, I realize that both the 2001 and the 2008 rebates were initially proposed by President Bush. That doesn’t relieve Congress of their complicity and their ultimate responsibility. The 2001 rebate was “justified” by a “budget surplus”–a surplus that anybody with a fifth grade education knew was an illusion. This year’s rebate was “justified” by the current economic situation. Congress would have you believe that a windfall of a few hundred dollars (up to $1,200, as I recall) would “stimulate the economy” and soften the recession. Any thinking person could have told you that the result would be a short term spike in consumer spending, followed by a quick return to normal. I can’t prove this yet, but I suspect that it also resulted in people putting down payments on things they can’t afford, figuring they’d find a way to make the monthly payments.

Congress, of course, knew that the tax rebates wouldn’t have an effect on the economy other than to increase the size of the federal debt. But that’s okay. What’s a few billion more dollars compared to the time honored tradition of buying votes? It is an election year, after all. Besides, it made for good press coverage and retail store managers drooled over the prospect of Christmas in July. The rebates seem so popular that Senator Obama proposed a $1,000 rebate to fight energy costs.

The reaction of those receiving the rebates was predictable. Most squandered it like drunken sailors on leave. Those few who know the names of their Congressmen or Senators might have lifted a glass in salute, but most just thanked the government for the handout. That’s what surprises me the most. It’s like having somebody cut your arm off at the shoulder and then thanking him when he returns the forearm and hand. Idiots.

The “mortgage crisis”

This one is fun because there are so many levels of idiocy. Lenders made high-risk loans to people who were demonstrably incapable of paying them back, then sold those loans to a government sponsored enterprise, which ultimately will be bailed out by taxpayers when the original borrowers default.

When borrowing money in good faith, both the lender and the borrower are responsible for ensuring that the money can be paid back. But when the lender is just a middleman who gets paid for making the loan and selling it to somebody else, there is little incentive for him to vigorously check the borrower’s documentation. On the contrary, there is ample incentive for him to be very creative in putting together a loan package, both by making the terms of the loan appear attractive to the borrower and by making the borrower look attractive to the third party who’s buying the loan. Sure, the middleman will eventually be found out, but the short term rewards are incredible.

And when the ultimate buyer is a government sponsored enterprise like Fannie Mae or Freddie Mac, there is almost no oversight. When you have, with government’s blessing, a virtual monopoly on the secondary mortgage market, you know that you’ll get bailed out if things go bad. So where’s the incentive to insist on real documentation for the loans that you buy?

I’m not an economist by any stretch of the imagination. I’m not even a financial analyst. But I’m not an idiot, either. I and many others saw this coming three years ago. Congress ignored the problem at the time, or discounted it as scare mongering. I’ll go out on a limb here and say that most of them probably knew what was coming. But they also knew that there wasn’t anything they could do about it and that bringing it up would be very unpopular. Our elected representitives are many things, but stupid is not one of them.

Now that the real extent of the problem has become apparent, Congress is all over it with one proposal after another. They’re “doing something about the problem.” They know that there are only two possible solutions: either pump money into Fannie Mae and Freddie Mac to keep them afloat, or cut them loose and let people finally endure the consequences of their actions. We know, just by the the nature of elected officials, what their solution will be: another hundred billion dollars or more shelled out to fix a problem that Congress created in the first place. And We the Sheeple just nod our heads and thank Congress for taking care of us once again.

More is better?

All three of the above examples demonstrate extreme incompetence on the part of government. The Congress-proposed solution to those problems, as with all others, is more government regulation. As if making even more and larger bureaus, agencies, and departments will somehow transform government into an intelligent and effective organization. And we let them do it! When will people learn that the cure for a headache is to stop beating your head against the wall?

I used to get upset when I’d think about this stuff. I used to rant and carry on about the proper function of government, and how intrusive government is in our daily lives. But nobody listens. Nobody seems to care. I learned a while back to stop bashing my head against that particular pile of bricks. Now I just laugh and hope that the coming violent overthrow (which will almost certainly happen if government continues on its current path) doesn’t occur until after I’m gone.

August 10th, 2008

Paranoia versus productivity

We had an interesting discussion at the office about how much validation a collection type should do in its constructor. The key question, I think, came down to this:

If the constructor can determine that using the instantiated object will throw an exception, should the constructor fail rather than returning the instantiated object?

In other words, if I know that the instantiated object won’t work, shouldn’t I just throw the exception now, rather than let you be surprised later?

There are two extremes here: 1) the constructor should go to heroic efforts, and; 2) let the buyer beware. I tend to lean towards putting the onus on the caller, figuring that whoever is instantiating the object knows what he’s doing.  Let me provide an example.

Consider the .NET SortedList generic collection type. To do its job (that is, keep a collection of items sorted), it requires a comparison function. If you don’t specify a comparison function when you call the constructor, the collection uses the default comparison function for whatever type you specify as the key. This sounds simple enough, right? A list of employees that’s sorted by employee number, for example, would be defined like this:

SortedList<int, Employee> Employees =
    new SortedList<int, Employee>();

Since the System.Int32 type (which the C# int type resolves to) implements IComparable, everything works.

But imagine you have an EmployeeNumber type:

class EmployeeNumber
{
    public string Division { get; private set; }
    public int EmpNo { get; private set; }
    public EmployeeNumber(string d, int no)
    {
        Division = d;
        EmpNo = no;
    }
}

Now, if you create a SortedList that’s keyed on that type, you’ll have:

SortedList<EmployeeNumber, Employee> Employees =
    new SortedList<EmployeeNumber, Employee>();

Allow me to show the entire program here, so we don’t get confused.

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace genericsTest
{
    class EmployeeNumber
    {
        public string Division { get; private set; }
        public int EmpNo { get; private set; }
        public EmployeeNumber(string d, int no)
        {
            Division = d;
            EmpNo = no;
        }
    }

    class Employee
    {
        public string Name { get; set; }
        public Employee(string nm)
        {
            Name = nm;
        }
    }

    class Program
    {
        static SortedList<EmployeeNumber, Employee> Employees =
            new SortedList<EmployeeNumber, Employee>();

        static void Main(string[] args)
        {
            Employees.Add(new EmployeeNumber("Accounting", 1),
                new Employee("Sue"));
            Employees.Add(new EmployeeNumber("Dev", 2),
                new Employee("Jim"));
        }
    }
}

If you compile and run that program, you’ll see that it throws an exception when it tries to add the second employee to the list. The program fails because it can’t compare the items. Neither implements IComparable.

Those who lean towards the first extreme above will argue that the SortedList constructor should determine that the key type doesn’t implement IComparable, and should prevent you from instantiating the collection. It should throw an exception because it knows that trying to add items to the collection will fail.

The constructor could do this. It’s possible for the constructor to get the default comparer and call it. If the comparison function returns a value, then all is well. If it fails, then the constructor throws an exception saying, “Sorry, but you didn’t supply a comparison function.”

The only problem with that scenario is that it’s wrong. Not wrong philosophically, but wrong in a very concrete sense. Extending my example will illustrate why.

Suppose you have two different types of employee numbers. Maybe an OldEmployeeNumber that looks like the one I defined above, and a NewEmployeeNumber that has different fields. Because you want to keep both employee number types in the same list, you define a base class, EmployeeNumberBase from which they both can inherit. The definitions would look like this:

abstract class EmployeeNumber : IComparable
{
    // Some common employee number functionality goes here.

    public int CompareTo(object obj)
    {
        throw new NotImplementedException();
    }
}

class OldEmployeeNumber : EmployeeNumber, IComparable
{
    public string Division { get; private set; }
    public int EmpNo { get; private set; }
    public OldEmployeeNumber(string d, int no)
    {
        Division = d;
        EmpNo = no;
    }

    int IComparable.CompareTo(object obj)
    {
        int rslt = 0;
        if (obj is OldEmployeeNumber)
        {
            var o2 = obj as OldEmployeeNumber;
            rslt = Division.CompareTo(o2.Division);
            if (rslt == 0)
                rslt = EmpNo.CompareTo(o2.EmpNo);
        }
        else if (obj is NewEmployeeNumber)
        {
            // OldEmployeeNumber sorts before NewEmployeeNumber
            rslt = -1;
        }
        return rslt;
    }
}

class NewEmployeeNumber : EmployeeNumber, IComparable
{
    public string Country { get; private set; }
    public decimal EmpNo { get; private set; }
    public NewEmployeeNumber(string c, decimal no)
    {
        Country = c;
        EmpNo = no;
    }

    int IComparable.CompareTo(object obj)
    {
        int rslt = 0;
        if (obj is NewEmployeeNumber)
        {
            var o2 = obj as NewEmployeeNumber;
            rslt = Country.CompareTo(o2.Country);
            if (rslt == 0)
                rslt = EmpNo.CompareTo(o2.EmpNo);
        }
        else if (obj is OldEmployeeNumber)
        {
            // NewEmployeeNumber sorts after OldEmployeeNumber
            rslt = 1;
        }
        return rslt;
    }
}

Yeah, I know. That’s quite a mouthful.

The EmployeeNumberBase class implements the IComparable interface, but its implementation just throws NotImplementedException. Furthermore, the class is marked as abstract to prevent it from being instantiated. Only derived classes can be instantiated.

The derived classes each explicitly implement the IComparable interface. The company-defined sorting rules are that old employee numbers always sort in the list before new employee numbers. Within the same type, the numbers are sorted using their own rules. [Note here that my CompareTo implementations aren't terribly robust. They'll return zero (equal) if the object passed is not of a known type, and they'll fail if the passed object is null. But those details aren't terribly relevant to the example.]

Now, the Employees list is created in exactly the same way:

SortedList<EmployeeNumber, Employee> Employees =
    new SortedList<EmployeeNumber, Employee>();

We can then add items to the list:

Employees.Add(new NewEmployeeNumber("USA", 2.002m),
    new Employee("Jim"));
Employees.Add(new OldEmployeeNumber("Accounting", 1),
    new Employee("Sue"));
Employees.Add(new OldEmployeeNumber("HR", 3),
    new Employee("Dana"));

If you make those changes and run the program, you’ll see that it does indeed run, and work as expected, and I didn’t change the comparison function that the constructor sees.

If SortedList had attempted to protect me from myself–that is, call the default comparison function and throw an exception because the comparison function had failed–then this final code would not work. By trying to protect me from myself, it would have prevented me from doing what I wanted to do.

Understand, the above is something of a contrived example. I certainly can’t imagine implementing the employee list that way, even if I did have different employee number types. But somebody else might think it’s a perfectly reasonable thing to do. The point is that there could be very good reasons for instantiating a keyed collection with a key type that does not have a valid comparison function. The constructor cannot know if comparisons will fail.

Which brings us back to the original question: how hard should a collection class (or any library object) try to prevent you from instantiating an object that will fail? In my opinion, the constructor should instantiate the object if the immediate parameters look reasonable. My reasoning is that it’s extremely difficult, if not impossible, to know how the caller will be using the class. As you saw above, making broad assumptions about types in a polymorphic environment can be fatal.

This reasoning extends far beyond the question of how a collection class’s constructor should behave. As programmers, we have to strike a balance between paranoia and productivity. We have to decide daily how much trust to put in the code that calls our methods, and how much we can depend on the code we call. Do we write classes that hold the programmer’s hand to help him across the street, or do we provide a “walk” signal and a warning that says, in effect, “If you cross on red, all bets are off”?

August 8th, 2008

Hey, you deleted my files!

We got a rather strongly worded message the other day from a Webmaster who was threatening legal action because our crawler deleted a bunch of files from his site.  The news that our crawler is capable of deleting files was quite a surprise to us.  Like other crawlers, ours just downloads HTML files, extracts links, and then visits those links.  There is no “delete a file” logic in there.  But if the crawler stumbles upon a link whose action is to delete a file, then visiting that link will indeed delete the file.

Further investigation in this particular case revealed a file management page that includes, among other things, links that have the form:  www.example.com/files/?delete=filename.txt.  Surprisingly enough, clicking on that link deletes the file.  The file management page is not protected by a password, nor is there any kind of confirmation displayed before the file is permanently deleted.

Examining the logs, we saw accesses from other search engine crawlers.  We also learned from the Webmaster that some time back, a kid had “hacked in” to the site and deleted a bunch of files.

I’m a little surprised that anybody would create such a page and not provide any protection.  I’m very surprised to find out that a supposedly professional Web developer would do such a thing and not learn the lesson when a random surfer came in and deleted files.  And I’m shocked that, even after we explained this to the Webmaster, he insists that we can take this as an opportunity to learn from our “mistake” and “fix” the crawler so that it doesn’t happen again.

It’s unfortunate that our crawler visited those links, causing the files to be deleted.  But the mistake was on the part of the person who posted those destructive links.  The crawler was operating exactly as it should.  Exactly, in fact, as every major search engine crawler acts.  It’d be nice if we could imbue the crawler with enough intelligence to “understand” Web pages and know in advance what the effects of clicking a link will be.  But that kind of machine intelligence is far, far in the future.

If you post something on the Web, it will be found, unless you take active measures to protect it.  Posting a destructive link on an unprotected page and then blaming somebody else when the link is clicked by an “unauthorized” person is akin to running out into a busy street and then blaming your injuries on the driver of the bus that hits you.

August 4th, 2008

Multicore Crisis?

There’s been some talk recently of the next “programming crisis”: multicore computing. I’ll agree that we should be concerned, but I don’t think we’re anywhere near the crisis point. Before I address that specifically, I think it’s instructive to review the background: why multicore processors exist, how they affect existing software, and the issues involved in writing code to make use of multiple cores.

Moore’s law has been quoted and misquoted so often that it’s almost a cliché. His original statement was simply an observation on the rate at which transistor counts were increasing on integrated circuits, and that he expected the trend to continue for at least 10 years. That was 1965. The trend has continued, and there’s no indication that it will slow.

Some people think Moore’s Law has become something of a self-fulfilling prophecy: because we believe that it’s possible, somehow we strive to make it so. One wonders what would have happened if Moore had said that he expected the rate of growth to increase. Would transistor densities have increased at an exponential rate?

Self-fulfilling prophecy or not, it’s almost certain that the trend in increasing transistor densities will continue (it has through 2007) and that as a result we’ll get ever more powerful CPUs as well as faster, higher-capacity RAM. Absolute processor speed as measured by clock rate will continue to increase, but not at the astounding rates that we saw up to 2005 or so. Quantum effects and current leakage have put a little damper on the rate of growth there. Better materials will solve the problem–are solving the problem–but absent a fundamental breakthrough by the chemists working on the problem, clock speeds won’t be doubling every 18 months like they had been in the recent past. The Clock Speed Timeline graph makes this quite evident.

Today’s trend is towards multiple cores on a single processor, running at a somewhat slower clock rate. The machine I’m writing this on, for example, has a quad-core Intel Xeon processor running at 2 GHz. The clock speed is somewhat slower than you can get in a high-end Pentium, but the multiple cores provide more total computing power. Quad core processors today are quite common. Intel demonstrated an 80-core chip in February of 2007, and promised to deliver it within five years. I fully expect to have a 256-core processor in my desktop computer ten years from now.

The trend towards multiple cores and very slowly increasing clock rates has some interesting ramifications for software developers. In the past, we have depended on more RAM and faster processors to give us some very nice performance boosts. All indications are that the amount of available RAM and the size of on-chip caches will continue to grow, but we can’t count on the biannual doubling of processor speed. Unless we learn to write programs that use multiple cores, we will soon reach a very real performance ceiling.

Not all applications can benefit from multiple cores, but you’d be surprised at how many can. And even in those cases when a single program can’t make use of multiple cores, users still benefit from having a multicore processor because the machine is better at multi-tasking. Imagine running four virtual machines on one computer, for example. If the computer has a single processor core, all four virtual machines and all of the operating system services share that one core. On a quad-core processor, the work load is spread out over all four cores. The result is more processor cycles per virtual machine, meaning that all four virtual machines should run faster.

Software systems that consist of multiple mostly-independent processes can make good use of multicore processors without any modification. Consider a system consisting of two services that are constantly running. On a single-core computer, only one can actually be working at a time. You could almost double performance simply by upgrading to a dual-core processor. Such software systems are quite common, and they require no code changes in order to benefit immediately from the new multicore processor designs.

Contrary to popular belief, writing code that is explicitly multi-threaded–designed to take advantage of multiple cores–isn’t necessarily a huge step up in complexity. Such code can be much more complex than single-threaded code, but it doesn’t have to be. Some programs are more multi-threaded than others. I’ve found it useful to think of programs in terms of the following four levels of complexity:

  1. No explicit multi-threading.
  2. Infrequent, mostly independent asynchronous tasks.
  3. Loosely coupled cooperating tasks.
  4. Tightly coupled cooperating tasks.

Obviously, it’s impossible to draw exact boundaries between the levels, and many programs will use features found in two or more of the levels. In general, I would classify a program by the highest level of multi-threading features that it uses.

Level 1 requires little in the way of explanation. This is the most common type of application in use today. In a batch mode program, execution proceeds sequentially from start to finish. In a GUI program, user interface events and processing execute on the same thread. This type of application has served us well over the years.

Most Windows programmers have some experience with the next level of complexity. A GUI application that performs background processing and periodically updates the display is an example of this type of program. Typically, the program starts the background process, which from time to time raises events which the GUI thread handles and updates the display. Data and process synchronization between tasks is limited to the event handlers that respond to asynchronous events. Modern development environments make it very easy to create such programs. These programs can benefit from multiple processor cores because the background thread can operate independently of the the GUI thread, making the GUI thread much more responsive.

I have found the third level of complexity–loosely coupled cooperating tasks–to be a very useful and relatively simple way to make use of multiple cores. The idea is to construct a program that operates in an assembly line fashion. For example, consider a program that gathers input, does some complex processing of the input data, and then generates some output. Many such programs are processor bound. If you structure the program such that it maintains an input queue, a pool of independent worker threads, and an output queue, then there is little danger of running into the problems that often plague more complex programs. You have to supply synchronization (mutual exclusion locks, or similar) on the input and output queues, but the worker threads operate independently. Using this technique on a quad-core processor, it’s possible to get an almost 4x increase in throughput over a single-core processor, with very little danger of running into resource contention issues.

Written correctly, programs that have multiple tightly-coupled cooperating tasks make the best possible use of processor resources. However, explictly coding thread synchronization is perhaps the most difficult type of programming imaginable. Forgetting to lock a resource before accessing it can lead to unexplained crashes or data corruption. Holding a lock for too long can create a performance bottleneck. Locks that are too granular increase complexity and also the chance for deadlock situations. Locks that are not granular enough will stall worker threads. Race conditions are endemic. Assuming you get such a program working, even a small change will often cause new, unanticipated problems. Writing this kind of code is hard. You’re much better off re-thinking your approach to the problem and casting it as a Level 3 problem. Whatever price you pay in performance will be returned many fold in increased reliability and reduced development time.

If you’re writing a Level 3 or Level 4 program, you should very seriously consider using a existing multi-tasking library if at all possible. Doing so will require that you think about your problem differently, but you leverage a lot of known-working code that is almost certainly more robust in all ways than what you’re likely to write yourself in the time allotted. Two good examples of such libraries are the Parallel Extensions to .NET 3.5 and the Java Parallel Processing Framework. Such libraries exist for many other programming environments. Although still in their infancy, these libraries promise to greatly simplify the move to multicore. If you’re contemplating development of a program that makes good use of multiple cores, you definitely should learn about any parallel computing libraries that support your platform.

So, back to the crisis. Bob Warfield over at SmoothSpan Blog has had and continues to have quite a lot to say about it, and many others share his sentiments. I, on the other hand, don’t think we’re anywhere near the crisis point. Nor do I think we’re likely to get there. Whereas it’s true that most current software isn’t multicore ready, software developers have understood for several years now that they need to begin writing applications that take advantage of multiple processor cores. It’s likely that some shops have taken an ad hoc approach to the problem, and they’re probably suffering with the issues I pointed out above. It’s also likely that many (and I would hope, most) development shops have done the prudent thing and adopted a parallel computing library that takes care of the difficult areas, leaving the programmers to worry about their specific applications. Doing so is no different than adopting an operating system, development environment, GUI library, report generator, or any other third party component–something that development shops have long experience with.

In short, the multicore “crisis” that the doomsayers are warning us about is almost a non-issue. It’s going to require a small amount of programmer retraining and there will undoubtedly be a temporary plateau in the rate at which our processing of data increases, but in a very short time we’ll again have mainstream applications that push all this fancy hardware to its limits.

July 29th, 2008

The Ultimate Development Machine?

In Understanding the Hardware, Jeff Atwood describes his “best bang for the buck developer x86 box,” at a cost of about $1,100.  The system he describes is quite a nice development machine, although it’s probably overkill for a lot of developers.  Seriously.  How many developers do you know who really need a 10,000 RPM drive and a screaming video card?

Surprisingly, he doesn’t mention what case he’s going to put all that fancy hardware in.  I’d really like to know.  I’ve mentioned before that I like the Antec Sonata cases because they’re very quiet.  But with their fans, they almost certainly create more noise than whatever Jeff’s using for passive cooling.

My development machine these days is quite a bit different from what he describes, but I realize that I have somewhat different needs.  I’ll give you a quick rundown.

Start with a Dell Precision 490 case, with power supply and motherboard.  These can be had for under $200 on eBay, or from Dell surplus suppliers.  They’re starting to become a bit scarce on the surplus market now, because most have gone off lease and Dell doesn’t make that model anymore.  One drawback to this system is that it creates a bit more noise than the Antec case, but I’ve found that I can accept a certain amount of noise.  And it’s hard to beat the price.

Add a quad-core Xeon E5335 processor running at 2 GHz.  Granted, 2 GHz isn’t exactly blindingly fast, but it’s quite well suited to the work that I do.  Unlike most developers, the code I’m working on does benefit from multiple cores.  The motherboard in this 490 has two processor slots, so I could potentially run two of those quad-core Xeons.  And I can make good use of all eight cores.  The Xeon is pretty pricey if you buy it new.  You might consider picking one up on eBay.  We’ve purchased dozens of these processors on eBay and haven’t had a problem with any of them.

I would have been shocked a year ago if somebody told me that I’d have a need for more than 8 gigabytes of RAM.  But the stuff I’m doing is memory hungry in the extreme.  This is another reason we go for the Dell 490 motherboard:  it was one of very few that supported 16 gigabytes a year ago, and I use every bit of it.  At about $80 for four gigabytes, memory is still a bit expensive.  But the stuff we’re working on really does need all the memory it can get.

I also use a lot of disk space.  Hard disk speed is important, but capacity is way more important to me.  I’ve loaded the box with two 7,200 RPM 750-gigabyte drives.  Terabyte drives are available, but at a huge premium.  The 750 GB drives go for about $120, or 6.25 cents per gigabyte.  A terabyte drive will run about $220, or 22 cents per gigabyte.  If I need more storage, I’ll find a way to shoehorn a third drive into this Dell box.

I’m not writing computer games, and I’ve turned off all the fancy Windows Aero features that do nothing but annoy me and chew up system resources.  My video card is a low-end ATI Sapphire 1650 for which we paid less than $50.  It drives my 24″ LCD at 1920 x 1200 resolution just fine.  I have no need for really high end video performance.

When you add everything up and throw in the DVD burner, we can put together one of these machines for under $1,500, which isn’t very much more than Jeff’s system once he adds the case and DVD.

I realize that I’m somewhat out of the ordinary, working with programs that require multiple cores as well as enormous amounts of memory and disk space.  I suspect that my ultimate development machine would be complete overkill for most developers.  But I find it interesting to compare what other developers need against what I’m using.

Do you have an ultimate developer machine?  Drop me a note.

An aside:
Jeff also uses the word commodification, as in, “This industry was built on the commodification of hardware. If you can snap together a Lego kit, you can build a computer.”  I had to read that twice before I realized that he wasn’t talking about turning hardware into toilets.  Commodification?  Please stop.

July 24th, 2008

Is that code really from Sun?

I updated my Java runtime the other day, and now every time I open a new tab in Internet Explorer, I get this message box:

It looks like somebody at Sun forgot to sign their update agent.  At least, I think this control came from Sun.  But there’s no way to be sure, is there?  Do I blindly assume that this really is from Sun and that they made a mistake in generating the build, or do I do the prudent thing and permanently disallow it?