Jim’s Random Notes

About a month ago, Debra and I started noticing that hot water pressure was lower than normal.  At first I thought it was my imagination, but it steadily got worse.  My first hypothesis was sediment in the tank, which fit with what others online will say.  So I hooked up a hose, drained the tank, flushed it a bit, and then refilled it.  Still low pressure.

My next thought was to verify that the problem was with the water heater and not somewhere else in the pipes.  So I connected the cold water input directly to the hot water output, removing the water heater completely from the system.  The resulting high pressure from the hot water side confirmed that the problem was indeed with the water heater.

It took a little looking around, but I finally found the problem:  the 3″ brass nipple that connects the water heater with the hot water pipes in the house was clogged with sediment.  It was so clogged that I’m surprised any water was coming out.  I’m a little embarrassed that it took me so long to check that out.  But since I had so much time in the project and it looked like I wouldn’t have to replace the water heater, I decided to refurbish it a bit.

Electric heating elements for my water heater are about $10 each.  Those and two replacement nipples, plus gaskets for the input and output, and the special tool for removing the elements set me back a total of about $25.  It was a pretty big time investment, though.  Draining the water heater takes a long time.  A few things to keep in mind (some of which I learned from experience):

• Before you drain your electric water heater or do any other work on on it, turn off the circuit breaker.
• If you don’t open a hot water faucet somewhere in the house, or open the T&P valve on the water heater, it’ll take almost forever to drain.  (I knew this one before, but thought I’d throw it in because a friend of mine ran into this problem.)
• Do not try to remove the elements with pliers or a pipe wrench.  Pay the $8 for the element removal tool.
• If the screwdriver you’re using as a handle for the element removal tool starts to bend, stop.  It’s likely you’ll break the screwdriver before the element comes out.  Find a longer and stiffer piece of metal to use as a handle.  Or dispense with the handle and wrap a pipe wrench around the element removal tool.  Works wonders.  (I was smart enough to stop when I saw the screwdriver shaft flexing.)
• Be absolutely sure the water is below the level of the element before you try to remove it.  You will not believe how fast water can come pouring out of that hole, and you will not be able to screw the element back in with the water pouring out.  And the water is hot. (Yes, I’m guilty of this one.)
• You can clean corrosion from heating elements by soaking them in vinegar.  If you decide to re-use your elements after cleaning and inspecting them, be sure to replace the rubber gasket.  Otherwise they will almost surely leak and you’ll have to drain the water heater again to remove and replace them.  The way I figure it, if I’m going to the trouble of draining the water heater, I’ll just replace the elements.

One other thing.  Electric water heaters contain a sacrificial anode rod that helps prevent corrosion of the tank.  The idea is that the anode, being a more active metal than what the tank is made of, will corrode first.  As long as there’s a more active metal than the tank’s metal, the tank won’t corrode (or will do so much more slowly).  Water heater warrantees are typically based on how long the manufacturer thinks the anode rod will last.  You can replace the anode rod.  I haven’t tried it yet.

Most manufacturers recommend that you drain a few quarts from your water heater every three months (some say every month).  That will prevent sediment buildup in your tank.  They recommend draining the tank and inspecting the elements annually.  They also recommend an annual inspection of the anode rod.

By the way, elements that are covered in corrosion don’t work very well at all.  They require a lot more electricity to generate the same amount of heat as new elements.  Especially if you have very hard water, you’re probably money ahead if you replace the elements annually.  The money you save in electricity will more than offset the cost of the new elements, and your water will heat much faster.

Most people (myself included, usually) never think about their water heater until they have no hot water or they notice a leak.  That’s too bad, because with a little periodic maintenance a water heater can last 15 or 20 years rather than the five or so years that they typically last these days.  Considering the cost of a replacement water heater and installation (sometimes over $1,000) and the aggravation of a leaking heater or no hot water, you’re much better off with the periodic maintenance.

If you’re having a problem with your water heater, a good place to look for a solution is Waterheaterrescue.com.  Whereas it’s true that they’re trying to sell you things, they have very good information about common problems and simple solutions.  Oh, and in case you’re interested in how this stuff works: How Water Heaters Work.

My second programming job was with a small commercial bank in Fresno, CA, where I helped maintain the COBOL account processing software.  I was still pretty inexperienced, having only been working in the industry for about 18 months.  My previous job involved maintaining software, also in COBOL, for small banks in western Colorado.

One of the first things my new boss asked me to look at was a program that computed loan totals by category:  a two-digit code that was assigned to each loan.  Federal regulations required that we report the total number and dollar amount of all loans, by category, as well as the number and dollar amount that were 30, 60, and 90 days or more past due.  The problem was that the report program was taking way too long to run.  The bank had recently acquired a bunch of new loans, and the program’s run time had increased sharply—several times more than what one would expect from the increase in the number of loans.

Understand, this was a very simple program.  All it had to do was go through the loans sequentially and compute totals in four columns (total, 30 days past, 60 days past, 90+ days past) for each of the 100 categories.  The data structures are very simple.  I don’t remember enough COBOL to write it intelligently, so I’m showing them translated to C#:

struct CountAndTotal
{
    public int Category;
    public int Count;
    public double Total;
}

// Arrays for Total, 30, 60, and 90+ days past due
CountAndTotal[] TotalAllLoans = new CountAndTotal[100];
CountAndTotal[] Total30Past = new CountAndTotal[100];
CountAndTotal[] Total60Past = new CountAndTotal[100];
CountAndTotal[] Total90Past = new CountAndTotal[100];

I’ll admit that I was a little mystified by the Category field in the CountAndTotal structure, but figured it was an artifact from early debugging.

Those definitions and the description of the problem above lead to a simple loop:  for each loan, determine its category and payment status, and add the totals to the proper items in the arrays. The program almost writes itself:

while (!LoanFile.Eof)
{
    LoanRec loan = LoanFile.ReadNext();
    AddToTotals(loan, TotalAllLoans);
    if (loan.PastDue(90))
        AddToTotals(loan, Total90Past);
    else if (loan.PastDue(60))
        AddToTotals(loan, Total60Past);
    else if (loan.PastDue(30))
        AddToTotals(loan, Total30Past);
}

What surprised me when I looked at the code was the implementation of the AddToTotals function. You would expect it to be a simple index into the array from the loan’s category code. After all, the category was guaranteed to be in the range 0-99. That just begs for this implementation:

void AddToTotals(LoanRec loan, CountAndTotal[] Totals)
{
    ++CountAndTotal[loan.Category].Count;
    CountAndTotal[loan.Category].Total += loan.Balance;
}

What I found was quite surprising. Rather than directly index into the array of categories, the program would do a sequential search of the array to see if that category was already there. If it was, the total was added. Otherwise the program made a new entry at the next empty spot in the array. That explained the mysterious Category field and the absurdly long run time. The code is much more complicated:

void AddtoTotals(LoanRec loan, CountAndTotal[] Totals)
{
    int i = 0;
    while (i < 100)
    {
        if (Totals[i].Category == loan.Category)
        {
            ++Totals[i].Count;
            Totals[i].Total += loan.Balance;
            break;
        }
        else if (Totals[i].Category == -1)
        {
            // Unused position.  The category wasn't found in the array.
            Totals[i].Category = loan.Category.
            Totals[i].Count = 1;
            Totals[i].Total = loan.Balance;
            break;
        }
        ++i;
    }
}

The difference in run times is enormous! The first implementation accesses the array directly from the loan.Category field. The second has to search the array sequentially—an operation that involves, on average, looking at 50 different items every time.  The second version of the program is 50 times slower than the first.  In addition, the second required a subsequent sort to put things in the proper order before printing the results.

Being new at the job, I went to my boss, explained what I’d found, and said, “What am I missing?”  His response:  “Why do you think you’re missing something?”

He went on to explain that my analysis was correct, and that the industry (at least back then) was full of programmers who had no business sitting at a terminal.  It was something of a revelation to me, because I had assumed that the people who wrote this stuff really knew what they were doing.  It also taught me to question everything when faced with a problem.  It’s always a good idea to assume competence when you start debugging somebody else’s (or your own) code, but when things stop making sense, it’s time to re-evaluate that assumption.

You’re looking at the front end of my riding lawn mower. After 12 seasons of use and my own maintenance, I finally had some problems with it that were better handled by a repair shop. I got the machine back on Wednesday after they replaced the starter and ring gear, did a tune-up, adjusted the front end alignment, and fixed the mower clutch. My plan Saturday morning was to mow the front yard before heading to the office. I got about halfway through when the engine died.

An engine requires four things to operate: compression, fuel, air, and spark. It was pretty clear that I had compression, as the engine had been operating and I would have known if it had eaten a valve or thrown a rod. I took off the air cleaner and cranked the engine to verify that fuel was getting into the engine, and since the air cleaner was clean I figured that wasn’t a problem. That leaves spark.

The most common reason for no spark is a bad spark plug. I removed the plug, connected it, and cranked the engine. Sure enough, no spark. After a quick run to the shop for a new spark plug, put it in and … still no spark.

It didn’t take much debugging to find the problem.

In the picture above, you can see a black wire leading from the cowling up front, under the starter, and to a wiring harness that’s just below the gas tank. But when I first looked at it, that wire was running behind the starter. The mechanic at the repair shop must have got the wire caught there when replacing the starter. The result, as you can see in the picture below, was quite impressive:

Being wedged between the starter and the block flattened the wire, and the heat of engine operation finally melted enough of the insulation that the wire grounded against the engine block, and the engine died.

While I was working on this, I got to thinking how much what I was doing resembled what I do every day when debugging code: determine the possible causes of the failure and then check them off, one by one. In a very large number of cases, debugging really is that simple if approached logically. Granted, intermittent problems in multi-threaded programs can be much, much more difficult to find, but even they finally succumb to the same basic method: form an hypothesis to explain the behavior, formulate a test case to verify the hypothesis, and then correct the code that is in error.

I’ve found that most problems I run into in life are best handled in this manner. It’s certainly more productive than flailing around without a clue.