By Jim, on August 30th, 2011 I’m in the process of (re)organizing my garage to remove clutter and turn it into a woodworking shop of sorts. Not that it’ll be what you call a well-stocked shop, even for a hobbyist. My power tool collection includes two bandsaws (see below), a compound miter saw, some drills, a circular saw, jigsaw, and belt sander. Maybe a few other hand tools. Still, one can build a surprising number of things with those few tools.
But the garage currently has a few major problems. Most importantly it’s cluttered with 15+ years of stuff, much of which I have no use for. That has to go, and I need to organize what’s left. The workbench, too, needs to be replaced. Ever since we’ve been in this place, the workbench’s primary use has been as a place to hold stuff that I don’t have a place for. That is, there’s stuff piled on it. And it’s not really designed for wood working. It’s too tall, a little too long for the space it’s in, and too deep.
I’ve been looking at shop layouts to get ideas of how I want to organize my garage work area. I have a few restrictions, though, that make things a bit more difficult.
First, the garage is primarily for storing cars. With the Mustang gone, we now have two cars and two garage bays, and I rather like the idea of parking my truck in the garage. Whatever I do, I have to ensure that I can still pull my truck into the garage.
There is an 8 foot wide area beside one of the bays (where the workbench is), so it’s possible to have both cars in the garage and still have space for other things. My idea is to organize the space so that I can store everything in that 8′ space. When I need to work on a project, I can pull the truck out and spread out the tools. So pretty much everything has to be on wheels.
My first step in that direction will be adding a mobile base to my bandsaw. Perhaps to two different bandsaws, since I recently acquired a second one. My original machine, purchased used two years ago, is a Craftsman 12″ bandsaw, built in 1987 or so. It has a 3/4 horsepower motor and with a 1/2 inch blade has no trouble cutting through a 6″ mesquite log. My only real complaints with the machine are that it’s difficult to move, and changing blades is a bit of a pain in the neck. If I want to cut up a log and then cut out some patterns I have to install the 1/2 inch blade, rough-saw the log, and then install the smaller blade (1/4 or 3/16 inch) for the detail cuts.
A friend of mine recently cleaned out his garage and gave me his old bandsaw that he hasn’t used in years. It, too, is a Craftsman 12″ bandsaw, although it’s 10 years older and only has a 1/2 inch motor. But fitted with a 3/16 inch blade it should be perfect for cutting stuff up to about 4 inches thick. Having two bandsaws should prevent me from having to change blades except when they need to be replaced.
Although my primary concern is wood carving, I’m also interested in trying my hand at building some simple furniture, some storage shelves, and other things. Before I get additional tools, I need an appropriate workbench that I can use for projects. I also need additional storage space. So I’m looking at plans for a rolling workbench with built-in storage. I don’t know yet whether that storage will be shelves or drawers. But once I build that, I can free up some of the space that’s currently being occupied by my big workbench.
Wood storage is a problem. I put a couple of 4′ x 8′ pieces of cheap plywood above the rafters, giving me a lot of space in which to store lumber, and perhaps there’s enough space up there to store my carving wood, too, if I organize things better.
The next task will be storage. Currently, we have a mish-mash of shelving units scattered throughout the garage. It works, but it’s less than ideal. I can make better use of the wall space if I build custom shelves.
On the strictly carving front, I’ve had a bad run of mistakes lately. Over the weekend I cut out blanks for three shallow spoons and two of those stirrers with holes in them. I broke one of the stirrers (cut it too thin and it snapped while I was working with it) and one of the spoons. I carved the bowl on one of the other spoons too deeply and went through the bottom. The third spoon will likely survive, although I carved it a bit too deeply, too. It’s very thin on the bottom. I found a quicker way to hollow out the spoon bowls, but in doing so got a bit too aggressive.
It’s nice having the bandsaw working again, though. I cut out a couple of animal caricatures over the weekend that should turn out to be fun to carve. We’ll see.
I’m hoping to get the garage organized in the next four or five weeks so that I can enjoy working (playing) in there when it cools off. In the summer (especially this summer), it gets unbearably hot in there before noon. But the weather in October and November and even the first part of December should be great for working in the garage.
Eventually I’ll have a dedicated workshop that has air conditioning and heating. Even then, space will be somewhat limited. So the work I’m doing now to put the tools, workbench, and other things on wheels will be quite useful in the future.
By Jim, on August 22nd, 2011 I went looking for plans to build a simple rolling base for my bandsaw, as I’m getting tired of dragging it across the floor. I ran across this site, which is just great. He has pictures of a simple bandsaw base, and I think even an amateur wood hacker like me can figure out how to put it together.
But that’s not the coolest part about this site. This guy has built a working bandsaw completely from wood (except for the motor and blade). There are a few other machines, too. For example, a home-built thickness sander that I think I need to build.
I just love the simplicity of this guy’s designs. His work bench and table are easy enough that I can build them with the few tools I have in my shop, and should be quite effective. Granted, it’s not home decor furniture, but that’s just fine.
I like this guy’s approach. He strives for simplicity, and as a result his designs are quite approachable for the amateur. Highly recommended.
By Jim, on August 19th, 2011 The Janka hardness test measures the resistance of a type of wood to withstand denting and wear. It measures the force required to embed a 11.28 mm (0.444 inches) diameter steel ball into wood to half the ball’s diameter. Why 11.28 mm? Because a circle with a diameter of 11.28 mm has an area of 100 square millimeters.
Of course, wood hardness depends on if you’re talking about side hardness (imagine pressing against the side of a tree) or end hardness (imagine pressing into the top of a stump that you cut off). It also depends on the wood’s moisture content and a few other factors. The numbers you’ll see most often are for side hardness at 12% moisture content.
The important thing to know about the Janka scale is not so much the absolute number (which can be given in pounds-force, kilograms-force, newtons, or kilonewtons), but rather the relative difference between two species. For example, mesquite with a hardness rating of 2,345 lbf is quite a bit harder than purpleheart (1,860 lbf).
When I first started carving, I was told that the Janka rating gives a good idea of the difficulty of cutting a particular type of wood. Seeing that mesquite is harder than purpleheart, I thought that I could carve some great things from purpleheart. How wrong I was! Whereas mesquite cuts very nicely, purpleheart is very difficult to cut. It’s so hard that it damaged my knife blade.
Most oaks have a Janka hardness rating around 1,350 but, as with purpleheart, I find them more difficult to cut than mesquite. Obviously, the Janka scale doesn’t tell the whole story.
I’ve done quite a bit of searching and haven’t found anything similar to the Janka scale that gives the relative cutting resistance of different woods. Many woodworking sites will give informal measurements. For example, mesquite is said to have “low cutting resistance” and purpleheart has “moderate cutting resistance.” In those cases, it’s the resistance to cutting with a saw rather than with a knife, but it does give me a little more evidence to back up my statement that the Janka scale is not the thing to use in determining cutting resistance.
In his 1950 Ph.D. thesis, Cutting force in wood working (just the abstract is free; I haven’t been able to find a free copy of the thesis), Eero Kivimaa did a lot of research to determine the forces involved in cutting wood. He did most of his research using air-dried Finnish birch, but some tests with 21 different wood species led him to write in his Summary:
The cutting force was found to vary approximately linearly with the specific gravity of wood (air-dry Finnish Birch). The cutting force rose to a maximum when the moisture content increased up to 10-13 %, and fell again with higher moisture content. Comparison of results obtained with 21 species showed that it is possible to estimate the main cutting force of any wood species on the basis of its sp. gr.
Aha! If you look up the specific gravity of mesquite, you’ll find that it’s about 0.80. The specific gravity of purpleheart is 0.86. From Kivimaa’s research, then, you would expect purpleheart harder to cut. Southern live oak, which is pretty common around here, has a specific gravity of 0.88. Again, these woods are “softer” than mesquite on the Janka scale, but have higher specific gravity and are more difficult to carve. Hickory, too, has a lower Janka rating (1,820), but a specific gravity of 0.83.
Note that, as with Janka hardness, the force required to cut a particular type of wood will depend a lot on the moisture content. As Kivimaa pointed out, force required increased as moisture content increased to between 10 and 13 percent, and then decreased as moisture content increased. “Green” wood will cut more easily than kiln dried wood.
Cutting force required will also depend on the sharpness of the knife, the bevel angle, the angle of attack, whether you’re carving with the grain, across the grain, or on the end grain, the smoothness of the blade as is moves through the wood, and the type of cut (pressing or slicing). We can hold those values constant in order to get a relative cutting resistance measure.
I’m not convinced that specific gravity is the sole deciding factor, but it’s much better than Janka hardness in determining how hard a wood will be to cut.
A good place to find information about a particular type of wood, including its hardness and specific gravity, is The Wood Database.
Speaking of specific gravity, common wisdom is that wood floats. Whereas it’s true that most woods float, there are some species that sink. Lignum vitae, Arizona desert ironwood, some types of Ebony, and many other species have specific gravity greater than 1.0. Even kiln dried, these woods are more dense than water, meaning that they will not float in water. Black ironwood has a specific gravity of 1.49; it’s 50% more dense than water.
By Jim, on August 18th, 2011 I cut this pattern from a piece left over when I cut out the mesquite spoon last month. I put the blank in my carving box and forgot about it until I was going through the box over the weekend.
Nothing special about the design: just a foot-long wooden stirrer with a hole in it. Rough shaped with the coping saw, hand carved and sanded. Finish is Howard Butcher Block Conditioner.
I really do like the way that mesquite finishes. Click on the image for a full-size view.

By Jim, on August 16th, 2011 I finally finished my Whittle Pup tutorial. If you’re interested in how I carve my little dogs or if you want to give it a try yourself, check it out.
Carving the Whittle Pup
By Jim, on August 16th, 2011 The previous pair of salad claws were a bit too short, resulting in messy fingers. These are five inches long and almost three inches wide.
I cut these out on the band saw and did most of the shaping with a sanding drum on an old rotary tool. Then a lot of hand sanding (successive grits up to 600), and finished with Howard Butcher Block Conditioner.
That black cherry is some beautiful stuff. I think I need to make a spoon or two from what I have left.
Pattern info
Several people asked for the pattern I used for these. I didn’t have a pattern, but rather just drew the outline on the wood. So I traced the outline and made some annotations. That rough pattern should give you the basic idea. Here’s a text description.
The basic shape is 2 7/8 inches wide and 5 1/4 inches long. I would suggest going 3 inches wide, and between five and six inches long. The fingers should be about 2.5 inches. The wood is right at 1/4 inch thick after you finish cutting it out.
I started with a 3″ x 3″ piece of black cherry that I cut to 5 1/2″ long. I then drew the side view and cut the two pieces on the band saw. I then drew the fingers and, with the curve down (so both ends rested on the table), cut the spaces between the fingers. The fingers should be on 0.6 inch centers, and should be between 3/8 and 1/2 inch wide. If the fingers are 1/2 inch wide, the space between the fingers is 1/8 inch, which isn’t quite enough in my opinion.
After cutting out on the band saw, do simple shaping with a sharp knife and sandpaper, or with a rotary tool. Then sand to the smoothness you like (I use successive grits up to 600), and apply your favorite finish.
By Jim, on August 8th, 2011 When working with a power carver (think, Dremel), you have to be very careful about protecting yourself from the effects of wood dust. Dust collection is a big deal to power carvers. And even when hand-sanding a small carving, one has to be careful. The effects of inhaling wood dust are uncomfortable at best and at worst, deadly.
This is especially true when carving “found wood.” If you’re working with basswood, the effects of inhaling a few dust particles will be a slight cough and maybe a sore throat for a few days. You probably don’t want to inhale a lot of it, though. But unless you inhaled enough basswood dust to physically block your lungs, you wouldn’t die from it.
Not all woods are as benign as basswood.
Large pieces (3 inches or more thick) of poison ivy, for example, have beautiful grain. But you don’t see people carving the stuff because most people are highly allergic to it. Other woods that cause allergic reactions on contact include (among many others) poison oak, poison sumac, walnut, and fig. Most people are allergic to the poison oak, ivy, and sumac. Allergies to walnut and fig are not uncommon, but usually not as severe. In any case, serious health problems due to contact allergies are rare.
That’s not so with dust. The dust from many different types of wood can cause severe respiratory problems. Pages such as this Toxic Wood Chart list dozens of species whose dust can be the cause of everything from mild skin irritation to severe respiratory distress. Wood dust can be deadly toxic.
A common ornamental tree here in Texas is called “mimosa.” It’s a medium-sized tree, fast growing, with beautiful flowers that have a very pleasant odor. The trees typically live 20 to 30 years. There’s been a lot of development in my area in the last 30 years, and I’m seeing more dead mimosa trees. The heartwood is beautiful. I want to carve the stuff, but somebody told me, “mimosa is deadly toxic.” I don’t take much on faith these days, so I thought I’d do some research to see if it really is toxic.
There are many dozens of plants that are commonly called “mimosa.” So far as I can tell, the only thing those plants have in common is that they are members of the family Fabaceae, or the legume family. The sub-family Mimosoideae contains many of the plants called “mimosa.” But the tree we commonly call mimosa around here is probably Albizia julibrissin, which is not very closely related to Mimoseae. To further confuse matters, several species of Acacia are also referred to as “mimosa.” Those, too, have some nice looking wood that I’m sure people carve.
I can undoubtedly discover the exact type of tree we’re calling “mimosa” around here, but I don’t know how to determine whether or not it’s toxic. I suppose I could try working it without wearing any kind of mask or filter, but that seems like a high-risk endeavor. I’d rather somebody could tell me one way or the other.
Mind you, I typically wear a simple mask when sanding or working with the power carver, but if this stuff is as toxic as the charts say then I’ll want to wear something a bit more protective and that includes a full face mask. I don’t have any qualms about carving the stuff, but if I’m going to raise dust then I need to be a lot more careful.
There’s a disconnect between botanists and wood carvers. Botanical sites tell me lots of things about the trees, including what parts of the tree are toxic when ingested. But botanical sites don’t tell me if the dust is toxic. And wood carving sites will tell me that “mimosa” is toxic, but they don’t tell me which type of mimosa they’re referring to!
This isn’t uncommon, by the way. There are over 600 different species referred to as “oak,” and “willow” covers around 400 species. “Cypress” encompasses dozens of different species, most in the same genus but not all. And don’t even get me started on ironwood, which can mean any one of a couple dozen species.
In any case, I’d sure like it if botanical sites included information on dust toxicity, and if wood carving sites were more specific in identifying their woods.
I should mention here that if you’re working with a wood that you can’t identify, or if you can identify the type of wood but don’t know where it came from, you should be cautious. Pallets, for example, often contain useful wood. The wood might be easily identified, but pallets are often treated with some rather nasty chemicals that, if inhaled, can cause some serious problems. Old doors or tables that were finished with some type of stain or varnish can also be dangerous. The same thing goes with fungi that cause spalting in maple and in other woods. Some of those fungi, if inhaled, will want to set up shop in the warm, dark, moist environment in your lungs. Fungus can be a problem even in kiln-dried wood such as the ambrosia maple that I love working with.
When in doubt, wear a protective mask.
By Jim, on August 4th, 2011 The Dow Jones Industrial Average lost 512 points today, down 4.31%. The S&P 500 is down 4.78%. NASDAQ down over 5%. That’s on top of a 2+ percent loss on Tuesday. I can’t blame it all on the debt ceiling deal, but that’s a major contributor. Let me explain why.
Investors hate uncertainty. When they don’t don’t know what Congress is going to do, investors get nervous and they tend to flee from stocks like rats deserting a sinking ship. During the run-up to Tuesday’s historically idiotic culmination of the most recent tempest in a teapot, investors believed that Congress would do something to resolve the issue. Instead, Congress passed and the President signed a bill that just kicks the can down the road a bit, which is what they’re best at. They’ve put the whole thing off until November. Investors, now with no idea of what train wreck Congress is going to perpetrate next, are pulling their money out of the market. They’re going to sit on it for a while (a week, a few months, a year or two, who knows) until they can figure out what the new rules are.
It’s fitting that the new legislation was called the Debt Control Act of 2011. Congress excels at passing legislation that does exactly the opposite of what one would expect, based on the bill’s title.
The stock market is not the national economy. Often it’s not even a good indicator of the national economy. But it acts like the national economy in many ways. In particular, the primary driver of the stock market is investor sentiment, much as the primary driver of the national economy is consumer (and, to some extent, producer) sentiment. If people think that things are getting better, they spend money much more freely. When people think things are getting worse or not going to change, they tend to sit on their money and only let go of it when absolutely necessary.
Recent polls show that most people believe that things aren’t getting better. That’s due in large part to their belief that government drives the economy–a belief reinforced by the mainstream media, either on purpose or as a side effect of superficial reporting. Both major political parties, and most of the smaller parties count on that belief and do everything in their power to reinforce it. It’s wrong, but it’s beneficial for some that most believe it to be true. And it’s convenient for the masses, because it relieves them from any responsibility.
When people armed with that belief see Congress wasting months on a spending bill that ultimately amounts to more of the same, they are not going to be filled with confidence. A poll taken yesterday shows that of the people polled, 41% believe that the new Debt Control Act will make the economy worse. 17% believe it will make it better. And almost one-third of respondents said that it won’t make a difference.
It looks to me like 83% of people are going to sit on their money until they see what Congress does next.
It’s kind of funny that back when things were going well, people were spending money they didn’t have on things they didn’t need, because they thought they could make it up in the future. Now, many of them are reluctant to buy things they really need with money they already have because they’re unsure of how long their money will last. This probably points to a basic flaw in the way that most people see the world.
As long as the American people believe that government controls the economy, we’re going to see longer periods of contraction or stagnation, and fewer and shorter periods of growth. My conclusion, based on 30+ years observing Congress in real time, and my reading of history from prior years, is that government can create short-term bubbles at the cost of long-term problems. Government policies that use borrowed money to target certain industries or certain groups of people usually result in short-term gains for those affected areas. But those policies almost invariably lead to unintended negative consequences, particularly when the temporary programs end and the industries that were built up to take advantage of those programs fail.
“Government money” in the economy, either through higher taxes or borrowed money, is like taking amphetamines. It’s go, go, go until the drugs wear off, followed by a crash. Our economy is now suffering from the equivalant of amphetamine dependence, where it takes larger and more frequent doses to get any kind of reaction.
As with amphetamine dependence, whereas decreasing or eliminating government’s role in the economy would be a Good Thing, doing so will involve some very painful withdrawal symptoms. Unfortunately, I doubt that we have the national will to suffer through that pain, even though the result would be a much more stable and robust economy.
By Jim, on August 1st, 2011 You’ve probably heard by now that President Obama and House Speaker Boehner have agreed on a plan that will allow an increase in the debt ceiling. We’re being told that the agreement is a great achievement. One news report, for example, says:
The White House and congressional leaders worked Monday to align lawmakers from both parties behind their formula for averting a financial meltdown and halting the government’s prolific spending habits.
President Obama said in a tweet: “The debt agreement makes a significant down payment to reduce the deficit– finding savings in both defense and domestic spending.”
Senator Reid’s comment sums up the agreement very well, at least in terms of how members of Congress view the plan: “People on the right are upset, people on the left are upset, people in the middle are upset. It was a compromise.”
The American people should be disgusted.
Understand that in the discussion below, the figures are projected decrease in deficit over 10 years.
The full agreement is H.R. 2693 — Budget Control Act of 2011, but you likely won’t be able to get much in the way of useful information from that. The relevant portions of the bill include absolute spending limits for different programs. There’s no comparison of previous figures. In order to estimate savings, you need to know what they’re comparing against.
The Congressional Budget Office did the comparisons against the March 2011 “baseline” figures and came up with a savings estimate. The overview of their analysis projects:
- $912 billion reduction based on explicit spending limits outlined in the bill.
- Up to $1.5 trillion reduction to be determined later by a bipartisan committee.
- $1.2 trillion “automatic” reduction if the committee doesn’t come to an agreement.
- Total of at least $2.1 trillion reduction, and potentially up to $2.4 trillion.
Before you celebrate, you should read the full report from the CBO.
If appropriations in the next 10 years are equal to the caps on discretionary spending and the maximum amount of funding is provided for the program integrity initiatives, CBO estimates that the legislation–apart from the provisions related to the joint select committee–would reduce budget deficits by $917 billion between 2012 and 2021. In addition, legislation originating with the joint select committee, or the automatic reductions in spending that would occur in the absence of such legislation, would reduce deficits by at least $1.2 trillion over the 10-year period. Therefore, the deficit reduction stemming from this legislation would total at least $2.1 trillion over the 2012-2021 period.
In other words, if nothing changes between now and then, we can expect the total amount that government overspends in the next 10 years to reduce from $15 trillion to $12.9 trillion.
This is what the President calls “a significant down payment to reduce the deficit?” I want some of what he’s smoking.
It’s interesting to note that the majority of the $912 billion is achieved by limiting discretionary spending. There’s some reduction in Pell grants and farm programs, and a few other things. However, discretionary spending (including the Department of Defense) makes up only about forty percent of the entire annual budget. If you eliminate all discretionary spending, we’d still run a deficit. And yet there is no discussion of reduction in non-discretionary programs. It’s possible we’ll see some of that from the “joint select committee,” but I wouldn’t hold my breath.
Whereas it’s very likely that this agreement will pass in the Senate, there’s still some question in the House. I suspect that it will pass, over some very vocal objections by members on all sides who have to look good for their constituents. And all sides will declare “victory.”
So Congress has spent two or three months wrangling over an agreement that reduces deficits by at best 15 percent over the next 10 years. They have not done a thing to address the cause of government overspending, nor have they provided anything like a realistic framework for doing so. But that won’t stop either side from claiming victory over the other.
Perhaps the strangest part of the whole thing is that President Obama is being hailed by some as “fiscally conservative.” It boggles the mind.
Oh, and the new debt ceiling? $16,994,000,000,000.
By Jim, on July 27th, 2011 We’ve been hearing for the last couple of months that Congress needs to raise the debt limit before August 2 in order to prevent the U.S. Government from defaulting on its obligations. This isn’t new; in the past, Congress has increased the debt limit as a matter of course. The only thing different this time is that there are members of Congress who have to show their constituents that they are “doing something about the problem” of excessive government spending.
Don’t get me wrong. I agree that our government squanders entirely too much money. I’ve noted many times in the past that I think our government’s spending is completely out of control and that we must rein it in if we want to maintain our standard of living and our status as a leader in the world economy. But refusing to increase the debt limit is not the way to do it.
Those who say we shouldn’t increase the debt limit are, at best, confused. In the first place, we’ve already committed to trillions of dollars in new spending–money that we don’t currently have. We will have to borrow in order to meet those commitments. We can’t just say to our creditors and others we’ve made those commitments to, “sorry, we’re out of money.”
Second, there’s no way that failing to increase the debt limit will prevent us from meeting those commitments. Our government will meet its obligations, at least for the immediately foreseeable future. It’s not like government will stop functioning on August 2. We will be “breaking the law” by continuing to spend beyond the currently authorized debt limit, but government will continue to function.
Those who think that the way out of this mess is to increase taxes haven’t been paying attention. Congress has shown that it’s unable to exercise any kind of restraint when it comes to spending. If government takes in more, it just spends more. We’ve seen this happen as recently as the Clinton years, when a combination of an unusually prosperous few years and a President and Congress who actually kept each other in check produced a projected budget surplus. That surplus wasn’t nearly as large as some would like you to think, and it disappeared pretty quickly when the economy turned down, helped of course by the next President and Congress who had a “you wash my back and I’ll wash yours” approach to spending.
Our current President and the previous Congress had the same type of relationship, and as much as the Republicans who control the House would like you to think otherwise, that hasn’t changed much since last year’s election. Neither really cares how much money is spent. They just want to make sure that when money is spent, they can spin it to make the other guy look bad. Or, if that fails, make themselves look good. That is, it’s best if one can say, “The other guy spent that money!” It’s almost as good to say, “Well, yes, I spent that money, but it was for this Good Thing.”
All the noise you hear coming out of Washington recently about the debt limit boils down to nothing but a bunch of hot air. Members of both parties will blame the other for reckless spending, obstructionism, confiscatory taxation, stealing food from the mouths of babies, “playing politics” with our grandchildren’s futures, balancing the budget on the backs of the poor (or the rich), and all manner of other heinous crimes.
The whole mess is a result of Republicans insisting on some type of deficit reduction as a condition of increasing the debt limit. Whereas I think they’re right in raising objections to our profligate spending, their choice of battleground is not particularly wise. They somehow think that the good press they’ll get from “making a stand” against excess spending will outweigh the bad press they’re getting from all corners. I suspect that they’re going to come out smelling like they rolled in a pig pen.
President Obama and the Democrats in Congress said they were willing to participate in a discussion. The original proposal was to cut four trillion dollars off the deficits over the next 10 years. The project deficits over that period amount to about $15 trillion. So even if the four trillion figure were attainable, we’d still amass $11 trillion in new debt. But Republicans won’t entertain any kind of increase in taxes, and Democrats at first weren’t willing to pass anything that didn’t include more revenue (i.e. higher taxes). Increased taxes are now off the table, and current proposals by both parties amount to about $2.5 trillion reduction over 10 years. Those proposals are so full of holes that the Congressional Budget Office estimates they’ll result in a savings that’s between 25% and 30% lower. Not that either proposal will fly. The House bill is D.O.A. in the Senate, and likewise the Senate bill in the House.
Both parties are playing a game of chicken, arguing over what amounts to at best a 12% reduction in deficits over the next 10 years, and holding up an increase in the debt limit that probably doesn’t matter at all and most likely will eventually be passed, regardless of how this particular little squabble turns out. In other words, all this noise is just an argument over an imaginary spending reduction that started over a routine bit of meaningless bookkeeping.
The real issue has nothing to do with spending, taxation, deficits, or debt limits. If you’ve been paying attention at all, you can see that the real issue is simply election politics. Next year is a big election year. Republicans are pushing for a small increase in the debt limit, hoping that we’ll have to go through this again next year, shortly before the election. Democrats want a larger increase in the debt limit so that they can skip having to discuss this until after next November’s election. Republicans are hoping to make President Obama look bad going into next year’s election, and Democrats are trying to prevent that.
Of course, each party is accusing the other of “playing politics.” And the mindless parrots who make up the majority of the American electorate go along with it happily, never stopping to consider that their selected tribe is doing the same thing that they’re excoriating the other tribe for doing. It never occurs to the denizens of either tribe to engage their brains and figure out that they’re being hoodwinked by their tribal leaders who don’t give a damn about the American people, our economy, or the long-term health of our country. Their only consideration is power: how to get it and how to keep it. Whatever helps the tribe amass or retain power is good. Everything else is irrelevant.
There’s no doubt that we need to rein in spending, but the people currently in charge of the purse strings aren’t going to do it. They’re not even interested in it, and neither are any of the people who are thinking about standing for election next fall. You might think that a “throw the bums out” reaction in next year’s election will “send a message,” and it will. The message received will be, “Suckers! You think you can get rid of us that easily?” The new crop of elected parasites will set up shop and continue to fleece you.
The only way out of this mess is to throw off the idiotic illusion that your tribe is better than the other tribe, and that if your tribe has control everything will be okay. Ditch party politics. Think for yourself. Refuse to vote for anybody who claims a party affiliation. Judge based on actions and on real answers to real questions.
You can continue in your slavish devotion to your tribe, and continue to blame the other tribe for things getting worse. And they will get much worse. Or you can engage your brain, see the parties for the manipulators that they are, and elect real people who actually care about doing what’s right rather than what will keep themselves and their tribes in power.
By Jim, on July 22nd, 2011 The Twitter convention for referencing a stock is to put a dollar sign in front of the ticker symbol. For example, a tweet that contains $MSFT is talking about the stock for Microsoft Corporation. The tweet likely contains a link to an article about the company’s performance, or perhaps somebody’s sentiment about the stock. I got to wondering which stocks get the most tweets.
Twitter has a very easy to use API, and fairly liberal usage restrictions. It was a matter of a few minutes to get a list of the stocks on the S&P 500 and write a simple program that does a Twitter search for each stock and computes a “tweets per hour” figure for each stock. From there, it’s a simple matter of sorting to come up with the most-tweeted stocks.
If you assume that Twitter is a reliable proxy for overall chatter about stocks, then you can say that the stocks below are the 20 most talked about stocks. I don’t know enough about the market to say with any certainty that the assumption holds true, but the list below does include more than just technology companies. So I suspect there is at least some correlation between tweets and overall market sentiment.
| Symbol |
Company |
Tweets / hour |
| AAPL |
Apple Inc. |
44.12 |
| MSFT |
Microsoft Corp. |
40.64 |
| GOOG |
Google Inc. |
24.61 |
| GS |
Goldman Sachs Group |
17.74 |
| BAC |
Bank of America Corp |
17.74 |
| WAG |
Walgreen Co. |
17.09 |
| T |
AT&T Inc |
13.35 |
| MS |
Morgan Stanley |
13.08 |
| CSCO |
Cisco Systems |
12.95 |
| A |
Agilent Technologies Inc |
12.40 |
| S |
Sprint Nextel Corp. |
11.69 |
| NFLX |
NetFlix Inc. |
11.62 |
| C |
Citigroup Inc. |
11.61 |
| PEP |
PepsiCo Inc. |
10.79 |
| FCX |
Freeport-McMoran Cp & Gld |
10.49 |
| INTC |
Intel Corp. |
10.02 |
| ESRX |
Express Scripts |
9.98 |
| AMZN |
Amazon.com Inc |
9.85 |
| MHS |
Medco Health Solutions Inc. |
9.85 |
| X |
United States Steel Corp. |
9.72 |
I was somewhat surprised to find Walgreen (WAG) in the top 10. PEP, FCX, MHS, and X were a bit surprising, too. The most likely reason for Express Scripts (ESRX) being on the list is that they announced a merger with Medco yesterday. Or was it Wednesday? Either way, Twitter reflects the recent buzz about ESRX.
Note that the list above is just a snapshot. I did a one-time snapshot to get the recent tweets for each stock at one particular time. There are all kinds of things that could skew the data in a single snapshot. You would need several snapshots per day over a few days to get a more reliable list. For my purposes, a single snapshot is just fine.
By Jim, on July 18th, 2011 I’ve carved dozens of these little dogs–perhaps more than a hundred. I know they’re not high art, but I enjoy them. Some people play Solitaire or Bejeweled, watch TV, or sit around drinking beer and talking with friends. I whittle.

You’d think that after carving so many of these I’d “have it down.” But whereas most carvers work primarily with kiln dried basswood, where every piece of wood is almost identical to every other, I tend to work with “found wood.” This piece, for example, is from an apricot limb. The dot between the dog’s ears is the pith, and the tail is a smaller limb that was growing off of this one. The blemish on the nose is part of a worm hole that I didn’t discover until the piece was half finished. I wasn’t about to throw the thing away just because I found a little rotten wood.
Working with found wood is challenging because every piece is unique. This apricot, for example, had been sitting out in the weather for quite a while after it was cut. It has a lot of bugs in it, some soft spots, and some very hard spots. I cook each piece in the oven for a couple of hours to kill any bugs. The apricot cuts very nicely, and is somewhat forgiving if I cut against the grain.
I’ve carved these dogs from more than a dozen different woods, including basswood, mesquite, walnut, mahogany, fig, cherry, pear, maple, ambrosia maple, sumac, oak, apricot, poplar, lyptus, elm, cedar, sycamore, and a few “mystery sticks” that I’ve picked up along the way. Every type of wood, and sometimes different limbs of the same type, has a unique character that presents its own challenges.
If you work in found wood, there’s almost never a shortage of material. Unless I’m stuck in an office building somewhere, I can almost certainly find a tree limb that’s about an inch thick. That and a knife is all I need. In an hour or so, I’ll have a little dog carving.
Another benefit of working in found wood is that I don’t have to paint the dogs. The natural wood grain provides much better coloring than I’ve ever been able to obtain with paint. Besides, I dislike painting.
Some people have asked me why I don’t sand the pieces to remove fuzzies or tool marks. I’ve sanded some of them to remove the fuzzy bits, but often when I carve one of these all I have is the knife. Sandpaper is another thing I’d have to carry around. And I’d never sand one perfectly smooth. The tool marks add character.
I keep saying that I’m going to make a step-by-step picture tutorial about carving these little things. I’ve started a few, but I always get involved with the carving and forget to pause to take pictures. One of these days. Really.
By Jim, on July 13th, 2011 I’ve spent most of my limited carving time recently on more spoons. A few of the little bears, but that’s about it.
My friend Mike was about to throw an old table into the burn pile up at the ranch. I grabbed a few slats to bring home, because I wanted to see what kind of wood was hiding under all that rot.

The wood turned out to be teak. I now have the entire table and will be making things from it.
I got ambitious on Sunday and made another mesquite log spoon. This one turned out really nice.


Finally, another coffee scoop. The wood used is ambrosia maple. I’m getting better with the square or rectangular bowls.

By Jim, on July 8th, 2011 Last month the Congressional Budget Office released their 2011 Long-Term Budget Outlook. Unsurprisingly, this was not reported by the media or mentioned by any of the players in the fiscal tug-of-war that’s currently occupying our elected parasites. It’s likely that most of them haven’t even read the report.
They should, and the the media should be all over this, because the CBO report paints a very bleak picture of our fiscal situation. You should also understand that historically the CBO almost always overstates revenues and understates expenditures. In addition, the language of their reports is purposely vague. Even with that, the picture they paint is frightening indeed.
The report analyzes the impact on the budget under two scenarios. Under the most optimistic scenario, things are bad. Under the more likely scenario, things are catastrophic.
Under the Extended Baseline scenario, expiration of tax cuts, changes in tax law, and the way that the tax system interacts with the economy result in increasing revenues, with revenue reaching about 23 percent of GDP by the year 2035, and continuing to grow after that. At the same time, spending on everything but health care, Social Security, and interest on the debt would decrease to the lowest levels since World War II.
The projected result of the Extended Baseline scenario is that federal debt will increase to about 84 percent of GDP by the year 2035, from about 69 percent now. Interest on the federal debt will be about 4 percent of GDP, or about 17 percent of all federal spending.
The Alternative Fiscal Scenario incorporates several expected changes of law. The 2001 tax cuts, extended in 2010, will be extended again, the reach of the alternative minimum tax is reduced, and in general tax law will change to keep revenues at their historical average of about 18 percent of GDP. In addition, health care costs are projected more realistically and spending on other government programs are not expected to decrease as much as in the Extended Baseline scenario.
Under the Alternative Fiscal Scenario, federal debt explodes, reaching 100 percent of GDP by the year 2021, and 190 percent of GDP by 2035.
According to the report:
Many budget analysts believe that the alternative fiscal scenario presents a more realistic picture of the nation’s underlying fiscal policies than the extended-baseline scenario does. The explosive path of federal debt under the alternative fiscal scenario underscores the need for large and rapid policy changes to put the nation on a sustainable fiscal course.
Note that the current spending is not sustainable, nor is either of the scenarios outlined above.
Scary as those projections are, the really frightening part of the report is contained in the section entitled The Impact of Growing Deficits and Debt. The first paragraph is so alarming that it should have every American calling his elected representatives to demand immediate changes to put us on a sustainable path:
CBO’s projections in most of this report understate the severity of the long-term budget problem because they do not incorporate the negative effects that additional federal debt would have on the economy, nor do they include the impact of higher tax rates on people’s incentives to work and save. In particular, large budget deficits and growing debt would reduce national saving, leading to higher interest rates, more borrowing from abroad, and less domestic investment—which in turn would lower income growth in the United States. Taking those effects into account, CBO estimates that under the extended baseline scenario, real (inflation-adjusted) gross national product (GNP) would be reduced slightly by 2025 and by as much as 2 percent by 2035, compared with what it would be under the stable economic environment that underlies most of the projections in this report. Under the alternative fiscal scenario, real GNP would be 2 percent to 6 percent lower in 2025, and 7 percent to 18 percent lower in 2035, than under a stable economic environment.
In other words, the budgetary impact of our fiscal overindulgence results in a reduction of GNP in the next 15 years.
As I said before, CBO projections in the past have typically been overly optimistic because they are based on current policies and projections of historical averages. They don’t take into account the rate at which government spending has increased over time. Regardless, even the CBO’s most optimistic projection here is unsustainable.
Meanwhile, Congress and the President are wrangling over a plan to reduce deficits by up to four trillion dollars over the next 10 years. Whereas that sounds like a lot (about a 27% decrease in deficit), it still saddles us with about $11 trillion more debt by the year 2021. Assuming, of course, that the plan actually has the projected impact. More likely, flaws in whatever convoluted legislation they cook up will result in much lower savings, and future legislation will negate whatever actual savings is realized.
When you find yourself in a hole with a shovel in your hand, sinking ever deeper, the first thing you should do is stop digging and drop the dang shovel. When you find yourself sinking deeper into debt, the first thing you have to do is stop borrowing money!
Any fifth grader can do the math. If you keep taking more than you’re getting, eventually you have nothing.
Putting our government on a sustainable fiscal path will require pain. A lot of pain. Historical evidence and the most recent CBO report indicate that raising taxes will not have a positive impact on our budgetary problems. The only way we can solve the problem is to reduce spending. As I pointed out in Out of Control, more than 60% of federal spending is on “mandatory” programs: Social Security, Medicare, Medicaid, unemployment, welfare, etc. We spend more on those programs alone than we take in every year in revenues. Those programs must be cut.
But cutting any of those programs is political suicide. Because our elected “leaders” are more interested in keeping their jobs than they are in prolonging our way of life, Congress and the President will end up passing bullshit “reform” legislation that just whitewashes over the problem, perhaps extending the inevitable for another few years–long enough that some other suckers have to worry about it.
So go ahead, Mr. President. Mr. Speaker. Mr. Senate Majority Leader. Have your fun. Make all your silly speeches and look good for your constituents. Play the game of making the other guy look bad so that you and your parties can remain in power. But please don’t expect those of us who actually paid attention in fifth grade math class to swallow the shit you’re shoveling.
By Jim, on June 18th, 2011 The C# conditional operator is a kind of shorthand for an if...else statement. For example, this code:
int result;
if (y == 0)
result = 0;
else
result = x/y;
Can be rewritten using the conditional operator:
int result = (y == 0) ? 0 : x/y;
Terseness isn’t a goal in and of itself, but very often the conditional operator really does make code more readable.
This morning I was working on code that will produce output in XML or JSON, depending on a command line parameter. I created a delegate, two methods (one for each output type), and code to assign it, like this:
delegate void OutputRecordDelegate(MediaRecord rec);
OutputProcDelegate OutputProc = null;
void OutputRecordXml(MediaRecord rec)
{
}
void OutputRecordJson(MediaRecord rec)
{
}
// code to initialize
if (OutputFormat == "xml")
OutputProc = OutputRecordXml;
else
OutputProc = OutputRecordJson;
That all works as expected. It compiles and runs just fine. But this is exactly the kind of code that I think is more readable when written with the conditional operator, like this:
OutputProc = (OutputFormat == "xml") ? OutputRecordXml : OutputRecordJson;
Imagine my surprise when the compiler issued an error message:
Type of conditional expression cannot be determined because
there is no implicit conversion between 'method group' and 'method group'
Now why is that? It looks right!
The problem has to do with type inference. In the statement:
OutputProc = OutputRecordXml;
The compiler uses type inference to convert OutputRecordXml to the proper type: OutputRecordDelegate. It’s as if I had written:
OutputProc = new OutputRecordDelegate(OutputXml);
The compiler has enough information and the smarts to infer the type and do the proper conversion.
But the specification for the conditional operator doesn’t allow that. The rules for type conversion in the conditional operator are a bit different:
The second and third operands of the ?: operator control the type of the conditional expression. Let X and Y be the types of the second and third operands. Then,
- If X and Y are the same type, then this is the type of the conditional expression.
- Otherwise, if an implicit conversion (Section 6.1) exists from X to Y, but not from Y to X, then Y is the type of the conditional expression.
- Otherwise, if an implicit conversion (Section 6.1) exists from Y to X, but not from X to Y, then X is the type of the conditional expression.
- Otherwise, no expression type can be determined, and a compile-time error occurs.
Since there is no implicit conversion that will convert between the two methods, the compiler issues an error message.
I can rewrite the code to do the explicit conversions:
OutputProc = (OutputFormat == "xml") ?
(OutputRecordDelegate)OutputXml :
(OutputRecordDelegate)OutputJson;
Whereas that works, having to do the explicit casts eliminates most of the benefit of using the conditional operator. The simple version (the one that doesn’t work) is, to me, unquestionably more readable than the standard if...else construct. Having to add the casts makes the code a bit awkward, and I see little or no benefit as a result.
By Jim, on June 16th, 2011 The .NET WebClient class abstracts away most of the complexity associated with downloading data from and uploading data to Web sites. Once you instantiate a WebClient instance, you can upload or download a page with a single line of code. For example:
var MyClient = new WebClient();
// Download a page
string pageText = MyClient.DownloadData("http://example.com/index.html");
// send a file via FTP
MyClient.UploadFile("http://example.com/uploads/file.txt", "file.txt");
WebClient also has methods for easily doing asynchronous requests so that you can write, for example:
MyClient.UploadFileCompleted += UploadCompletedHandler;
MyClient.UploadFileAsync(new Uri("ftp://example.com/file.txt"),
"STOR", "file.txt", null);
The file will be uploaded in the background and when it’s finished the UploadCompletedHandler method is called.
As always, the devil is in the details. Documentation for UploadFileAsync says that the method will throw WebException if the arguments are incorrect or if there are errors downloading. For example, if the fileName argument is empty, the method is supposed to throw WebException.
Unfortunately, it doesn’t. This line should throw WebException:
MyClient.UploadFileAsync(new Uri("ftp://example.com/file.txt"), "STOR", string.Empty, null);
Instead, it throws InvalidCastException in a rather odd place, as you can see from this stack trace.
System.InvalidCastException: Unable to cast object of type 'System.ComponentModel.AsyncOperation' to type 'UploadBitsState'.
at System.Net.WebClient.UploadFileAsyncWriteCallback(Byte[] returnBytes,
Exception exception, Object state)
at System.Net.WebClient.UploadFileAsync(Uri address, String method, String fileName, Object userToken)
at testo.Program.DoIt() in C:\DevWork\testo\Program.cs:line 38
This is a problem because WebClient.UploadFileAsync doesn’t say anything about throwing InvalidCastException. If you’re writing code that handles possible errors, you’re going to allow for WebException, but you’ll ignore InvalidCastException. At least, that’s how you should write the code: only catch those exceptions you know about and are prepared to handle. InvalidCastException thrown by UploadFileAsync is definitely unexpected and is an indication of a more serious error.
Ever curious, and since I already had the .NET Reference Source installed, I thought I’d go see why this happens.
The line that throw the error is in a method called UploadFileAsyncWriteCallback, the first few lines of which read:
private void UploadFileAsyncWriteCallback(byte[] returnBytes, Exception exception, Object state)
{
UploadBitsState uploadState = (UploadBitsState)state;
So the value passed in the state parameter must be of type UploadBitsState or of a type that can be cast to UploadBitsState. Otherwise the cast is going to fail. Something, it seems, is passing the wrong type of value to this method.
The code that makes the call is in the UploadFileAsync method. The code in question looks like this:
try
{
// Code that uploads the file.
}
catch (Exception e)
{
if (e is ThreadAbortException || e is StackOverflowException || e is OutOfMemoryException)
{
throw;
}
if(fs != null)
{
fs.Close();
}
if (!(e is WebException || e is SecurityException))
{
e = new WebException(SR.GetString(SR.net_webclient), e);
}
// This next line causes the error.
UploadFileAsyncWriteCallback(null, e, asyncOp); }
I took out the actual uploading code, because it’s not really relevant here. The problem is in the exception handler, in particular the last line. The asyncOp that is being passed as the last parameter to UploadFileAsyncWriteCallback is of type AsyncOperation. But we’ve already seen above that UploadFileAsyncWriteCallback expects the last parameter to be UploadBitsState. Oops.
Because this bugs is inside the exception handler, any exception (other than ThreadAbortException, StackOverflowException, or OutOfMemoryException) that is thrown in the code will end up causing UploadFileAsync to throw InvalidCastException.
Until the code is fixed, there’s nothing you can do to avoid this bug other than by not calling UploadFileAsync. You can code around it by writing your code to handle InvalidCastException in the same way that it would handle WebException, but understand that doing so will hide other problems. If, for example, something caused UploadFileAsync to throw SecurityException, it’s going to be reported as InvalidCastException, and your code won’t be able to tell the difference. It’s not like there’s an inner exception to look at.
I’ve reported the bug at Microsoft Connect. The number is 675575, WebClient.UploadFileAsync throws InvalidCastException.
Update 2011/07/30
The following is from an email acknowledgement I received from Microsoft Connect:
This appears to be a regression in .NET 4.0 and we will plan to address it in a future release. Note that all of the Async Upload methods are similarly affected.
By Jim, on June 13th, 2011 We do a fair amount of batch processing at work, and I have some fairly involved scripts that are combinations of Windows batch files and VBScript. Both languages are wonky in the extreme, and I’ve been uncomfortable working with them. But it’s what I knew I knew four years ago. What were simple scripts early on evolved (as such things are wont to do) over the years until they were large, complicated, and a complete mess.
I resisted replacing those scripts because I thought I didn’t have time to learn a new scripting language. I’d heard of Windows Powershell, but avoided it because I thought it would take too long to convert my scripts. I bit the bullet last week and was pleasantly surprised. It took all of a couple hours to learn enough about Powershell to fully replace the kludge of batch files and VBScript. The result is a cleaner and more robust update process.
Before you write your next batch file, VBScript, or JScript program to automate something on your Windows server, give Powershell a look. Most likely you’ll be able to do what you want more easily, handle errors better, and end up with a much more readable and more easily modified script.
I’ll be the first to admit that I’m not a Powershell expert. But I’ve already been incredibly productive with it. It’s a real command shell that allows access to the entire .NET Framework. Like a .NET interpreter. Very, very cool. A C# or Visual Basic .NET programmer will feel comfortable in Powershell in a matter of minutes.
Very well done and quite useful. Highly recommended.
By Jim, on June 11th, 2011 I’ve often heard, “You are what you eat” It’s true. What you eat and how much you eat accounts for a very large part of your physical development.
What you don’t hear as often (if ever) is that, mentally, you are what you read, watch, listen to, or otherwise experience.
Information theory deals with the quantification of information. A primary concern is entropy, which can be thought of as how much a random value differs from the expected value. In this context, the expected value is based on what is already known. The more the random value deviates from the expected value, the higher the entropy and thus the higher the information content.
Think of it this way. Imagine you’re standing outside in a rainstorm. Your friend approaches and says, “It’s raining.” The information content of that message is almost zero. The only thing you learned is that your friend is a master at stating the obvious, and you probably knew that already. Of course it’s raining!
Regardless of how many people approached to tell you that it’s raining, you wouldn’t learn anything new.
If your primary means of learning about what’s happening in the world is Fox News, Rush Limbaugh, and Glenn Beck, you’re not going to learn anything new. Oh, to the extent that you’re “keeping up with world events,” you’ll learn things. But that’s just trivia. The same thing will happen if your only source of news is The Huffington Post or Michelle Malkin, or whatever narrowly-focused outlet you subscribe to.
If you depend on just one or a very few sources for your news, the information content of the news that you consume will be incredibly low. Why? Because you’re going to hear exactly what you expect. If Congress passes a new jobs bill that the President signs, you already know what Rush Limbaugh is going to say about it. If the President signs a bill that reduces corporate income taxes, you already know what The Daily Kos is going to say about it.
If you hear exactly what you expect, then you’ve received no new information. You’re not learning anything.
The only learning that matters much is that which modifies your world view. Reading the same old opinions expressed on new topics only reinforces your current world view. If you want to learn something, you have to seek out information from sources that will present world views that differ from yours. Sure, some of them are bunk. Most of what Glenn Beck has to say is bunk, too, but that doesn’t stop millions of lazy people from agreeing with him because they can’t be bothered to seek out new sources of information and put forth the effort to evaluate it.
I’m not talking about changing fundamental beliefs. But all of us hold minor beliefs that are based on incorrect or outdated information. We believe things without understanding how we arrived at those beliefs, and then we hold onto those beliefs in spite of obviously correct conflicting evidence.
There’s nothing wrong with strong opinions. Without a strong opinion, it’s impossible to develop a strong argument for or against anything, and it’s impossible to devote your full energy to any pursuit. But those opinions must be weakly held, subject to examination and revision at any time based on new information. A truly wise person has strong opinions that are weakly held–the exact opposite of what partisans or tribalists of all stripes advocate.
There are incredibly intelligent people who are not particularly wise. They continually express just one point of view (typically in the political arena, but sometimes in others) blindly, pointing out the virtues of their side and the faults and foibles of the other side, but acknowledging neither the virtues of the other side nor the faults of their own side. At best, these people are unaware of their own ignorance. At worst, they’re intentionally trying to mislead or manipulate you. Either way, they are not credible sources of information.
By Jim, on June 10th, 2011 Serendipity is an odd thing. Shortly after I wrote my previous entry, I stumbled across a possible solution to the problem while I was reading comments on an unrelated blog post. I spent a little time this morning checking it out.
According to Microsoft Technet, the LargeSystemCache registry entry controls file caching behavior. This key, which is stored at HKEY_LOCAL_MACHINE/CurrentControlSet\Control\Session Manager\Memory Management, “[s]pecifies whether the system maintains a standard size or a large size file system cache, and influences how often the system writes changed pages to disk.”
There are two possible values: 0 and 1. Here’s what the documentation has to say:
| Value |
Meaning |
| 0 |
Establishes a standard size file-system cache of approximately 8 MB. The system allows changed pages to remain in physical memory until the number of available pages drops to approximately 1,000. This setting is recommended for servers running applications that do their own memory caching, such as Microsoft SQL Server, and for applications that perform best with ample memory, such as Internet Information Services (IIS). |
| 1 |
Establishes a large system cache working set that can expand to physical memory, minus 4 MB, if needed. The system allows changed pages to remain in physical memory until the number of available pages drops to approximately 250. This setting is recommended for most computers running Windows Server 2003 on large networks. |
LargSystemCache works in concert with the Size value specified in HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters. The documentation goes on to say:
| Option setting |
Large System Cache value |
Size value |
| Minimize memory used |
0 |
1 |
| Balance |
0 |
2 |
| Maximize data throughput for file sharing |
1 |
3 |
| Maximize data throughput for network applications |
0 |
3 |
That sounds promising. The registry settings on my Windows Server 2008 system were set for file sharing, with a LargeSystemCache value of 1.
Unfortunately, changing the settings had no visible effect on the machine’s behavior when reading large files from the server. I changed LargeSystemCache to 0, set Size to 1, and restarted the computer. When it came back up, I verified the registry settings and then started copying a 60 GB file from the server to my workstation.
I opened Internet Explorer on the server and began surfing the Web. As expected, memory usage increased on the server as the file copy progressed. What wasn’t expected, though, was that when memory filled, Internet Explorer become unresponsive, as did other programs. This is the same behavior I saw when LargeSystemCache was set to 1.
I tried all of the documented values for Size, rebooting the machine after each try, and found no difference in caching behavior.
It’s possible that the LargeSystemCache value is no longer used. After all, the linked TechNet article was written for Windows Server 2003. A blog entry from February 2008, WS2008: Upgrade Paths, Resource Limits & Registry Values, shows the value as “Not Used,” although what that means is unclear. In addition, the last comment on this thread indicates that the value is not used in Server 2008.
I have no way to test whether the registry entries work as advertised in Windows Server 2003. My testing indicates, though, that they have no effect in Windows Server 2008.
By Jim, on June 7th, 2011 Operating systems use file caching to prevent having to read commonly-accessed information from the disk repeatedly. Depending on the situation, the operating system might keep the most recently read stuff in memory, or it might keep the most commonly accessed stuff in memory. Either way, the system dynamically adjusts how much memory it uses for caching. The idea is that if your programs aren’t using the memory, the OS can use it for cache with the understanding that if your programs need it, the cache gets flushed.
It’s kind of like using your neighbor’s covered parking place when he’s away, with the understanding that when he comes back, the space is his again.
A good caching scheme can make a big difference in the speed of disk access. Main memory is slow compared to the CPU, but the disk drive is glacial. If you can avoid hitting the disk for data that’s already been read, you’re way ahead.
A poor caching scheme can result in slower disk access, and an idiotic caching scheme can bring your computer to a halt. In at least one instance, Windows has an idiotic caching scheme that is in a very real sense a security vulnerability.
I’ve mentioned several times the idiotic file caching bug that causes Windows to come to a screeching halt when copying large files. I did a bit more research and testing, and I can say with some certainty what’s happening, although I don’t know all the internals of why. Here’s what appears to be happening.
- Program requests the first block of the file.
- The server reads the first block into memory, and sends it to the client.
- Steps 1 and 2 are repeated many times, and each time the server saves the sent data in memory.
- Over time, memory begins to fill with data that the server is caching. I can’t tell if the server is reading ahead in order to have the next block ready to send, or if it’s holding on to data already sent, just in case somebody else might want to read it.
- When the cache fills free memory, the operating system starts looking for other memory to use. It starts paging idle programs to virtual memory. Then active programs. And then, as bizarre as it may seem, it looks like the operating system starts paging the cache to memory!
In the past, I thought that Windows was reading ahead, caching parts of the file that hadn’t yet been sent to the client. But my test results point more towards the other conclusion: that Windows is buffering things that it’s already sent. I freely admit that I could be wrong, as I haven’t yet been able to say with certainty which of the two is true. Read-ahead caching at least makes some limited amount of sense. Paging a disk cache to disk would be especially idiotic, but it would definitely explain why the system’s responsiveness continues to degrade after memory fills up. Still, I can’t prove that it’s actually doing that.
Without some serious kernel-level debugging, I don’t have any way to determine exactly what is being cached, but it mostly doesn’t matter. The result is the same: Windows pages out working programs and active data in favor of the file cache. Ultimately, this makes the server unusable.
It doesn’t take an exceptionally large file by today’s standards in order for this to happen. For example, trying to copy a 50 gigabyte file from a server that has 16 gigabytes of memory will exhibit this behavior. I’ve experienced this problem on Windows XP, Server 2008, and Windows Server 2008 R2.
This is a security vulnerability because a very large file in a shared directory is an invitation to a denial of service attack. All a user has to do is start copying that file. In short order, the file server’s memory will fill up, it will start swapping, and will stop serving requests. Worse, because the machine becomes non-responsive, there will be no way to see what’s causing the problem or to disconnect the offending computer. The system administrator is left with the options of 1) trying to find and disconnect the perpetrator at the other end; or 2) reset the file server. The first can be very difficult in a large organization, and the second will cause data corruption if there are any files open for writing.
Understand, a user doesn’t have to be a “black hat” in order to perpetrate this “attack”. He could be reading a file that he is authorized to read. The operating system is the culprit here.
What causes this?
In the past, I thought that the problem was caused by the CopyFile and CopyFileEx API functions, which the standard Windows copy commands (COPY, XCOPY, and ROBOCOPY) call to do their jobs. Whereas it’s true that those two functions do exhibit the problem, it’s not limited to those two functions.
It turns out that any program that reads through a large file can trigger this problem. Any program that uses the CreateFile API function to open a network file can cause this to happen. And CreateFile is what ends up being called by the runtime libraries of almost every Windows programming language. Certainly the C++ I/O subsystem calls it, as does the .NET runtime. If you’re writing C# code that accesses very large files across the network, you will see this happen.
I don’t have any way to say for certain what’s going on inside of Windows. That is, I don’t know the internal mechanism that causes this idiotic caching behavior. But I do know that calling CreateFile with default parameters will trigger it, and CopyFile and CopyFileEx appear to call CreateFile with default parameters.
What can you do about it?
From a user’s standpoint, the only option is to find some other program to copy files. TeraCopy works well as a replacement for copying with Windows Explorer. I’ve verified that it does not trigger the caching problem. Although TeraCopy has a command line interface, it just starts the GUI with the parameters you give it. Because it creates a new window, it doesn’t block the command interpreter. As a result, TeraCopy is useless in scripts. At least, I don’t see how to make it work well in a script.
I’ve heard good things about FastCopy, but I haven’t tried it yet. It might be a good replacement for the Windows tools. I don’t know whether FastCopy triggers the caching problem, but I strongly suspect that it doesn’t. I will know more once I test it. Be forewarned: the program has an unusual command line syntax.
Programs that need to copy large files can call CopyFileEx, and pass the new (as of Windows Vista and Server 2008) COPY_FILE_NO_BUFFERING flag in the last parameter. As the documentation for CopyFileEx says:
The copy operation is performed using unbuffered I/O, bypassing system I/O cache resources. Recommended for very large file transfers.
It really does work. If you have a way to call the native CopyFileEx API function, then the copy file problem is solved. C and C++ programmers are in the clear. I’m building a replacement for the .NET File.Copy method, which will allow you to include this parameter. It’s not an ideal solution, but it’s not terribly painful, either. I should have that code available soon.
The bigger problem is how to handle file streams in general. As I said above, any program that reads large files across the network can trigger the bad caching behavior on the file server. This common pattern (in C#), for example, can cause the file server to lock up:
using (var reader = new StreamReader(@"\\server\data\bigfile.txt"))
{
// read lines from the file
}
This code can cause the problem, too:
using (var reader = new BinaryReader(File.Open(@"\\server\data\bigfile.bin")))
{
// read binary file here
}
I’ve never run into the problem using StreamReader, but then I’ve never had to stream a 50 gigabyte text file. I have, however, been tripped up by this problem when using BinaryReader.
The good news for C and C++ programmers and others who use native file I/O (i.e. CreateFile, ReadFile, etc.) is that there’s a solution to the problem, although that solution is somewhat inconvenient. The bad news for .NET programmers is that there is no solution that doesn’t involve unsafe code and some serious mucking about with native memory allocations.
Programmers who call the Windows API functions directly can pass the FILE_FLAG_NO_BUFFERING flag to CreateFile. As the documentation says:
The file or device is being opened with no system caching for data reads and writes. This flag does not affect hard disk caching or memory mapped files.
There are strict requirements for successfully working with files opened with CreateFile using the FILE_FLAG_NO_BUFFERING flag, for details see File Buffering.
The important parts in the File Buffering article have to do with the alignment and file access requirements. Specifically, the buffer size has to be a multiple of the volume sector size and should be aligned on addresses in memory that are integer multiples of the sector size. The second requirement is hardware dependent. So your system might work okay if the alignment is off, but other systems might crash or corrupt data.
So any time you call ReadFile, the address you specify in the lpBuffer parameter must be properly aligned. You can’t pass an arbitrary offset into the buffer like &buffer[15]. You have to do your own buffering. It’s slightly inconvenient, but it’s easy enough to create a wrapper with a buffer and the requisite logic to handle things properly.
.NET programmers are stuck. .NET file I/O ultimately goes through the FileStream class, which knows nothing about the FILE_FLAG_NO_BUFFERING option. There is a FileStream constructor that allows you to specify some options, but the FileOptions enumeration doesn’t include a NoBuffering value. And if you examine the source code for FileStream, you’ll see that it has no special code for ensuring the alignment of the allocated buffer, and no facility for ensuring that reads are done at properly aligned addresses.
You can fake it by creating your own NoBuffering value and passing it to the constructor, but there’s no guarantee that it will work because NET programs can’t control memory allocation alignment. At best, it would work sometimes.
It appears that the only solution for .NET programmers is to create an UnbufferedFileStream class that uses VirtualAlloc to allocate the buffer, and has code to properly handle reads and writes. That’s a big job, especially if you want it to serve as a plug-in replacement for the standard FileStream.
Even if you were to write that proposed UnbufferedFileStream class, it wouldn’t handle all cases, and some others would be a bit inconvenient. For example, the code to open a StreamReader would be:
var ufs = new UnbufferedFileStream(filename, bufferSize);
using (var reader = new StreamReader(ufs))
You can even put it on a single line, since StreamReader will automatically close the base stream:
using (var reader = new StreamReader(new UnbufferedFileStream(filename, bufferSize)))
But other things would still be a problem. For example, .NET 4.0 introduced the File.ReadLines method, which returns an enumerator that reads the file a line at a time. It makes quick work out of reading a text file:
for (var line in File.ReadLines(@"\\server\data\bigfile.txt"))
{
// do something with the line
}
But that will undoubtedly exhibit the bad caching behavior if the file you’re reading is very large, and there’s nothing you can do about it because there’s no corresponding method (at least, not that I know of) that will enumerate the lines in a stream that you pass to it. You can certainly create such a method, but then you have to remember to use it rather than the standard File.ReadLines.
This is a serious bug
I don’t know if this bug still exists in Windows 7. I find very little information about it online, although there are some reports of users experiencing it when copying relatively small files (four gigabytes, for example) from computers that have only one or two gigabytes of RAM. I expect the problem to get increasingly worse as file sizes and hard drive capacities increase.
I’m somewhat surprised that the bug exists at all. I can’t imagine building an operating system that swaps program code and active data to disk in favor of a file cache. That seems like an incredibly stupid thing to do. But I understand that bugs happen. From what I can glean, though, this bug has existed at least since Windows 2000, which would be more than 10 years. And that really surprises me. You would think that in 10 years they’d be able to come up with a solution that would provide the benefits of file caching without the disastrous side effect of bringing a server to its knees with large files.
There is some chatter online about being able to set caching limits, but I haven’t seen anything like a definitive solution, or even anything that looks promising. Certainly nothing I’ve tried solves the problem. If there is a configuration that will prevent this problem, it should be the default. Users shouldn’t have to go digging through tech notes in order to disable an optimization (file caching) that by default is the equivalent of a ticking time bomb.
|
|
|