Jeff Duntemann mentioned in his Web diary post for May 30 that we’d been puzzling over some strange email messages we’ve both been receiving.  These messages have none of the standard header fields beyond the tracing information (i.e. the “Received:” lines):  no subject, no to field, no from field, and no message body.  I’ve been seeing these on and off for a few months now, but they seem to be getting more prevalent.  For a while I thought it was malfunctioning SMTP or POP servers because I figured the specifications wouldn’t allow such a message to be passed.  I was wrong.  SMTP servers are quite happy to pass on empty messages.  As it turns out, SMTP servers don’t need any of the message header information in order to accept and deliver a message.  Everything they need to get a simple message across is supplied by the MAIL FROM and RCPT TO commands to the server.  I was able to send myself one of these blank messages, although I won’t say how it’s done.  Read RFC2821 and figure it out for yourself if you’re so inclined.

One other thing to note is that the original messages could have contained some text in the message body, but the resulting message is badly formed (not conforming to RFC2822).  Mail clients barf when trying to parse the message.  I connected to my POP server with telnet to examine one of my test messages and found that the body text is placed in the wrong position; immediately after the headers without the intervening blank line that’s required by the specification.  I have to wonder if this is a feature of the spec or a bug in some server implementations.

The more interesting question is the source of these messages.  Who would send badly formed email messages?  I’d suspect a denial of service attack, except that a dozen or so messages per week hardly represents an “attack” in my book.  My best guess is a malfunctioning home-grown mail program, most likely a spam utility, but without more information it’s hard to say.  I do find it odd, though, that a spam utility would relay through the spammer’s SMTP server rather than connecting directly to the target SMTP server.  I can show that this happens by examining the tracing information in the message.

On my morning scan of Techdirt I picked up this story about Yahoo working on an email authentication plan that would let senders prove they are who they say they are.  This only two and a half years after I suggested it here (May 15, 2001).

The beauty of Yahoo’s plan is that it will continue to work with existing message traffic, allowing even unauthenticated email to pass.  That might seem folly at first glance.  The article is short on detail, but I suspect that Yahoo will have a way to flag a message as authenticated or not, thereby giving email clients a method of filtering unauthenticated messages.  Yahoo will make the source of their “Domain Keys” software available to open-source email software and systems, which means that a large percentage of clients will have the ability to create and filter these messages.  I wonder if they’ll also make it available to developers of proprietary systems.  I certainly hope so.  Otherwise we’ll end up with competing standards that will make the problem even worse.

This is exactly what we’ve needed: a major player in the email space to take the lead and implement something.  If it works out well for Yahoo, then the other major email providers will have ample incentive to follow suit.  Some will argue that reverse DNS lookup is already available, and since very few servers use it today there’s no reason to expect that they’ll use this new system that is essentially the same thing: a DNS “private key” lookup.  The article doesn’t provide any detail, but I would suspect that Yahoo’s people looked into SMTP authentication and found it lacking.  The system the article describes sounds stronger than what’s already available.  I sure hope it works out.

I honestly thought somebody would have fixed the email spam problem by now.  I resisted installing a filter for years, first because my spam problem wasn’t all that bad, then because filters just shift the problem, and finally out of sheer pig-headedness:  I hate sub-optimal solutions.  I still have to review every message sender and subject line.  I’d still like a real solution, but the amount of spam I get is nearly unbearable.  On Jeff Duntemann‘s recommendation, I downloaded and installed POPFile—a trainable Bayesian filter.  It’s been 2 weeks now since I first installed the thing, and after about 1500 messages (oddly, my spam count has gone down markedly since mid-October) I’m still teaching it the difference between spam and good mail.  I’ve added “magnets” that automatically classify important personal and work-related message, but I’ve resisted adding magnets for everybody in my contact list in the hopes of training the silly thing to tell the difference between jokes from friends and ads for questionable drugs.

Beyond automatically throwing almost everything in the “Junk” box, the filter isn’t yet saving me much time.  Just as I feared, I still have to review the filter’s output to ensure that it hasn’t mis-classified an important message as spam.  I’m hoping that it gets smarter as it gets more experience, but at the moment it’s just as much work with the filter as without.  I’m going to give it until the end of the year.  If it’s still missing 5% or more after that, then I’ll have to re-evaluate the wisdom of using this type of filter.

I’m very disappointed that nothing has yet been done at the protocol level to address the spam problem.  Maybe that’s still coming?  I won’t hold my breath.  From where I sit, it looks like another 5 to 10 years (if ever!) before the protocols can be changed to prevent  most types of spam.

Eric Allman, creator of Sendmail, has weighed in on the spam problem with a well written article about the current state of spam, possible methods of preventing it, and problems inherent with those techniques.  He doesn’t paint a pretty picture.  With the spam doubling rate at something like 8 weeks, a spam filter that lets through only 1.5 percent of spam will, in one year, be letting through as much spam as it blocks today.  So if your filter is blocking 197 out of 200 messages today, in a year it will be letting 197 messages through!  It’s a big problem.  As Allman says:  It’s an arms race and nobody wins but the arms dealers.

Allman is the latest in the growing number of informed industry leaders who agrees that protocol changes are required.  They’re not going to happen quickly, though.  He mentions a time frame of 10 years!  It looks like filtering is the only possible short-term solution.  Gah!

I checked my email after dinner last night.  A few hours later I checked it again and had 70 new messages.  PocoMail flagged every one of them as junk mail because all but one of them was a variant of the message claiming to contain the latest security patch from Microsoft.  The oddball was a pitch to sell me $100,000 homes for $10,000.  As of this evening—24 hours after I first got one of those messages—I had received over 250 of those emails.  That’s in addition to my apparently meager spam load of about 80 messages per day.  Oddly enough, Debra hasn’t received even one of these worm emails.

This has gotten totally out of hand.  I’m surprised that, after the last 3 or 4 years of worms and viruses, there are still so many people who will unquestioningly run a program that they get via email from an unknown person.  Are these the same people who will give their credit card numbers over the phone?

This kind of attack would be much less apt to succeed if the email protocols required end-to-end authentication.  If there was no way to get an anonymous message into the system, then it would be very difficult for somebody to start this without getting caught.  In addition, people opening their messages could easily check to see if the message really was from Microsoft Support before running the attachment.  Laws that prescribes penalties for perpetrators are wholly ineffective at stopping such attacks, because it’s very difficult or impossible to prove who sent the message.

I sent email to one of my clients today and promptly got a message back from their email server telling me that my message was rejected because our IP address is being blocked, and to check the relays.osirusoft.com website (purposely not linked) for details.

It turns out that Joe Jared, operator of relays.osirusoft.com and a very aggressive (to put it mildly) anti-spammer, decided yesterday to discontinue his service.  Rather than do something reasonable like just shut down the server or remove all entries from his blocking list, he decided to mark every IP address as a spammer domain so that people would get the message that he’s no longer offering the service.  This affected a huge number of mail servers because over-zealous systems administrators had been relying on that list as their primary or only weapon against spam.  Never mind that there are some well documented cases of the operators of this and similar lists behaving quite irresponsibly.

Yet another argument against using filtering to fight the spam problem.  If you think a client-side filter is subject to false positives, imagine what happens when a legitimate ISP is blocked by one of these filters because one person has a personal score to settle.

I’ve been saying for a couple of years now that the only way we’ll be able to get a handle on the spam problem is by making a fundamental change to the email protocols.  And for most of those two years, I’ve been inundated by messages from people saying that it’s not possible, that it’d cause too much upheaval, that any system could be broken anyway, and that I must be some kind of communist because I don’t believe that people sending email have some right to anonymity.  Not one of those messages, and none of the garbage I’ve seen regurgitated on Slashdot and other public forums have given any evidence of why this isn’t possible.  It’s all a bunch of reactionary scribbling by people who engage their fingers on the keyboard before they engage their brains.

Finally, about three years too late, knowledgeable people are beginning to poke their heads up out of the sand and step out from behind their proposed legislation, and saying what many of us have known for years: it’s time to rewrite the protocol.

Back in April I mentioned the possibility that some spam messages are encoded communications sent to millions of people in order to mask the identity of the receiver and possibly the sender.  Here’s another possibility:  spam as a method of destabilizing society.  As more people get email and everybody gets more spam, we’re wasting huge amounts of time and effort deleting it, blocking it, and bitching about it.  The cyber terrorists who have hatched this evil plot are smart enough not to bomb the system with unmanageable amounts of spam right off, but rather are slowly increasing the amount.

It’s like that urban legend about the frog. Dropped in a pan of boiling water, a frog will do whatever it can to get out.  But if you put the frog in a pan of warm water and slowly increase the heat, the frog will stay there and die.  The spam terrorists are using the same technique—slowly increasing the amount of spam until it makes a serious dent in our productivity.  They know that if they bombed our systems with unmanageable amounts of spam right off, we’d do something right now to prevent it.  But by starting light and slowly increasing the amount of spam, they can disrupt us for a long time, and make it very difficult for people to agree on when enough is enough.

I just have to come up with reasonable (even if far-fetched) reasons for the increase in spam.  There just can’t be enough stupid people buying penis enlargement pills or human growth hormone to make even the low cost of spam worthwhile.  Can there?