Back in April I mentioned the possibility that some spam messages are encoded communications sent to millions of people in order to mask the identity of the receiver and possibly the sender.  Here’s another possibility:  spam as a method of destabilizing society.  As more people get email and everybody gets more spam, we’re wasting huge amounts of time and effort deleting it, blocking it, and bitching about it.  The cyber terrorists who have hatched this evil plot are smart enough not to bomb the system with unmanageable amounts of spam right off, but rather are slowly increasing the amount.

It’s like that urban legend about the frog. Dropped in a pan of boiling water, a frog will do whatever it can to get out.  But if you put the frog in a pan of warm water and slowly increase the heat, the frog will stay there and die.  The spam terrorists are using the same technique—slowly increasing the amount of spam until it makes a serious dent in our productivity.  They know that if they bombed our systems with unmanageable amounts of spam right off, we’d do something right now to prevent it.  But by starting light and slowly increasing the amount of spam, they can disrupt us for a long time, and make it very difficult for people to agree on when enough is enough.

I just have to come up with reasonable (even if far-fetched) reasons for the increase in spam.  There just can’t be enough stupid people buying penis enlargement pills or human growth hormone to make even the low cost of spam worthwhile.  Can there?

If you also read my friend Jeff Duntemann’s Web diary, you know that he’s been working on ways to filter spam.  His idea?  Rather than filtering on senders, domains, or specific words, parse the URLs to which messages point.  That’s the point of most spam, after all—to get you to click on a URL that’s embedded in the message.  If a program can identify the target URL as being a spam URL, then you have an (almost) foolproof filter.  Spammers already go through a lot of trouble to obfuscate those URLs, and also place garbage HTML in the message to confuse HTML parsers.  Unfortunately, you can’t just reject all badly-formed HTML because so many mail clients and other HTML tools do such a poor job of generation.

Jeff and I have split the project along fairly logical lines.  I’m writing the communications infrastructure, and he’s working on the filtering and database design.  I got the easy part.  Assuming that the Internet Direct (Indy) POP3 components work (a fair assumption, given my previous success with the Indy components), putting together the proxy won’t be terribly difficult.  Testing it with the major mail clients may prove a little more interesting, and there’s always the question of user interface.  It should be an interesting project, and a welcome change from my day-to-day work writing .NET training presentations.

Last week, Microsoft announced that they would begin capping the number of email messages that Hotmail users were allowed to send each day.  The limit?  A more-than-reasonable 100.  You can read more about the announcement here.  My response?  It’s about @#$ time!  Limiting outgoing emails is probably the easiest and most effective way to prevent spam from casual spammers, and it’ll make the chickenboners’ job much more difficult.  Why Hotmail, Yahoo, and the countless other free email services didn’t do this two years ago is a source of some puzzlement.  I can think of very few legitimate personal users of these services who would need to send anywhere even close to 100 emails a day.  I know that I wouldn’t mind at all if my ISP were to limit my outgoing emails to 100 per day.  Let’s hope that the other email services and other ISPs follow Microsoft’s lead here.

AOL announced today that its front-line filters trashed a billion spam email messages in a single 24-hour period.  That’s one billion messages that were canned before they were delivered to subscribers’ inboxes, or an average of 28 messages per subscriber.  That’s an astonishing number.  It’s interesting to note, though, that AOL’s filters have a false positive problem.  As the article says:  “…an extremely small fraction of messages snagged in AOL’s spam filters were legitimate ones.”  The AOL spokesman declined to give any figures on the false positives.  Yes, folks, spam is a real problem.  I’ve noticed an increase here recently, including at least one every day that has the “klez” virus.  RoadRunner strips the attachment from those, fortunately.

The Internet Engineering Task Force (www.ietf.org) has finally decided to get involved.  They’ve formed the Anti-Spam Research Group (ASRG) to study the problem and propose solutions.  One wonders why the IETF has taken so long to get involved.  It’s not like spam is a new problem.  It looks like they’re focusing on a consent-based system which looks like a fancy term for filtering.  They do mention the addition of a source tracking component, which I think is necessary for effective spam blocking.  Source tracking, of course, makes anonymous email a bit more difficult, so the idea will meet some resistance.  It will be interesting to see how this proceeds.

My friend Jeff Duntemann posted a note yesterday in his web diary about using trainable Bayesian filters to filter spam.  I still don’t agree that filtering is the best way to combat spam, but it’s probably the best we’re going to get, all things considered.  Blocking spam at the source (i.e. preventing it from entering the system in the first place) would be much more effective, but the design of the email protocols, and resistance to change prevent implementation of an effective Internet-wide spam blocking scheme.  So we’re left with filtering at the delivery end.

The nice thing about Bayesian filters, as Jeff points out, is that they are trainable.  And the one that everybody’s talking about (see Jeff’s site for the link) has a 99.5% success rate, with zero false positives.  It’s impressive, and perhaps this is the way to go.  But on the client?  Like spam blocking, filtering should be done on the server.  All it would take is some simple modifications to the email server, a few extensions to the POP and IMAP mail protocols, and everybody could have spam filtering regardless of what email client they’re using.  Filtering on the server would be much more efficient than having each individual client do the filtering.  Plus, servers could implement black list filtering on a per-user basis, and perhaps stop a large amount of unwanted email from ever being accepted.

Do I expect this to happen?  Sadly, no.  Even as outdated and inefficient as our mail protocols are, I don’t expect them to be changed any time soon.  We’re left waiting for the established email clients to include this kind of feature, or for somebody to come up with a new email client that has a good interface, includes all of the features we’ve come to expect, and also has advanced spam blocking features.  I think it’s going to be a long wait.

So spam is killing email, and the way to stop it is with opt-in systems.  Or so Kevin Werbach says in his article on Slate.  This is stupid!  It will work, true.  Whitelists very effectively block mail from everybody except those that you specifically allow.  Your inbox will be free of clutter.  It also will be free of order confirmations, mailing list messages, status updates, and the dozens of other types of automated mail that you want to arrive.  Not only that, but whitelists only treat the symptom.  They keep your inbox clean, but do nothing to stop spam (is it really one-third of the Internet mail traffic?) from clogging the email system.

Some genius who posted a comment on the article proposed that we change the system to make it computationally expensive to send email.  The theory is that it wouldn’t affect normal users who send maybe a few dozen emails a day.  They won’t care if it takes five or ten seconds to process an email before sending it.  But bulk mailers would be out of business.  A ten second delay between emails would limit them to fewer than 10,000 messages a day.  This, too, would do more harm than good.  First off, you’d have to change the entire email protocol.  I’m not sure if the people at Slate who left that comment visible are poking fun at the guy (I sure hope so), or if they really think it’s a good idea.

If people are willing to change the entire email protocol base, then it’s time to design a high-performance, secure, flexible, and extensible system that’s based on current technology.  I would suggest that end-to-end accountability be part of the new system.  That would eliminate spam, as well as provide a modern system that can more easily handle the features and volume that we need.  The current system is based on and designed to work with 30-year-old technology.  That it’s lasted this long reflects well on its designers.  But it wasn’t designed for today’s environment, and its limitations are fast becoming a hindrance to continued use.