If you also read my friend Jeff Duntemann’s Web diary, you know that he’s been working on ways to filter spam.  His idea?  Rather than filtering on senders, domains, or specific words, parse the URLs to which messages point.  That’s the point of most spam, after all—to get you to click on a URL that’s embedded in the message.  If a program can identify the target URL as being a spam URL, then you have an (almost) foolproof filter.  Spammers already go through a lot of trouble to obfuscate those URLs, and also place garbage HTML in the message to confuse HTML parsers.  Unfortunately, you can’t just reject all badly-formed HTML because so many mail clients and other HTML tools do such a poor job of generation.

Jeff and I have split the project along fairly logical lines.  I’m writing the communications infrastructure, and he’s working on the filtering and database design.  I got the easy part.  Assuming that the Internet Direct (Indy) POP3 components work (a fair assumption, given my previous success with the Indy components), putting together the proxy won’t be terribly difficult.  Testing it with the major mail clients may prove a little more interesting, and there’s always the question of user interface.  It should be an interesting project, and a welcome change from my day-to-day work writing .NET training presentations.

Last week, Microsoft announced that they would begin capping the number of email messages that Hotmail users were allowed to send each day.  The limit?  A more-than-reasonable 100.  You can read more about the announcement here.  My response?  It’s about @#$ time!  Limiting outgoing emails is probably the easiest and most effective way to prevent spam from casual spammers, and it’ll make the chickenboners’ job much more difficult.  Why Hotmail, Yahoo, and the countless other free email services didn’t do this two years ago is a source of some puzzlement.  I can think of very few legitimate personal users of these services who would need to send anywhere even close to 100 emails a day.  I know that I wouldn’t mind at all if my ISP were to limit my outgoing emails to 100 per day.  Let’s hope that the other email services and other ISPs follow Microsoft’s lead here.

AOL announced today that its front-line filters trashed a billion spam email messages in a single 24-hour period.  That’s one billion messages that were canned before they were delivered to subscribers’ inboxes, or an average of 28 messages per subscriber.  That’s an astonishing number.  It’s interesting to note, though, that AOL’s filters have a false positive problem.  As the article says:  “…an extremely small fraction of messages snagged in AOL’s spam filters were legitimate ones.”  The AOL spokesman declined to give any figures on the false positives.  Yes, folks, spam is a real problem.  I’ve noticed an increase here recently, including at least one every day that has the “klez” virus.  RoadRunner strips the attachment from those, fortunately.

The Internet Engineering Task Force (www.ietf.org) has finally decided to get involved.  They’ve formed the Anti-Spam Research Group (ASRG) to study the problem and propose solutions.  One wonders why the IETF has taken so long to get involved.  It’s not like spam is a new problem.  It looks like they’re focusing on a consent-based system which looks like a fancy term for filtering.  They do mention the addition of a source tracking component, which I think is necessary for effective spam blocking.  Source tracking, of course, makes anonymous email a bit more difficult, so the idea will meet some resistance.  It will be interesting to see how this proceeds.

My friend Jeff Duntemann posted a note yesterday in his web diary about using trainable Bayesian filters to filter spam.  I still don’t agree that filtering is the best way to combat spam, but it’s probably the best we’re going to get, all things considered.  Blocking spam at the source (i.e. preventing it from entering the system in the first place) would be much more effective, but the design of the email protocols, and resistance to change prevent implementation of an effective Internet-wide spam blocking scheme.  So we’re left with filtering at the delivery end.

The nice thing about Bayesian filters, as Jeff points out, is that they are trainable.  And the one that everybody’s talking about (see Jeff’s site for the link) has a 99.5% success rate, with zero false positives.  It’s impressive, and perhaps this is the way to go.  But on the client?  Like spam blocking, filtering should be done on the server.  All it would take is some simple modifications to the email server, a few extensions to the POP and IMAP mail protocols, and everybody could have spam filtering regardless of what email client they’re using.  Filtering on the server would be much more efficient than having each individual client do the filtering.  Plus, servers could implement black list filtering on a per-user basis, and perhaps stop a large amount of unwanted email from ever being accepted.

Do I expect this to happen?  Sadly, no.  Even as outdated and inefficient as our mail protocols are, I don’t expect them to be changed any time soon.  We’re left waiting for the established email clients to include this kind of feature, or for somebody to come up with a new email client that has a good interface, includes all of the features we’ve come to expect, and also has advanced spam blocking features.  I think it’s going to be a long wait.

So spam is killing email, and the way to stop it is with opt-in systems.  Or so Kevin Werbach says in his article on Slate.  This is stupid!  It will work, true.  Whitelists very effectively block mail from everybody except those that you specifically allow.  Your inbox will be free of clutter.  It also will be free of order confirmations, mailing list messages, status updates, and the dozens of other types of automated mail that you want to arrive.  Not only that, but whitelists only treat the symptom.  They keep your inbox clean, but do nothing to stop spam (is it really one-third of the Internet mail traffic?) from clogging the email system.

Some genius who posted a comment on the article proposed that we change the system to make it computationally expensive to send email.  The theory is that it wouldn’t affect normal users who send maybe a few dozen emails a day.  They won’t care if it takes five or ten seconds to process an email before sending it.  But bulk mailers would be out of business.  A ten second delay between emails would limit them to fewer than 10,000 messages a day.  This, too, would do more harm than good.  First off, you’d have to change the entire email protocol.  I’m not sure if the people at Slate who left that comment visible are poking fun at the guy (I sure hope so), or if they really think it’s a good idea.

If people are willing to change the entire email protocol base, then it’s time to design a high-performance, secure, flexible, and extensible system that’s based on current technology.  I would suggest that end-to-end accountability be part of the new system.  That would eliminate spam, as well as provide a modern system that can more easily handle the features and volume that we need.  The current system is based on and designed to work with 30-year-old technology.  That it’s lasted this long reflects well on its designers.  But it wasn’t designed for today’s environment, and its limitations are fast becoming a hindrance to continued use.

I got the following message from my company’s email server today when I tried to send a message to a client.  I’ve removed the client’s name and company:

Subject: FW: Inquisite 360
Sent: 9/10/2002 2:59 PM

The following recipient(s) could not be reached:

‘name_removed@client.com‘ on 9/10/2002 3:02 PM
Your mail system could not find a way to successfully
communicate with the destination system. Please notify your
administrator.
<cerberus.catapultsystems.com #5.5.0>

I scratched my head over this one for a couple of hours, trying all kinds of different things to get the message to go through.  Somebody finally suggested that I zip up the Microsoft Word document that I was sending as an attachment.  Presto!  The mail went through.  Apparently the destination server is rejecting .doc attachments by returning a fatal error.

This isn’t the first one of these I’ve gotten.  On December 7 last year, I reported a similar message from another client’s mail server.  We’ve checked those files every which way, and have found no virus.  I think some administrators need to take a better look at their configurations.

I’ve seen variants of this scam for the last year or two, but this one adds a new wrinkle.  Here’s the letter, exactly as I received it:

From:  femi douglas <femidouglas@yahoo.com> 
Subject  ATTN: Mischel
Date:  Mon, 9 Sep 2002 03:53:33 -0700 (PDT)
To:  <femidouglas@yahoo.com> 

Barister Femi Douglas.
Morgan Partners.
Ahmed Onibudo Crescent,
Victoria Island,Lagos-Nigeria.
email: femidouglas@yahoo.com

Dear Mischel ,

I am Barrister Femi Douglas ,a solicitor at law. I am the personal attorney to Mr Mark H. Mischel ,a national of your country, who used to work with shell development company in Nigeria. Here in after shall be referred to as my client.

On the 21st of April 1999, my client, his wife And their three children were involved in a car accident along sagbama express road. All occupants of the vehicle unfortunately lost there lives.

Since then I have made several enquiries to your embassy to locate any of my clients extended relatives this has also proved unsuccessful.

After these several unsuccessful attempts, I decided to track his last name over the Internet, to locate any member of his family hence I contacted you.

I have contacted you to assist in repartrating the money and property left behind by my client before they get confisicated or declared unserviceable by the bank where this huge deposits were lodged.

Particularly, the finance company where the deceased had an account valued at about US$3,500,000.00 dollars has issued me a notice to provide the next of kin or have the account confisicated within the next ten official working days.

since I have been unsuccesfull in locating the relatives for over 2 years now I seek your consent to present you as the next of kin of the deceased since you have the same last name so that the proceeds of this account valued at US$3,500,000.00 dollars can be paid to you and then you and me can share the money. 60% to me and 40% to you

I have all necessary legal documents that can be used to back up any claim we may make. all I require is your honest cooperation to enable us see this deal through.

I guarantee that this will be executed under a legitimate arrangement that will protect you from any breach of the law.

please get in touch with me by my email to enable us discuss further.

Best regards,
Barister Femi Douglas esq.

It’s an interesting twist on the corrupt minor African government official who desperately needs somebody to help him spirit an embezzled fortune out of the country.  What surprises me the most about this letter is that it’s so poorly written.  You’d think that somebody wanting to appear as a lawyer would have impeccable grammar and spelling.  Perhaps it’s calculated to sound like an especially stupid and somewhat corrupt lawyer.  After all, he is trying to stick it to me by taking 60% of my supposed inheritance.

I understand that people are falling for these types of scams regularly.  Somehow they get talked into giving up critical bank account information, and the scam artist cleans it out.  I may not be the most worldly person, but I didn’t fall off the turnip truck yesterday.  Sometimes you just shake your head and move on.

The August 19 issue of eWeek has a group of articles about spam.  One article describes the problem, and provides some interesting numbers about how spam has grown over the last year, and what spammers are selling.  Other articles discuss current filtering techniques, and some proposed legislation.  What strikes me about these articles is that almost all of the “experts” agree that filtering is at best temporarily effective, and that legislation likely will be wholly ineffective.  And yet, those same “experts” continue to support legislation.  Huh?

Just how effective do you expect anti-spam legislation to be when the very same legislators who will be passing the laws will use unsolicited bulk email as a tool to get re-elected?  If you doubt that, take a look at this article from The Mercury News.  In related news, the Federal Election Commission decided that it’s okay for political ads transmitted over SMS (short messaging service) to forego transmitting disclosure information.  To be fair, disclosure exemptions are common practice for media that are limited to small numbers of characters.  Still, expect your mobile phones and perhaps your text pagers to be flooded with political spam in the next few months.  I’m sure your newly elected legislators will be happy to pass an anti-spam bill after they see how effective those campaigns are.

I’ve mentioned the Internet Mail Consortium before.  Today I found two reports on their site:  Unsolicited Bulk Email: Definitions and Problems, and Unsolicited Bulk Email:  Mechanisms for Control.  Both reports were written in 1997, and the “Mechanisms for Control” report was updated in May of 1998.  The articles do a good job of defining the problem and identifying in broad terms the possible solutions.  One thing of note is that this is the only place I’ve seen serious discussion of my “trusted server” idea.  They call it “First-hop Accountability.”  I find nothing on their site that takes this idea any further, nor any real discussion of the spam problem in general, other than support of legislation.  I’m disappointed, but not terribly surprised, I guess, by the IMC’s ineffectiveness.  Their main web page says:

The Internet Mail Consortium is the only international organization focused on cooperatively managing and promoting the rapidly-expanding world of electronic mail on the Internet. The goals of the IMC include greatly expanding the role of mail on the Internet into areas such as commerce and entertainment, advancing new Internet mail technologies, and making it easier for all Internet users, particularly novices, to get the most out of this growing communications medium.

In truth, I think the member organizations are members only to protect their own interests, and ensure that they’re kept abreast of any proposed changes.  That they haven’t actually done anything to help combat the spam problem shows me that the IMC is just another group of industry “leaders” who aren’t at all interested in solving anything.

Finally, I wonder how receptive Brightmail and other anti-spam software providers would be to an effective solution to the spam problem.  If I were a conspiracy theorist, I might even accuse these companies of supporting spammers.  But, no, they wouldn’t do that.  Would they?