Tag Archives: spam

SpamAssassin and Vipul’s Razor

Posted on by

My friend Darrin Chandler wrote regarding my June 16 entry about spam filtering.  He pointed me at SpamAssassin and Vipul’s Razor, two more attempts at spam filtering.  And, sadly, two more that are bound to fail.   SpamAssassin plugs into a number of different places in the email flow, and uses heuristics and blacklists (like Vipul’s Razor) to attempt to identify and filter the spam.  SpamAssassin is for Unix and Unix-like (i.e. Linux, BSD, etc.) operating systems.  Perhaps a Windows port is forthcoming?  Vipul’s Razor is, according to the web site:

…a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content.

We’ll see how well it works.

Darrin also pointed out a fundamental flaw in my trusted server idea:  getting a critical mass of important ISPs to subscribe to such a thing.  If AOL were to do it, for example, they’d get millions of emails from irate subscribers who all of a sudden can’t get email from Aunt Tilly, because Aunt Tilly’s ISP doesn’t subscribe to the trusted server network.  That’s really too bad.  It leads me to believe that most people don’t really care about spam.  Sure, they’d like not to get it, but not if it means having to change ISPs or possibly lose communication with Uncle Harold.

Opt-in spam filtering

Posted on by

A friend of mine ran a spam filtering idea past me last week:  Rather than accept everything and then spend time filtering the trash from your inbox, why not create a list of people from whom you’ll accept mail, and then block everybody else?  The system he envisioned would still accept mail from non-approved senders, but put it in a separate folder for you to scan at your leisure.  This has the benefit of ensuring that what ends up in your inbox is all good.  He had a blocked senders list, of course, and also word filtering. Monday, Jeff Duntemann mentioned MailCircuit, which implements this “opt-in” type of mail filtering, in his web diary.  Perhaps some people will find this kind of filtering useful.  It seems like more work to me, though, or at least the same amount of work.  The only thing it does is unclutter my inbox, but I still have to scan those other messages.  I’m not interested in any anti-spam measure that requires more work on my part.

On Tuesday, osopinion.com ran this article by John Christie advocating what he calls “Reverse FIltering.”  His system, too, is an opt-in list.  You set up a list of approved senders whose mail will always be accepted.  Messages from non-approved senders will be bounced back, with a message describing how to get onto the approved list.  That method typically would be to visit a web page and fill out an application of some sort—presumably an application that cannot easily be filled out by an automated system.  This certainly would work, but it would block way too much legitimate mail.  For example, if you give your email address to anybody, you’d better be sure to add those people to your approved senders list before they send a message.  This includes friends and relatives, as well as web site signups that require email validation.

Others in the Talk Back Forum for the OS Opinion article have put forth variations on these opt-in systems.  All of them suffer from one or more of the problems that I’ve already pointed out.  Perhaps the most common mistake is relying on the “from” email address for filtering.  That value just isn’t reliable—it’s ridiculously easy to put a bogus address there.  The “Reply-to” and “Return-Path” addresses also are unreliable.  It’s impossible to write a reliable filter based on those addresses.

I’ll say it again.  The only reliable way to prevent spam from getting to your inbox is to prevent it from entering the SMTP mail system.  Period.  Once it’s in the system, it’s impossible to filter without also filtering some potentially legitimate messages.  I have yet to see (and I’ve been looking!) any proposal for filtering spam that would be as unobtrusive or as effective as the system of trusted servers that I suggested in my May 15, 2001 entry.  Is nobody giving this problem any serious thought?  Are they all sitting on their hands waiting for legislators to come up with a solution, so they can then complain about how the new law is ineffective and overly burdensome?  I’m very disappointed.

Spam == bad data

Posted on by

In any software system, the only effective way to deal with bad data is to prevent it from entering the system.  User interface programs regularly do this by, for example, refusing to let you enter your name when you’re prompted for your date of birth.  If the data is guaranteed to be valid when it enters the system, then the rest of the program can operate without continually checking.  Can you imagine how much more complex and fragile the program would be if it allowed you to enter “George” for your date of birth?  What would happen if the program tried to compare “George” to the current date?

Email spam is bad data.  And once it enters the Internet mail system, it’s impossible to eradicate.  It gets passed from server to server, and then to the clients, wasting huge amounts of bandwidth in transmission and causing untold frustration to millions of users.  Why do we let it into the system?  The only effective way to cure the spam problem is to prevent spam from entering the system in the first place.  We have the technology to do this, but for some reason lack the will.  I simply cannot understand why ISPs and large companies have not come up with a system of trusted servers that guarantee no spam.  If somebody can shed some light on this, I’d be happy to listen.

Why keyword filtering won’t stop spam

Posted on by

It occurred to me this week, while I was evaluating anti-spam tools for Debra, and setting up the email filtering rules for her system, that any type of spam filtering based on sender’s email address or message subject line is doomed to fail.  Message content filtering based on key words or phrases is also doomed to fail.

Filtering based on sender’s email address is just silly.  It’s trivial for me to generate a message that appears to come from any arbitrary email address.  Want a message that appears to come from Bill Gates?  No problem.  I can make the “from” line in an email address look any way I choose.  I could write a program that generates random email addresses—a different “from” address for each message.  As long as the domain name (the part after the @ sign) is valid, the message will get through.  So go ahead and block that random address.  What do I care?

There are two things to consider about key word and phrase filtering.  The spammers send mail that contains certain words and phrases that we add to our filters.  Be assured that the spammers subscribe to the anti-spam sites, and they know which words and phrases are being filtered.  So they rewrite their emails with new words or unique constructions of the old words and phrases.  We build a better mouse trap, they build better mice.  It’s an escalation game in which the spammers have the upper hand.

The bigger problem with word and phrase filtering, though, is the increasing possibility of rejecting a legitimate message as spam.  As you add more words and phrases to your rejection list, you increase the probability that a good message will contain a “bad” word, and get shuttled off to the trash bin.  If you automatically delete suspected spam, then you’ve just lost a potentially important message.  And if you put suspected spam into a folder for later review, you’ve done nothing to reduce the workload.  Why spend time setting up message filters if you have to look through the messages by hand anyway?  The problem of legitimate information being lost during noise filtering is nothing new.  Audio engineers have dealt with it for decades.  That the Internet community seems to think that they can ignore the problem leaves me to wonder whether they’ve actually considered it thoroughly.

Outlook spam filter

Posted on by

I’ve finally gotten tired of all the junk email I get.  I’m still not going to suggest legislation, but I’ve gotten to the point where I don’t want to manually delete all that crap.  So I turned on the junk email filter in Outlook and started adding spam senders to the junk senders list.  I did this at home a while back, and at work (where I keep getting porn site ads in Spanish for some reason) just the other day.

I’ve had to turn the junk filter off at work because the first two messages (the only two so far) that it flagged as suspected junk were legitimate messages—one from a client and one from a co-worker.  I thought I’d take a look at Outlook’s filtering rules, but can’t seem to do that.  What I really want is an option like Outlook Express’s “Block Messages from Sender”.  If the option is there in Outlook, it’s hidden somewhere in that bewildering Options dialog.  Somebody at Microsoft needs to take a very close look at that program’s user interface.

Actually, the UI isn’t my only complaint with Outlook, and I’m looking to replace it on my home system.  Eudora looks like it might be worth another look—I used it briefly in 1997 but didn’t like having to swap out the MAPI DLL every time I wanted to receive a fax.  I don’t receive faxes anymore, so MAPI isn’t an issue, and my understanding is that Eudora has fixed many of the other problems I had with it.