Spam Free Email

I’ve been talking to people about SPF over the past few weeks. In those conversations most people who are running email servers don’t think SPF is working. In many cases they believe that SPF is being used more by spammers then it is by people who are protecting themselves from spam and I can’t argue with either of those points.

The simple fact that a domain has a SPF record does not mean the domain will never send spam. The idea is that once SPF records are in wide use we will know where the spam comes from more reliably then we do now.

I personally create SPF records for all of my domains, especially the domains that I have not intention of sending email from. A simple SPF record that says no email server anywhere has permission to send any email for this domain is way better then just ignoring the domain completely.

I personally think that SPF might be too complicated to implement on the email server side. I’d like to see a few more open source implementations of the full SPF specification, maybe I’m not looking hard enough or thinking it through well enough, but I had a hard time implementing SPF into SFE.

Another thing I think is holding people back is creating the SPF record itself. I stopped using Network Solutions completely because they did not provide a way to implement SPF records on their hosted DNS service.

Pobox.com has a great tool to help you create an SPF record at http://old.openspf.org/wizard.html which you can place into your DNS records after you know what you want the SPF record to say and I know that GoDaddy.com has an SPF wizard in their hosted DNS service as well.

I read an article today (that I already lost the link to) that was talking about how spammers are using SPF on their throw away domains and domain administrators are using SPF incorrectly. Their conclusion was that SPF or Sender ID was not a good technology for fighting spam.

Personally I think they didn’t get the point. SPF is one technology for fighting spam, not the only technology. If SPF can be used to filter out some email then it will work for what it is designed to do.

As for the people who don not have SPF configured properly or they having users who are not using the authorized server, how is this a problem with the technology. Greater adoption of SPF would eventually root out these problem, as domain admins get reports of problems from their users.

Right now I am getting one type of spam that is driving me crazy, spam from my own domain name that is not originating from my servers. SPF is the perfect technology for this category of spam, where RBLs and bayesian filters are better for other types of spam.

In the end, no one anti-spam technology is going to win the battle. But a toolkit of technologies that work together each solving a distinct part of the problem will stem the tide and again make email the killer app that is was.

To the past few days I’ve been getting more and more e-mail messages that are forging my own domain name to try and get a read them. So this is just a little thank you sent out to all the spammers who decided to forge my domain name, of which I am the only person that has an e-mail address, thinking that some random string of characters will get me to read the e-mail message.

Now of course this has been getting caught in my quarantine since the messages can’t get past the rest of the filters at spam free e-mail. A decent number of these spam and messages have been getting caught and placed in my bad list, although none of them have managed to make it into my actual e-mail in-box.

The reason I’m thanking spammers for this barrage of messages forging my own domain name is that I needed some inspiration of late, and they have now provided it. After looking at the messages I have come to the conclusion that they would have very easily failed in the SPF test. I had placed creating my own SPF filter on the back burner for while; I have almost everything in place for it except the actual logic to do the IP address checking.

Now thanks to a litany of messages to which I know for a fact have not come from my own domain name, or my e-mail servers for that matter I have been annoyed into action. It may not be today or tomorrow but definitely this has moved to the top of my priority list. So hopefully by the end of this week or next I will have my SPF filter in place.

I’ve been working on and off with creating an SPF module for SFE. I’ve always intended it to be one of the major parts of the spam analysing process, but the specification is more complicated then I care for at times, so I’ve been going pretty slow.

One of the things I have been doing is caching the SPF records for 24 hours so that I don’t have to do the DNS look-up every single time. I’ve written a few reports that are pretty interesting about SPF and how well it is being adopted by Internet users.

At this point only 22% of domains that I have seen send email through SFE have SPF records.

You can find the real-time reports at http://spf.spamfreeemail.com These reports are cached for 1 hour and they SPF records are updated every 24 hours and they do reflect the real data for SpamFreeEmail.com.

I’ll create new reports as I find the time. If you have any ideas on reports to create feel free to leave them in the comments.

For anyone who does not know, in SMTP the HELO and EHLO commands are how an SMTP server responds to a greeting from another SMTP server.

Over the past few days I have been monitoring my new SMTP server that I am building, Watching the commands as the come through and some RBL stuff as well. I’ve build some SPF tools and integrated the IP-to-Country data for the most part as well.

One of the things that I have been noticing have been the randomness of the HELO commands. The same IP address will give different HELO commands every time it connects. Not only that but the HELO command does not match up with any of the email addresses.

While this is not a sure sign of spam it surely is a red flag and something that I will be tracking into the future.

Might be an odd question for most people, but for the system administrator it is a thankless job that only gets harder.

So why wouldn’t every admin use every tool at their disposal?

Some might not know about the tools, but I don’t consider ignorance an excuse.

Others might be implementing new anti-spam protocols but their budgetary constraints and overly complex networks prevent them from doing so. These are more acceptable excuses, but they are still excuses.

Simple technologies, like SPF that require no more then 30 minutes of any administrators time to implement, but they can help reduce spam immeasurably.

I guess I’m on a bit of a rant at the moment, I’ve been developing a new anti-spam email server and I have started looking through the SPF logs. I’m going to start keeping track of this in more detail very soon, but some extremely blatant spam has been coming through and it is obviously not originating from the networks they say they are.

In fact, I am pretty sure this particular case is a virus, but this is definitely something that could be stopped cold in it’s tracks with a few properly configured DNS records and a little more effort on the part of system admins as a whole.

Wonder what will make them take action to prevent the problems in their own houses, instead of just filing the complaints and doing nothing ….

While I currently do not have the answer to that question, that is one question I will be able to answer when I get this new anti-spam system up and running

While working on the core engine of this new service I’m creating I realized that the ability to block any email messages based on country of origin was going to be beneficial.

I had already developed scripts to automatically update an IP-to-country data for my IP database, so modifying it into a system that will be able to be part of the profile for each email message should not be too difficult.

Then giving people the ability to see how many emails they get originating in a certain country and then the ability to block all emails from that country could help a lot.

White lists and SPF could also be set to override a blocked country, so that you could still get an emai] if the sender was in that country. This would be good for the idea of having one or two foreign vendors, but then wanting to block everything from the rest of the country.

In any case this is one of the ideas that I am going to be working to this new service …. once I get it going

I’m still working on setting up my development system. It’s taking longer than I wanted, partially because I keep leaving to go do real work

As I’ve been creating this system I’ve started to think about how I want my anti spam solution to be different. When it comes down to it I want two things that I have not seen anywhere else; flexibility and control. I want to be able to know every reason why any particular email message might get blocked as spam and I want the end users to be able to see those reasons as well.

To that end I have decided that each email message needs to collect some data about it and then once all the data is collected the email message will be processed as spam or not. This meta data about each email message will include RBL data, SPF data as well as test data on the content of the message. Each message that passes through the system will collect as much data as possible and it will go through every test, even if it is already considered to be spam.

The reason for this is to better identify and describe what a spam message is and to give the feed back to the users on why a message might give a false positive or pass through the system when it truly is spam.

In my time working with spam filters they have tended to operate as black boxes that give no data back to the end user. The data that some of them do give back is near useless in describing an email message’s properties. I want to solve this part of the problem and give the users the tools to create better spam filters.

If a user sees that a spam message got through the filters, but it might not have gotten through if a new RBL was added, then the user will have the ability to add that RBL.

I also want to create reports to give an idea of exactly what feature is most capable of generating the best combination of filters. Perhaps filters would give fewer false positives if more then 3 RBLs are triggered or an exact combination of two of them. At this point I don’t have that information, but I plan on creating a way to get it.

I also hope to release this information to the public in very generalized reports. Information like the best RBL and how many domains are really using SPF.

In any case, all of this is built on the idea of creating meta data to describe each email message as it passes through this system. So this will be a large potion of the core of this system, which will in the end give the users the flexibility and control that I know I want from my spam filters.

I was talking with a good friend of mine last night and he made comment about how the only way that spam will disappear is when the lawmakers get more spam than they can handle themselves that they make it illegal.

While I see his point, the fact that the Internet is a global network makes legislation impractical to solve such a large problem. In fact I would think it would have about as much success as the war on drugs.

Innovation is the correct direction to move in to resolve the issues at hand. Finding the flaws and fixing them as well as sincere blocking of messages and servers that are not keeping with the times.

Technologies like SPF and bayesian filters will make more of a difference on a global scale than any legislation that comes out of any single country.

The bottleneck is not currently technology, it is the implementation of technology. If more companies would implement SPF in their DNS records alone, not even adding an SPF component to their incoming email server, spam and viruses would be reduced.

This is not a battle to completely obliterate the enemy, this is a game that we need only play to a stalemate. Our email systems are in place for a reason; to communicate with others. Spammers are in business because they exploit our desire to communicate with others.

Reduce a spammer’s ability to communicate with the email users on your email server and you have effected his bottom line. Accomplish this without impacting your users ability to communicate with the people they want to communicate with and your users will be happy.

I did some quick SPF testing this morning and set the new MailToaster to fail messages when they do not come from the correct SPF designated servers. I’ve also got SPF setup on my major domain names now, including:

* spamfreeemail.com
* simpleenigma.com
* homelocator.com
* spiderhunter.com

and several more domains.

I usually get several PayPal.com scam messages a week, so we’ll see how well SPF work by how many of these I will get over the next week or so…