Spam Free Email

I’ve been thinking about this for a while now and I’m pretty sure that I am going to try and offer a FREE anti-spam sort/process/forward system.

Basically the concept will be that you send your email to my mail servers by specifying that my mail servers are the highest priority in your DNS settings. Then my mail servers know that your email server is really the destination of your email and will forward the email to your server after processing it to remove spam and viruses.

I am planning on having a web based service for administrators and users to log in and adjust their own settings and check through any logs to see if they need to let any caught email through the system.

This is going to take a lot of time and resources for me to get setup, but this is one of my current goals. Any references in my blogs to ‘The Perfect Mail Server’ will be design elements that I am planning on writing into the system.

I’m not sure if I mentioned it here or somewhere else, or frankly at all, but I think and RBL style list to notify people when a virus has been detected from certain IP Addresses could be helpful.

To that point I’m adding this into my list of things I want to write into my mythical perfect email server.

Here is the basic concept:

Using an open source Anti-Virus software package, ClamAV for instance, a script could be easily written to scan emails for viruses and then when a virus is detected to log the IP address that the connection came in from into a database.

That database could then be exported into a series of RBL style DNS servers. The different RBL DNS servers would be different lengths of time that it has been since a virus had been seem from that IP Address. Link 1 day, 3 days 1 week, 2 weeks and maybe even a month, but I think that might be one hell of a penalty box.

IF the script were written properly and placed in the correct order of the server then it could easily keep finding and detecting viruses even if it knew that viruses had come from that IP before. So if a particular IP address sends a virus to this mythical email server once every 23 hours then that IP address would forever be on the 1 day virus list.

By sharing the information about which IP Addresses are sending known viruses it could reduce the chances of being infected yourself, but more importantly it would reduce the chances of others being infected.

Also, the IP Addresses that would end up on this list on a long term basis would not be real email servers. They would tend to be people’s home computers on cable modems and DSL lines. These are the computers most likely to get infected and they are the least likely to ever be scanned or cleaned.

This means that this RBL anti-virus list would have less of a possibility of blocking real email messages then most other RBL lists.

I was wandering around the ‘net yesterday looking for more topics to write about and I ran across this great set of articles by Paul Graham. http://www.paulgraham.com/antispam.html

The basic principle is to use Bayesian Filters to evaluate spam messages based on data sets of good messages and spam messages. Since each message is evaluated based on a data set of the actual users email, and not some general catch, all the statistics tend to allow for things to come through that the user would allow and some other people may not.

I tried this approach on a cache of messages that I keep around for analysis and I was quite impressed with the outcome. I would definitely need different tools to accomplish this on a wider scale then I did yesterday, but the basic principles held true.

The most interesting part of this for me is the idea that this would be customized for each and every user. I have several client sites that I often talk about spam and the word ‘mortgage’ has come up in the conversations several times. One of the clients is in the sales field and taking any message with the word ‘mortgage’ and deleting it does not impact this business. Another client of mine is the financial sector and removing that word would be an obvious mistake.

This the Bayesian Filter approach users that never get a good mail message with the word ‘mortgage’ in it will give a very high spam probability to any message with that word. Users that have both good and bad messages with the word in it will have an average or neutral rating and the more good messages with the word in it the better the rating will be.

Also, since this article advocates only looking at the 15 to 20 more ‘interesting’ words, meaning the 15 to 20 words that the score is either the highest or lowest, any neutral words will not be evaluated in the message.

Took me 3 or 4 times of reading the article and doing some test programming to completely understand the concepts, but this is a solid concept and I will be adding this into my anti-spam arsenal as soon as I can find the right approach.

Apparently I’m on an ethics kick today. I just did another blog entry at http://www.spiderhunter.com on the ethic of cloaking, so I’m going to continue the theme over here.

Bots and spider have been going through the Internet pretty much since it’s inception. The idea behind them is to allow them to collect the data on the Internet and provide a service using that data. The most notable services are search engines, but some link validators and intellectual property spiders are wonderful tools as well. Many bots just simply gather information from one website to present it on another, presumably with permission. I even have bots that collect data from my own websites to give me an overview of what is going on with my services.

But as with every technology it is the use of the tool that determines if it is good or bad. Email harvesters are a bot designed to collect email addresses and return them to individuals, typically spammers. Many of the latest generation of viruses have their own built in email servers. No need for them to use the other resources on the network when the can carry the resources around with themselves.

Bottom line is that every technology that can be created by the human imagination can be used equally to benifit humanity or not. It is in how you use the technology that determines the kind of person you are and that is a decision for you.

Okay, so I’ve been thinking about writing my own mail server. Of course it would be the perfect mail server. As things stand now, qmail is the closest thing I know of to the perfect mail server, but I still think it need improvement.

First off, I would like to see cross platform capabilities. To do this right, I say you need to do the server in JAVA. And, of course I think this needs to be open source.

Next it needs to have some open APIs for adding anti-virus and anti-spam solutions. Preferably once the message comes in and is validated it would be sent to a processing engine where it could be processed against a set of rules or programs that are not necessarily internal to the email server.

I would like to see all the configuration files in XML and/or available to an SQL server, anything that has ODBC drivers for that platform. The XML files would be for use in a stand alone environment, while the SQL configuration would allow for web access on the front end.

Also, all of the configuration would need to be on a domain by domain basis. Allowing for certain settings to be set for some domains and not others, instead of all being server wide. This would let certain domains start using protocols like SPF before others and to let some of them have more aggressive settings then others. With all of these settings on a SQL server, these settings could easily be exposed to a web based front end that would administrators for the domain control the details of how their domain email works.

All email message would need to be individual files. I think I would end up adopting the Maildir format from qmail. It’s strong and solid and the best improvement to the concepts of the old MHS mail from novell.

Of course POP3 and IMAP access would be needed.

And setting be default will need to be set to a high security environment. Meaning to chances for an open relay out of the box.

More to come, just jotting down some thoughts….

Last week I was adding some rules to my firewall and thought I’d clean some stuff up at the same time. I saw a port or two that I didn’t think I was using anymore and so I cleaned them up, only to get a complaint 3 days later that the FTP download access from one of my sites was no longer working.

Moral: Always check before you tighten security too much

Okay, now that I’ve told everyone that they should secure their network so that people can’t hack in there is some fessing up to do

About two weeks ago I was going to a friends house, I called as I was getting close and told them I’d be there soon. They promptly told me that they wouldn’t be back home for an hour. So I decided to have lunch and turned on my laptop. Sitting in the parking lot across from my friends home I was able to access 5 different WiFi networks.

So after going and getting some mexican food, I went back to my car and surfed the ‘net till my friend got home.

Bottom line is that if everyone did their own part and secured their own networks half of the issues the bog down the Internet would go away. Using some simple principles and tools you can lock down your network so that only the most experienced hackers can get into your network.

Some simple things that will help are to make sure you have closed your email server so that it is not an open relay. If you need to have access from the outside work to your email server setup up some type of authentication or change the incoming port for your SMTP server.

If you are concerned about security then make sure to use WEP on your wireless router. WEP can be broken and hacked, but the effort is usually not worth it. If you don’t do this then many resources on your network are just ripe for people to use, including any email servers. If you have a closed relay and someone can attack to your network via a wireless connection, they are inside your network and can email anything they want.

If you are providing free wireless Internet service (on purpose) then you might want to see if there is a way to disable port 25 on your equipment. This is the primary port for SMTP email and this will make it so that no one can send spam from your wireless network.

Make sure to put passwords on everything. I’m not a huge advocate of making people change their passwords or anything, but leaving things wide open is an invitation for disaster.

Check for service packs and patches, you should do this every once in a while. I try to do this at least once a quarter.

Only allow access to port that you are currently using. Every open port is an invitation to try to hack it. If you only have access on the port that you are using then you only have to worry about the things that are necessary to make you network work.