Spam Free Email

I’ve spent about 30% of my time learning how Erlang programs are suppose to be designed using OTP and the only thing that I can take as a 100% fact is the documentation is not good enough.

I did find some great how-to documents at trapexit.org. While they still make some assumptions about how much the reader knows about erlang and the OTP their how-to’s are much easier to dissect and have much more documentation in the code. It’s almost inspired me enough to completely document my own code … almost …

In the end, once you’ve gotten OTP working in your Erlang programs it make things much simpler. You do end up creating a lot of code over and over again, but the error correction and supervision is well worth it in code that needs to have the maximum amount of up time.

While working on the SMTP server, I was noticing that the code was crashing 4 or 5 times a day. While I did not change the code, I did add OTP aspects and supervisors and the code started to run without any noticeable down time. While I have fixed the problems that I think were causing the crashes now, the OTP design handled the restarts effortlessly and let me get past the little things.

Right now I’m in the process of understanding how a full supervision tree with multiple worker processes is put together. Once I have that I’ll have a rock solid SMTP server with nearly 100% up-time.

While I am in the process of creating my own anti-spam email server I have been using two of my older domain names as test domains. I retired both domains on April 1st, 2005 and both domains had between 500 and 1,000 active using at the time I retired them.

It is now seven months later and these two domains are still getting spam like you wouldn’t believe.

As part of my server and the statistics that I am planning on keeping I started tracking the unique IP Addresses that are making connections to these two domains. The first day of tracking any I have nearly 1100 unique IP addresses that have made connections, and this is only since midnight.

Not only that but I often see the same IP address making multiple connections in a short period of time.

Of course this just going to and fuel to the argument that spam is a larger problem then people realize. Even those of us who complain about it constantly

Not too long ago I was working on making erlang connect via ODBC to my MySQL server. I spent 4 days installing and recompiling only to find out that my own erlang code was the problem, not erlang or the ODBC drivers. I figure I wasted 2 days on that, but in the process I really streamlined my installation process.

I now have a set of RPMs that lets me install a minimal version of Fedora Core 4, map a few NFS shares and run one script that installs ODBC, erlang, NTP and a few other things that are needed for everything to run. It also creates my configuration files and a few other settings as well.

From the point of booting off the install DVD to having a working erlang environment configured to communicate with my system it takes 15 to 20 minutes depending on the processor and the person doing the install.

Important stuff for anyone who hopes to be running 20+ erlang nodes within the next 6 months.

As part of this entire project I’ve wanted a good anti-virus solution that I can use to detect viruses in email messages as they go through the system. I’m not interested in using the big boys, because of licensing problems, so I’ve pretty much decided that use clamav.

As part of the licensing agreement to use the clamav libraries I will be releasing the clamav-to-erlang module into open source, or at least releasing my source code for others to use.

My basic idea for it’s implementation will be to pass the text of a file to clamav and get back the results which I will return in some erlang data structure.

Sound like I haven’t done much with it yet? … well I haven’t. It’s all still just a theory and nice thoughts in the back of my own head. Once I get the first rev of the anti-spam server functional (not necessarily finished) I’ll be looking into this further. Best I can tell it’s possible, just going to take a bit of time to make clamav and erlang play nice together

 http://www.clamav.net

Sorry I haven’t written in a bit, but I’ve been coding like a wild man

Just today I got my SMTPD application written in erlang to complete it’s first entire SMTP conversation, that I did not initiate from my test modules. This a huge step for the progress of this site and I’m totally jazzed by it …

While I would not call this an email server yet, this is my first step to that end.

I still need to perfect the processes that I’m going to use for my Queue and a module that I am calling a Profiler that is doing all of my anti-spam processes, of which I am including every technology I can think of to track and prevent spam.

After that it is simply a process of sending the good mail to the end user and removing the bad stuff … Sounds easy, eh?

For anyone who does not know, in SMTP the HELO and EHLO commands are how an SMTP server responds to a greeting from another SMTP server.

Over the past few days I have been monitoring my new SMTP server that I am building, Watching the commands as the come through and some RBL stuff as well. I’ve build some SPF tools and integrated the IP-to-Country data for the most part as well.

One of the things that I have been noticing have been the randomness of the HELO commands. The same IP address will give different HELO commands every time it connects. Not only that but the HELO command does not match up with any of the email addresses.

While this is not a sure sign of spam it surely is a red flag and something that I will be tracking into the future.

Might be an odd question for most people, but for the system administrator it is a thankless job that only gets harder.

So why wouldn’t every admin use every tool at their disposal?

Some might not know about the tools, but I don’t consider ignorance an excuse.

Others might be implementing new anti-spam protocols but their budgetary constraints and overly complex networks prevent them from doing so. These are more acceptable excuses, but they are still excuses.

Simple technologies, like SPF that require no more then 30 minutes of any administrators time to implement, but they can help reduce spam immeasurably.

I guess I’m on a bit of a rant at the moment, I’ve been developing a new anti-spam email server and I have started looking through the SPF logs. I’m going to start keeping track of this in more detail very soon, but some extremely blatant spam has been coming through and it is obviously not originating from the networks they say they are.

In fact, I am pretty sure this particular case is a virus, but this is definitely something that could be stopped cold in it’s tracks with a few properly configured DNS records and a little more effort on the part of system admins as a whole.

Wonder what will make them take action to prevent the problems in their own houses, instead of just filing the complaints and doing nothing ….