Spam Free Email

Current Project

simpleenigma — Mon, 24 Dec 2007 04:29:15 +0000

I know I haven’t posted for a long time on this site, but I have been busy with other email related activities and I wasn’t sure if this site was point people to where I am currently working on these projects.

I’ve been working on creating an open source email server in Erlang and I’ve been posting my activity over at http://erlsoft.org

Most of the project has been taking a lot of what I learned form SFE and making it available to anyone who wants to use the code. The project is also up on Google Code at http://erlmail.googlecode.com

The SMTP server it pretty stable at this point and I’m spending most of my time on the IMAP modules, mostly on the server. I’ll put together a POP server once I have the IMAP stuff working.

I recently created an anti-spam module that is mostly some hooks into the SMTP server that will allow for functions to be run on each email message before and after is have been save to the message store. The hooks I have in place would allow me to re-create SFE within ErlMail without having to recompile the ErlMail application, the only thing I would need to do is list the appropriate modules and functions in a config file and the next time the server restarts those checks would be active.

This approach would allow me to create a complete open source email server and still have the SFE components private is I choose to.

I am also designing a modular message store into the ErlMail server to allow for separate message store for different purposes. This would allow me to keep one message store fore the SFE servers and a separate message store for the email server that is access by the users on the same cluster of Erlang servers. This is a design specification that came directly from my work on the SFE project.

In the end, the ErlMail project would allow me to create the SFE project using the ErlMail server and most likely less then 1,000 lines of additional code, since all the real work is being done by ErlMail

SPF Creation tool

simpleenigma — Sat, 09 Jun 2007 00:33:09 +0000

I’ve been talking to people about SPF over the past few weeks. In those conversations most people who are running email servers don’t think SPF is working. In many cases they believe that SPF is being used more by spammers then it is by people who are protecting themselves from spam and I can’t argue with either of those points.

The simple fact that a domain has a SPF record does not mean the domain will never send spam. The idea is that once SPF records are in wide use we will know where the spam comes from more reliably then we do now.

I personally create SPF records for all of my domains, especially the domains that I have not intention of sending email from. A simple SPF record that says no email server anywhere has permission to send any email for this domain is way better then just ignoring the domain completely.

I personally think that SPF might be too complicated to implement on the email server side. I’d like to see a few more open source implementations of the full SPF specification, maybe I’m not looking hard enough or thinking it through well enough, but I had a hard time implementing SPF into SFE.

Another thing I think is holding people back is creating the SPF record itself. I stopped using Network Solutions completely because they did not provide a way to implement SPF records on their hosted DNS service.

Pobox.com has a great tool to help you create an SPF record at http://old.openspf.org/wizard.html which you can place into your DNS records after you know what you want the SPF record to say and I know that GoDaddy.com has an SPF wizard in their hosted DNS service as well.

Taking SFE beta down for a while

simpleenigma — Fri, 02 Mar 2007 15:49:01 +0000

I have had my own email running through the SpamFreeEmail.com system for well over a year now and it has been working great. I’ve watched it learn about my email and automatically start forwarding mail that was good, while I never once got an email message sent through the system that was bad.

I had a few false positives in that some email was marked as spam when it was not, but those were easily enough fixed in the system.

I consider the BETA a success and a full working prototype of what an Erlang based spam filter could be, but it’s not enough

After spending a lot of time on the Netflix Prize, which I was working on in pure Erlang, I have learned so much about the language that I wanted to redesign how SFE was working. Unfortunately for me, I did not have enough hardware to run my beta test and redesign the system at the same time. At least not enough hardware for myself to feel comfortable.

I also have been wanting to implement some of the modules I have been working on in my ErlMail project, which has now become part of the work I am doing at http://erlsoft.org, which involves rewriting most core Internet servers in a way that they fully integrate with each other.

Lastly, I want to redesign SFE to be written in pure Erlang. I had been suing MySQL as a storage engine for most of the data and I truly believe the interaction between the two was slowing the system down. They were communicating perfectly, but the MySQL server was having trouble performing the intense actions I was trying to do.

So the SFE BETA will be down for a while. It will be back up in a largely rebuilt and restructured form and will be better for it

Erlang’s binary matching rocks!

simpleenigma — Thu, 11 Jan 2007 01:23:00 +0000

Okay, so for the past few days I’ve been playing with some binary data that I wasn’t looking forward to. It wasn’t the data so much as the last time I dealt this this particular type of data was in LISP and I was not happy with the results. In fact that experience was one of the major ones that got me to switch from LISP to Erlang.

When I started working with the binary data in Erlang I went back and re-read the documentation. It’s amazing how much easier the documentation is to read after working with Erlang for hundreds of hours, but I digress. After looking at the binary matching syntax I began to realize exactly how powerful it is and the project that I was working on was up and running in about 3 hours instead of my LISP attempt where I gave up on it after 3 days.

Then I started to realize where I could use binary pattern matching in my other application, specifically any where I use IP addresses. I often store IP addresses in their 32 bit decimal notation where the IP address 192.168.1.1 would turn out to be 3232235777. It’s easier to store and sort in databases that way, but how to convert from an IP Address tuple into an IP decimal.

With pattern matching it’s simple:

IPTuple = {192,168,1,1},
IPList = tuple_to_list(IPTuple),
IPBinary = list_to_binary(IPList),
<> = IPBinary,
IPDecimal.

believe me I was going through much more work to get this done when I wrote the original code for Spam Free Email to do this. I had no idea that I could do the whole thing with built in functions and I can’t wait to rewrite that section of code.

If you are doing anything with binary, or even if you are looking for simpler ways to do binary math, you really need to take a close look at Erlang’s binary pattern matching … really.

Update:

Joe Armstrong emailed me with a better version of the IP to decimal conversation. It uses more of the native binary pattern matching and I like it better then the one I wrote. Here it is:

IPTuple = {192,168,2,3},
{A,B,C,D} = IPTuple,
<> = <>

Thanks Joe!

IMAP Client extensions

simpleenigma — Thu, 04 Jan 2007 22:19:00 +0000

Between last night and this morning I started to add some extensions to the IMAP client, mostly due the fact that I wanted to take advantage of my existing IMAP server and use it to sort the messages before my web-mail application gets them. So I started with the SORT command, I think this is still in the proposal stages, but it seems to be quite mature and easily implemented.

SORT is really an extension of the SEARCH command, allowing to to search the messages and then return them in a particular order. In the IMAP client the way the code currently works I took the time to check the capabilities and make sure that SORT is listed, if not it then defaults to SEARCH. That way you still get your results, they just aren’t in any defined order.

Along with the SORT command the THREAD command is defined. THREAD=REFERENCES looks most promising as it appears to be the way that Google’s Gmail is able to group email messages into conversations. Now that I know this I am planning on implementing this and adding a threaded email view as an option in my web-mail client.

I’m going to be looking into the other extensions that my existing IMAP server handles and deciding if I wish to implement them now or later. It will depend on how useful I think the extension is and how long I think it will take to implement it.

ErlMail status

simpleenigma — Sun, 31 Dec 2006 23:42:00 +0000

I’ve been slowly working the ErlMail during the past few weeks. Some client work and the holidays have made it more difficult to carve out programming time then I would have liked.

I’ve been reading many different RFCs and I have been starting to build some leex/yacc grammar files to parse the different types of content with. The process is working out better then I hoped in many cases, but I’m still running into trouble.

The biggest complaint I have at this point is the fact that many people simply don’t follow the RFCs. There are some instances that are innocent enough, but I’m getting annoyed with dates that look nothing like RFC822 dates or have information in then that was never intended to be part of a date field.

In one instance I found that a mail server fixed an error of omission from a client, making sure that a date was in the headers and then put “(added by postmaster@somedomain.com)”. I haven’t re-checked the RFCs, but I’m pretty sure that is blatantly wrong.

How exactly is someone suppose to write a RFC compliant mail client when they know for certain that they will inevitably receive mail that is not RFC compliant and no number of SOMEs, SHOULDs or MIGHTS will prepare you for the array of bizarre interpretations that people seem to have about what is meant in the RFCs.

This just re-enforces the amazement that I have in the fact that email even works. This system was never designed to handle the traffic, or frankly the content, that is does and the number of successful email messages that travel across the Internet every day is mid boggling.

If they only knew it was smoke and mirrors help together with bailing wire and string ….

Parsing Dates

simpleenigma — Tue, 19 Dec 2006 00:52:00 +0000

I’ve been testing erlmail-0.0.2 and I came across some problems in my date parsing code. In reality the problems are in the util package that erlmail is dependant on, but date parsing is the problem none the less …

I decided that the randomness of RFC822 dates was worth another leex/yecc proejct, which I have always intended to do with many of the mail based RFCs. So I started the RFC822 scanner/parser which can now handle dates and dates only. It handles them much better then my custom code and once again proves that leex and yecc are better for parsing then I am.

ErlMail-0.0.2 Release

simpleenigma — Fri, 15 Dec 2006 04:50:00 +0000

I just finished porting the SMTP client I had built into SpamFreeEmail.com into ErlMail. I had forgotten how much easier SMTP is then IMAP, I’m talking orders of magnitude easier.

In any case, I had a functional API and FSM that I was using in SFE, so I upgraded it to reflect my current knowledge level and we now have http://www.spamfreeemail.com/releases/erlmail/erlmail-0.0.2.tar.gz

I also added dnsbl.erl, which is the core code for a DNS black-hole list checking module. Technically it works, it just has no consequences towards anything at this point.

Erlmail still requires my util package to work:

Please direct any comment, questions or patches to sjackson@simpleenigma.com and put ErlMail in the subject line. My Email is aggressively filtered for spam, so if I don’t see something to catch my attention, I delete it.

ErlMail-0.0.1 released

simpleenigma — Wed, 13 Dec 2006 18:28:00 +0000

This is the initial release of ErlMail, the only functional part of ErlMail at the moment is the IMAP client. Most of the rest of the file sin this distribution are meant as framework for future improvements.

There is enough documentation in the imapc.erl file for most (Erlang) people to figure out how to use it.

I have not implemented the AUTHENTICATE or STARTTLS commands yet. There is some framework in place for them, but they are not functional at all. STARTTLS will crash the system at present

I have used some of my own utilities in ErlMail, they are in a separate package I an also releasing today. I may decide to remove these dependencies in the future, but the make my life easier as of today

So here they are:

http://www.spamfreeemail.com/releases/erlmail/erlmail-0.0.1.tar.gz
http://www.spamfreeemail.com/releases/util/util-0.0.1.tar.gz

[UPDATE: I fixed a quick bug or two in the last code I was working on and re-uploaded the file at about 4PM PST]

Back to FETCH … again

simpleenigma — Tue, 12 Dec 2006 00:10:00 +0000

I’ve been working feverishly to develop leex and yecc grammar files to parse IMAP command and responses and it’s going wonderfully.

I’ve anaged to get through all of the commands that I had previously parsed by hand and then got to the FETCH command … again …

This time the parsing of the syntax has been a dream and I finally understand how to read the BNF in the back of the IMAP RFC.

I’ve also developed a new appreciation for the IMAP protocol. I use think that the protocol was oddly obscure and formatted simply to make it difficult to parse, but now I understand that IMAP was developed around it’s BNF. Which when you are using yacc/yecc to create a grammar file for it make it a wonderful protocol to work with.

I recently had to go back into my leex file and add some code to pre-process the fetch command. Basically I found a regular expression and then added some double quotes in the appropriate places to Ame the fetch command look more like strings then individual tokens. It works great and now I can get back to the yecc grammar file.

After that pre-processing sidetrak the rest of translating the BNF into the yecc grammar file should be a breeze, now I just wish I had the time to finish it tonight