Spam Free Email - Anti-spam ideas, tools and services

August 2008
M	T	W	T	F	S	S
« Dec
	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Taking SFE beta down for a while

By simpleenigma

I have had my own email running through the SpamFreeEmail.com system for well over a year now and it has been working great. I’ve watched it learn about my email and automatically start forwarding mail that was good, while I never once got an email message sent through the system that was bad.

I had a few false positives in that some email was marked as spam when it was not, but those were easily enough fixed in the system.

I consider the BETA a success and a full working prototype of what an Erlang based spam filter could be, but it’s not enough

After spending a lot of time on the Netflix Prize, which I was working on in pure Erlang, I have learned so much about the language that I wanted to redesign how SFE was working. Unfortunately for me, I did not have enough hardware to run my beta test and redesign the system at the same time. At least not enough hardware for myself to feel comfortable.

I also have been wanting to implement some of the modules I have been working on in my ErlMail project, which has now become part of the work I am doing at http://erlsoft.org, which involves rewriting most core Internet servers in a way that they fully integrate with each other.

Lastly, I want to redesign SFE to be written in pure Erlang. I had been suing MySQL as a storage engine for most of the data and I truly believe the interaction between the two was slowing the system down. They were communicating perfectly, but the MySQL server was having trouble performing the intense actions I was trying to do.

So the SFE BETA will be down for a while. It will be back up in a largely rebuilt and restructured form and will be better for it

Posted in Erlang, Spam Free Email, ErlMail | 3 Comments

Erlang’s binary matching rocks!

By simpleenigma

Okay, so for the past few days I’ve been playing with some binary data that I wasn’t looking forward to. It wasn’t the data so much as the last time I dealt this this particular type of data was in LISP and I was not happy with the results. In fact that experience was one of the major ones that got me to switch from LISP to Erlang.

When I started working with the binary data in Erlang I went back and re-read the documentation. It’s amazing how much easier the documentation is to read after working with Erlang for hundreds of hours, but I digress. After looking at the binary matching syntax I began to realize exactly how powerful it is and the project that I was working on was up and running in about 3 hours instead of my LISP attempt where I gave up on it after 3 days.

Then I started to realize where I could use binary pattern matching in my other application, specifically any where I use IP addresses. I often store IP addresses in their 32 bit decimal notation where the IP address 192.168.1.1 would turn out to be 3232235777. It’s easier to store and sort in databases that way, but how to convert from an IP Address tuple into an IP decimal.

With pattern matching it’s simple:

IPTuple = {192,168,1,1},
IPList = tuple_to_list(IPTuple),
IPBinary = list_to_binary(IPList),
<<IPDecimal:32>> = IPBinary,
IPDecimal.

believe me I was going through much more work to get this done when I wrote the original code for Spam Free Email to do this. I had no idea that I could do the whole thing with built in functions and I can’t wait to rewrite that section of code.

If you are doing anything with binary, or even if you are looking for simpler ways to do binary math, you really need to take a close look at Erlang’s binary pattern matching … really.

Update:

Joe Armstrong emailed me with a better version of the IP to decimal conversation. It uses more of the native binary pattern matching and I like it better then the one I wrote. Here it is:

IPTuple = {192,168,2,3},
{A,B,C,D} = IPTuple,
<<IPDecimal:32>> = <<A:8,B:8,C:8,D:8>>

Thanks Joe!

Posted in Erlang | No Comments

IMAP Client extensions

By simpleenigma

Between last night and this morning I started to add some extensions to the IMAP client, mostly due the fact that I wanted to take advantage of my existing IMAP server and use it to sort the messages before my web-mail application gets them. So I started with the SORT command, I think this is still in the proposal stages, but it seems to be quite mature and easily implemented.

SORT is really an extension of the SEARCH command, allowing to to search the messages and then return them in a particular order. In the IMAP client the way the code currently works I took the time to check the capabilities and make sure that SORT is listed, if not it then defaults to SEARCH. That way you still get your results, they just aren’t in any defined order.

Along with the SORT command the THREAD command is defined. THREAD=REFERENCES looks most promising as it appears to be the way that Google’s Gmail is able to group email messages into conversations. Now that I know this I am planning on implementing this and adding a threaded email view as an option in my web-mail client.

I’m going to be looking into the other extensions that my existing IMAP server handles and deciding if I wish to implement them now or later. It will depend on how useful I think the extension is and how long I think it will take to implement it.

Posted in Erlang, IMAP | No Comments

ErlMail status

By simpleenigma

I’ve been slowly working the ErlMail during the past few weeks. Some client work and the holidays have made it more difficult to carve out programming time then I would have liked.

I’ve been reading many different RFCs and I have been starting to build some leex/yacc grammar files to parse the different types of content with. The process is working out better then I hoped in many cases, but I’m still running into trouble.

The biggest complaint I have at this point is the fact that many people simply don’t follow the RFCs. There are some instances that are innocent enough, but I’m getting annoyed with dates that look nothing like RFC822 dates or have information in then that was never intended to be part of a date field.

In one instance I found that a mail server fixed an error of omission from a client, making sure that a date was in the headers and then put “(added by postmaster@somedomain.com)”. I haven’t re-checked the RFCs, but I’m pretty sure that is blatantly wrong.

How exactly is someone suppose to write a RFC compliant mail client when they know for certain that they will inevitably receive mail that is not RFC compliant and no number of SOMEs, SHOULDs or MIGHTS will prepare you for the array of bizarre interpretations that people seem to have about what is meant in the RFCs.

This just re-enforces the amazement that I have in the fact that email even works. This system was never designed to handle the traffic, or frankly the content, that is does and the number of successful email messages that travel across the Internet every day is mid boggling.

If they only knew it was smoke and mirrors help together with bailing wire and string ….

Posted in Erlang, ErlMail | No Comments

Parsing Dates

By simpleenigma

I’ve been testing erlmail-0.0.2 and I came across some problems in my date parsing code. In reality the problems are in the util package that erlmail is dependant on, but date parsing is the problem none the less …

I decided that the randomness of RFC822 dates was worth another leex/yecc proejct, which I have always intended to do with many of the mail based RFCs. So I started the RFC822 scanner/parser which can now handle dates and dates only. It handles them much better then my custom code and once again proves that leex and yecc are better for parsing then I am.

Posted in Erlang, ErlMail | No Comments

ErlMail-0.0.2 Release

By simpleenigma

I just finished porting the SMTP client I had built into SpamFreeEmail.com into ErlMail. I had forgotten how much easier SMTP is then IMAP, I’m talking orders of magnitude easier.

In any case, I had a functional API and FSM that I was using in SFE, so I upgraded it to reflect my current knowledge level and we now have http://www.spamfreeemail.com/releases/erlmail/erlmail-0.0.2.tar.gz

I also added dnsbl.erl, which is the core code for a DNS black-hole list checking module. Technically it works, it just has no consequences towards anything at this point.

Erlmail still requires my util package to work:

Please direct any comment, questions or patches to sjackson@simpleenigma.com and put ErlMail in the subject line. My Email is aggressively filtered for spam, so if I don’t see something to catch my attention, I delete it.

Posted in Erlang, ErlMail | No Comments

ErlMail-0.0.1 released

By simpleenigma

This is the initial release of ErlMail, the only functional part of ErlMail at the moment is the IMAP client. Most of the rest of the file sin this distribution are meant as framework for future improvements.

There is enough documentation in the imapc.erl file for most (Erlang) people to figure out how to use it.

I have not implemented the AUTHENTICATE or STARTTLS commands yet. There is some framework in place for them, but they are not functional at all. STARTTLS will crash the system at present

I have used some of my own utilities in ErlMail, they are in a separate package I an also releasing today. I may decide to remove these dependencies in the future, but the make my life easier as of today

So here they are:

http://www.spamfreeemail.com/releases/erlmail/erlmail-0.0.1.tar.gz
http://www.spamfreeemail.com/releases/util/util-0.0.1.tar.gz

[UPDATE: I fixed a quick bug or two in the last code I was working on and re-uploaded the file at about 4PM PST]

Posted in Erlang, ErlMail | No Comments

Back to FETCH … again

By simpleenigma

I’ve been working feverishly to develop leex and yecc grammar files to parse IMAP command and responses and it’s going wonderfully.

I’ve anaged to get through all of the commands that I had previously parsed by hand and then got to the FETCH command … again …

This time the parsing of the syntax has been a dream and I finally understand how to read the BNF in the back of the IMAP RFC.

I’ve also developed a new appreciation for the IMAP protocol. I use think that the protocol was oddly obscure and formatted simply to make it difficult to parse, but now I understand that IMAP was developed around it’s BNF. Which when you are using yacc/yecc to create a grammar file for it make it a wonderful protocol to work with.

I recently had to go back into my leex file and add some code to pre-process the fetch command. Basically I found a regular expression and then added some double quotes in the appropriate places to Ame the fetch command look more like strings then individual tokens. It works great and now I can get back to the yecc grammar file.

After that pre-processing sidetrak the rest of translating the BNF into the yecc grammar file should be a breeze, now I just wish I had the time to finish it tonight

Posted in Erlang, IMAP | No Comments

Yecc, Leex and IMAP

By simpleenigma

I’ve gotten distracted again, but in a good way this time. I discovered what parsetools/yecc does recently and I figured it would be perfect to parse the commands and responses for an IMAP server and the IMAP client. I have proven to myself that this is true, but now I need to create a grammar file that fulfills the entire IMAP4 spec.

This is where leex comes in; as I need the data string that is either a command or a response to be tokenized into something that is reasonably easy to parse. I was workign with erl_scan:string/1, which was working adeqatly well, but after buying hte O’rielly book on Lex and Yacc I’ve come to the conclusion that leex needs to make a scanner for yecc to parse the tokens from.

I got the concepts of leex figured out last night just before I went to bed and ened up having regular expression and parsing dreams all night long. Annoying, but I solved a few of my problem that I knew about last night what I was dreaming

In any case this will slow down the release of ErlMail-0.0.1 for just a bit, but it will solve all of the bugs that I mentioned in an earlier post. Fair enough trade off in my book.

Posted in Erlang, IMAP | No Comments