Spam Free Email - Anti-spam ideas, tools and services

August 2008
M	T	W	T	F	S	S
« Dec
	1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

Taking SFE beta down for a while

By simpleenigma

I have had my own email running through the SpamFreeEmail.com system for well over a year now and it has been working great. I’ve watched it learn about my email and automatically start forwarding mail that was good, while I never once got an email message sent through the system that was bad.

I had a few false positives in that some email was marked as spam when it was not, but those were easily enough fixed in the system.

I consider the BETA a success and a full working prototype of what an Erlang based spam filter could be, but it’s not enough

After spending a lot of time on the Netflix Prize, which I was working on in pure Erlang, I have learned so much about the language that I wanted to redesign how SFE was working. Unfortunately for me, I did not have enough hardware to run my beta test and redesign the system at the same time. At least not enough hardware for myself to feel comfortable.

I also have been wanting to implement some of the modules I have been working on in my ErlMail project, which has now become part of the work I am doing at http://erlsoft.org, which involves rewriting most core Internet servers in a way that they fully integrate with each other.

Lastly, I want to redesign SFE to be written in pure Erlang. I had been suing MySQL as a storage engine for most of the data and I truly believe the interaction between the two was slowing the system down. They were communicating perfectly, but the MySQL server was having trouble performing the intense actions I was trying to do.

So the SFE BETA will be down for a while. It will be back up in a largely rebuilt and restructured form and will be better for it

Posted in Erlang, Spam Free Email, ErlMail | 3 Comments

ErlMail status

By simpleenigma

I’ve been slowly working the ErlMail during the past few weeks. Some client work and the holidays have made it more difficult to carve out programming time then I would have liked.

I’ve been reading many different RFCs and I have been starting to build some leex/yacc grammar files to parse the different types of content with. The process is working out better then I hoped in many cases, but I’m still running into trouble.

The biggest complaint I have at this point is the fact that many people simply don’t follow the RFCs. There are some instances that are innocent enough, but I’m getting annoyed with dates that look nothing like RFC822 dates or have information in then that was never intended to be part of a date field.

In one instance I found that a mail server fixed an error of omission from a client, making sure that a date was in the headers and then put “(added by postmaster@somedomain.com)”. I haven’t re-checked the RFCs, but I’m pretty sure that is blatantly wrong.

How exactly is someone suppose to write a RFC compliant mail client when they know for certain that they will inevitably receive mail that is not RFC compliant and no number of SOMEs, SHOULDs or MIGHTS will prepare you for the array of bizarre interpretations that people seem to have about what is meant in the RFCs.

This just re-enforces the amazement that I have in the fact that email even works. This system was never designed to handle the traffic, or frankly the content, that is does and the number of successful email messages that travel across the Internet every day is mid boggling.

If they only knew it was smoke and mirrors help together with bailing wire and string ….

Posted in Erlang, ErlMail | No Comments

Parsing Dates

By simpleenigma

I’ve been testing erlmail-0.0.2 and I came across some problems in my date parsing code. In reality the problems are in the util package that erlmail is dependant on, but date parsing is the problem none the less …

I decided that the randomness of RFC822 dates was worth another leex/yecc proejct, which I have always intended to do with many of the mail based RFCs. So I started the RFC822 scanner/parser which can now handle dates and dates only. It handles them much better then my custom code and once again proves that leex and yecc are better for parsing then I am.

Posted in Erlang, ErlMail | No Comments

ErlMail-0.0.2 Release

By simpleenigma

I just finished porting the SMTP client I had built into SpamFreeEmail.com into ErlMail. I had forgotten how much easier SMTP is then IMAP, I’m talking orders of magnitude easier.

In any case, I had a functional API and FSM that I was using in SFE, so I upgraded it to reflect my current knowledge level and we now have http://www.spamfreeemail.com/releases/erlmail/erlmail-0.0.2.tar.gz

I also added dnsbl.erl, which is the core code for a DNS black-hole list checking module. Technically it works, it just has no consequences towards anything at this point.

Erlmail still requires my util package to work:

Please direct any comment, questions or patches to sjackson@simpleenigma.com and put ErlMail in the subject line. My Email is aggressively filtered for spam, so if I don’t see something to catch my attention, I delete it.

Posted in Erlang, ErlMail | No Comments

ErlMail-0.0.1 released

By simpleenigma

This is the initial release of ErlMail, the only functional part of ErlMail at the moment is the IMAP client. Most of the rest of the file sin this distribution are meant as framework for future improvements.

There is enough documentation in the imapc.erl file for most (Erlang) people to figure out how to use it.

I have not implemented the AUTHENTICATE or STARTTLS commands yet. There is some framework in place for them, but they are not functional at all. STARTTLS will crash the system at present

I have used some of my own utilities in ErlMail, they are in a separate package I an also releasing today. I may decide to remove these dependencies in the future, but the make my life easier as of today

So here they are:

http://www.spamfreeemail.com/releases/erlmail/erlmail-0.0.1.tar.gz
http://www.spamfreeemail.com/releases/util/util-0.0.1.tar.gz

[UPDATE: I fixed a quick bug or two in the last code I was working on and re-uploaded the file at about 4PM PST]

Posted in Erlang, ErlMail | No Comments

Erlang IMAP client

By simpleenigma

For most of this week I have been working on an IMAP client written in Erlang to be part of the http://www.erlmail.com project and I’m happy to say that I am almost finished with it.

I have two or three commands to finish before I start doing some real world testing and two or three commands that I am not planning on implementing at this time. I haven’t decided if I need the search command for what I am planning on doing next.

I am planning on writing an Erlang based web-mail application on top of YAWS. to my knowledge this would be the first 100% Erlang web-mail application. This will be one of a series of web applications I am writing based on YAWS and the third time I have written a web-mail client. The previous two time I wrote them in Cold Fusion.

If the speed of the IMAP client that I have been working with is any indication of the performance of the web-mail application I will be working on (hopefully before the weekend) I expect it to be amazingly fast with moderate hardware.

Once I get the major features working and the bugs worked out I plan on releasing the very first build of http://www.erlmail.com. This will be Build 0.0.1 and the IMAP client will be the only component that will be working

Posted in Erlang, IMAP, ErlMail | No Comments

ERLMail.com

By simpleenigma

Recently I registered the domain name http://www.erlmail.com with the intention of using it to create a free open source email server for Erlang.

Currently it is pointing to SpamFreeEmail.com, mostly because this will be the original code base for the project.

My intention is to create a complete SMTP compliant server with IMAP4 and POP3 support. I want it to be easily distributed and highly concurrent, hence the choice of Erlang.

I have never even participated in an open source project before, so running on will be quite a learning curve for me

I have the core of the SMTP server and much of the queuing processes already designed for SFE. A decent amount of code will need to be removed from SFE as it pertains exclusively to how I have my system configured, although some of it may simply get put into configuration files.

I’m hoping that I can get the core of a server up and running within a month or so. I still need to continue developing SFE, but I think developing them concurrently will improve them both.

So if you are interested in help, let me know. If you are interested in using the final project, let me know. Either way the more interest in the project the better and faster it will get done.

Posted in Spam Free Email, ErlMail | No Comments

Push vs Pull

By simpleenigma

Over the past two days I’ve been experimenting with two distinctly different ways to handle message queues. Basically the easiest way for me to describe them is a Push vs Pull concept.

Originally I designed the system with a Push method. I would get a new message and I would push it through the profilers and then out the client.

While this method was working, it had some problems when I tried pushing to a server that went down. So I’d have to re-profile and then that became a problem when I was re-profiling the same message on several machines. I was also ending up with multiple copies of the good messages being sent out when they were completed.

The Pull method that I have switched to I designed a throttle so that only one server can pull one section at a time and all servers check the queue every 1 to 5 seconds. This makes sure that each server is only processing one thing at a time and all servers are working on something different.

I have a lot of code from the push method still in place, so some of the system looks redundant at times, but in reality this make the system more fault tolerant.

All in all I am much happier with the current pull system I have implemented.

Posted in Spam Free Email, ErlMail | No Comments