Spam Free Email

Between last night and this morning I started to add some extensions to the IMAP client, mostly due the fact that I wanted to take advantage of my existing IMAP server and use it to sort the messages before my web-mail application gets them. So I started with the SORT command, I think this is still in the proposal stages, but it seems to be quite mature and easily implemented.

SORT is really an extension of the SEARCH command, allowing to to search the messages and then return them in a particular order. In the IMAP client the way the code currently works I took the time to check the capabilities and make sure that SORT is listed, if not it then defaults to SEARCH. That way you still get your results, they just aren’t in any defined order.

Along with the SORT command the THREAD command is defined. THREAD=REFERENCES looks most promising as it appears to be the way that Google’s Gmail is able to group email messages into conversations. Now that I know this I am planning on implementing this and adding a threaded email view as an option in my web-mail client.

I’m going to be looking into the other extensions that my existing IMAP server handles and deciding if I wish to implement them now or later. It will depend on how useful I think the extension is and how long I think it will take to implement it.

I’ve been working feverishly to develop leex and yecc grammar files to parse IMAP command and responses and it’s going wonderfully.

I’ve anaged to get through all of the commands that I had previously parsed by hand and then got to the FETCH command … again …

This time the parsing of the syntax has been a dream and I finally understand how to read the BNF in the back of the IMAP RFC.

I’ve also developed a new appreciation for the IMAP protocol. I use think that the protocol was oddly obscure and formatted simply to make it difficult to parse, but now I understand that IMAP was developed around it’s BNF. Which when you are using yacc/yecc to create a grammar file for it make it a wonderful protocol to work with.

I recently had to go back into my leex file and add some code to pre-process the fetch command. Basically I found a regular expression and then added some double quotes in the appropriate places to Ame the fetch command look more like strings then individual tokens. It works great and now I can get back to the yecc grammar file.

After that pre-processing sidetrak the rest of translating the BNF into the yecc grammar file should be a breeze, now I just wish I had the time to finish it tonight

I’ve gotten distracted again, but in a good way this time. I discovered what parsetools/yecc does recently and I figured it would be perfect to parse the commands and responses for an IMAP server and the IMAP client. I have proven to myself that this is true, but now I need to create a grammar file that fulfills the entire IMAP4 spec.

This is where leex comes in; as I need the data string that is either a command or a response to be tokenized into something that is reasonably easy to parse. I was workign with erl_scan:string/1, which was working adeqatly well, but after buying hte O’rielly book on Lex and Yacc I’ve come to the conclusion that leex needs to make a scanner for yecc to parse the tokens from.

I got the concepts of leex figured out last night just before I went to bed and ened up having regular expression and parsing dreams all night long. Annoying, but I solved a few of my problem that I knew about last night what I was dreaming

In any case this will slow down the release of ErlMail-0.0.1 for just a bit, but it will solve all of the bugs that I mentioned in an earlier post. Fair enough trade off in my book.

For the past few days I’ve been working on getting the IMAP client in ErlMail to work with a YAWS based web-mail application that I am working on. To that end there is some good news and some bad news

The good news is that it is working pretty well. It’s fast and the data is getting back to the web-mail client exactly how it is suppose to, or rather exactly how I told it to, which is part of the bad new.

I always find it amazing how different test data is from real life data Which is where the somewhat bad news comes in. Bugs, lots of bugs. Things I never would have thought of while working with it originally.

It doesn’t seems to like spaces in file names. I understand that one, but haven’t figured out how I want to fix it yet.

I made an assumption that one email address was coming through at a time, and now when an email message goes to more then one person it break, which will just require another layer of parsing on the envelope.

And my personal favorite of my current test data set, it doesn’t like extra parenthesis in the subject line. The parenthesis are used as a delimiter items in the envelope and it breaks when it seem more then it should. I have an idea of how to fix this, but nothing solid yet.

So the really great news is that I can display email messages from different mailboxes. I can get the subject, from, date, size and other general information. and I can display the detail of a message, save the actual body of the message so far.

After working out these bugs and being able to display the full message with potential attachment, I’ll need to start working on the SMTP client so that I can send email as well as view what I have received.

So the FETCH command in the IMAP client gave me some problems. Turns out this command is overly complicated and the syntax that the server return is extremely difficult to parse.

Maybe it was just me, but it did not make sense at first and while it still does not make sense I can now parse it into Erlang data structures.

I still have one or two sub-commands to iron out before I release anything and some bug testing to do as well, but the IMAP client is functional, so long as you know how to use it.

I’m going to start writing my web-mail client this week and hopefully that will find all the major bugs in the IMAP client and I can release it well before the first of the year.

For most of this week I have been working on an IMAP client written in Erlang to be part of the http://www.erlmail.com project and I’m happy to say that I am almost finished with it.

I have two or three commands to finish before I start doing some real world testing and two or three commands that I am not planning on implementing at this time. I haven’t decided if I need the search command for what I am planning on doing next.

I am planning on writing an Erlang based web-mail application on top of YAWS. to my knowledge this would be the first 100% Erlang web-mail application. This will be one of a series of web applications I am writing based on YAWS and the third time I have written a web-mail client. The previous two time I wrote them in Cold Fusion.

If the speed of the IMAP client that I have been working with is any indication of the performance of the web-mail application I will be working on (hopefully before the weekend) I expect it to be amazingly fast with moderate hardware.

Once I get the major features working and the bugs worked out I plan on releasing the very first build of http://www.erlmail.com. This will be Build 0.0.1 and the IMAP client will be the only component that will be working

Today I stared building an IMAP4rev1 client module in Erlang. It appears to be a good candidate for a finite state machine and I’m going to be building it in a non-blocking fashion, similar to a tutorial on http://www.trapexit.org .

I’m planning this to be a stand alone module that I can easily give out as open source and I’m also thinking it will be a component of the erlmail server that I am planning, someday.

In developing the SMTP server for SFE, I found it much easier to write after I had already developed the SMTP client. So I also hope that this inspires me to start working on the IMAP server not long after I finish up the client.

The IMAP client is going to be the center piece of a new web-based email program that I am writing. This will be the 4th or 5th time I have written a web-based email client and the second time that it involves IMAP, but the first time I am writing the IMAP client myself.

The last one I used was written in Java and I wrote a wrapper to use it in Cold Fusion. While it worked, it was clumsy and slow. In taking from that experience I have a few thoughts on how to cache data in the Erlang version that was not available to me in Cold Fusion.

In any case, I had better get back to reading RFCs if I’m going to get any work done on this.

I’ve been working on the front end for the SpamFreeEmail.com service that I have been writing and I think I have come up with an interesting idea for a front end interface.

Instead of making people log into a web based interface, why not give them a minimalIMAP interface. That way they could constantly monitor everything concerning their email from the same mail client.

Three Main folders would be needed; Good, Bad and Neutral, possibly a fourth for viruses. Moving mail from one folder to another would automatically reclassify it to the new designation. Marking any message with a Flag would forward it onto the destination mail server.

This idea is to simplify the users life and let them regain control over their communications. Giving them an easier interface to access their anti-spam system certainly qualifies for this.

Plus I’m already using the Maildir format with the basic flags setup for IMAP, the only thing I would need to do is write the IMAP interface and link it directly to the mail server, and at the speed I’ve been developing code in Erlang, that would be easy.

I’m going to slate this on my to-do list for sometime after my go live launch, but that should still be in the first half of this year.

I just finished rewriting the web based email software for SimpleApps so that it now supports an IMAP back-end.

I did this for a few reason. First and foremost I was trying to get anti-spam and anti-virus support for this domain. This is working now

Next I wanted to use a complete email system that was under current development. I choose the MailToaster and I’m quite happy with it.

I choose IMAP to solve a few problems which ended up creating a few of their own. Cold Fusion does not natively support IMAP, even though it uses JavaMail as the core of it’s email engine. JavaMail does support IMAP, so I wrote a Cold Fusion CFC as a wrapper for JavaMail to gain the access I needed. (I’ve been working on this, on and off, for about two weeks now)

I’m pretty happy with this so far and I think I’m going to be even happier when this web site is moved onto new servers. If anyone is interested in the Cold Fusion CFC for JavaMail let me know. I’d love to have others use it too, even if it is pretty raw at the moment.

I’ve been playing with the MailToaster for the better part of a week now. I’ve got one of them ready to go into a production environment at one of my client sites and a question is in my mind:

IMAP or POP3?

First off the users at this client site are use to POP3, and knowing the personalities this is why I will be setting things back up as POP3.

But what could IMAP give them that POP3 can’t?

The first thing I can think of is flexibility. With IMAP you can have all of your email on the server and you can use multiple computers to check the same email box. I do have one user that has been asking if this was possible, but I have been steering them away because of the way that they have been using POP3. Next time they ask I’ll have a better answer.

The next thing that I can think of is better integration with anti spam tools like SpamAssassin. When using IMAP, at least on the MailToaster, SpamAssassin has access to the spam folder. This allows SpamAssassin to train better for the user specific email patterns and gets rid of more spam that way.

I’ll come up with more reasons over time, but those two alone are making me think if I should bring this up to the client, or wait to train them again later ….