Over the past week or so, since I added my own email to Spam Free Email, I’ve been optimizing the code to process the messages faster. It’s been quite and adventure.
I started to implement a lot of data in Mnesia tables, then I realized I was putting in too much data and I was updating it too often.
So I went to the source of the data, which is a MySQL server. I started optimizing the SQL statements to perform better and I found I was too limited there as well.
I ended up with a combination of SQL and Mnesia that is working rather well. The data is store in the MySQL server and it is retrieved when needed by Erlang into Mnesia. Once the data is no longer of use the Mnesia tables are cleared.
Certain Mnesia tables are prominent, since the data they hold never changes, but the more important ones are ram copies that are cleared on a regular basis.
I also ended up implementing a table that I am using as an SQL view, even though it is not a view. I put together a system that will update the view when things change, so that the data that is retrieved is always as fresh as possible.
After nearly a full week of working on this I have gotten my test message’s processing time down from 107 seconds (1 minute 47 seconds) to 6.5 seconds and the current average time is about 32 seconds.
Since I have 22 nodes currently that means that I can process 1 message every 1.45 seconds. Of course that is the a perfect world where statistics govern everything 