For most of the month of January I have been working on optimization and performance issues concerning my anti-spam service. It was feeling like anything I was doing to improve performance was creating a bottleneck somewhere else, but I think I’m finally past that.
I have the profiler distributing the messages between all of the nodes in a weighted random round robin fashion. Then each part of the profiler runs concurrently on that machine.
Thanks to the native Erlang module for MySQL my communications with my database server have stabilized, mostly due to the connection pooling and the streamlining that the MySQL module does for the overall design.
Also thanks to the native Erlang module for MySQL I am again able to run the profiler on Windows workstations as well as my Linux servers. The only piece that does not currently run on Windows is the anti-virus and I have setup the code to send the anti-virus to another PC (hopefully Linux based) to have that part of the profiler completed. It’s fast so there is very little impact on anything.
I have one version of the profiler that distributes everything to different nodes instead of doing all the work concurrently on one node and the profiler evaluates the message with the same code no matter if it is distributed or concurrent. I might go back to the distributed design for testing, but last time it was slower then the concurrent model when I thought it should have been faster.
My statistics show that the messages are being processed in a reasonable amount of time (less then a minute for 90% of messages) depending on the size of the message. I might have some more optimization to do, but it’s time to move on to newer things.
I still have a decent amount of work to do on the front end before anyone else will be able to use it well, but I’m going to keep focusing on the major things on the back end for the next few days. I might do the front end over the weekend, it shouldn’t take me more then a few days anyway.
I’m hoping to still be ready to put a few more domains onto the system around the first week of February. If you think you might want to be part of the beta testing drop me a note and mention ”’SFE beta testing”’ in the subject line or comment on this tread.