Since my crash about a week ago I have been rebuilding the system and trying to improve upon it in the process. I got the major things fixed and working by Thursday and last night I got the minor things fixed so that I am moving forward again instead of playing catchup.
In the process I have removed hundreds of lines of code that were duplicated code. Mostly I found ways of reducing several functions into one or two functions and give them more functionality most of the time as well.
I had also taken the time to remove many tables from my MySQL database. I reduced the number of tables almost in half by combining what was in several tables into a few of them. This has greatly improved the performance of the database and in come cases I have seen a 10 fold improvement. (Most of that is due to better indexes)
All of that makes sense to me when I am talking about making less into more, which I consider more messages being processes in less time the ultimate goal, but what is confusing me at the moment, is that less Erlang nodes are processing each message in less time.
Right now I have six nodes running and messages are precessing in 1 to 5 seconds. If I turn on all of my nodes, which is currently 50, the messages start to process in 50 to 200 seconds.
I’ve check the database code and it is working as streamlined as I can make it, but I think my current bottle neck is how I am using Mnesia.
I am using Mnesia to hold information about each message as it is being profile and the I throttle the nodes so that only a one node can be working on one process for each message. (There are currently 5 different processes that each node could be working on for each message.)
I’m beginning to think that I am passing more information between the node then they are able to handle, so with fewer node Mnesia is running better.
I’;m considering this a flaw in my own use of Mnesia, not in Mnesia itself. I think if I redesign the data I should be able to add all 50 nodes and make them work faster then just the 6 I have working now. The trick is figuring out the correct data structures in Mnesia to make it work.