It seems that whenever I manage to get the system faster on one component another bottleneck appears to take it’s place. I suspect that is what beta testing is for ….
The latest bottleneck was a needed optimization. While designing the system I had a function that linked each work in a message to that message in the database. Originally I had it setup to use 1 SQL statement each time I linked a word to that message.
This morning when I woke up I was noticing that the system fell behind again, and like many times before it looked as if I was overloading my MySQL server.
I wrote a few quick functions to clear up some obvious problems, which didn’t solve anything, then I got to the part where it was slowing down, which was where I was linking the words to the message.
In MySQL you can put multiple inserts into one statement, I’m not sure if this is a MySQL specific feature or part of the SQL standard, but I have changed the code so that it builds a large SQL insert statement and only execute one statement per message.
So in the case that a message has 100 words in it I just eliminate 99 SQL queries without removing any functionality.
Now it should be time to find a bottleneck on the Erlang side of things ….