Spam Free Email

Ever since I finished the telnet module to give me real time stats about the system I’ve been puzzled. I was watching message as they went through the system and counting. I counted to 15 or so and the system was telling me they took 50 seconds.

Finally I saw one that I counted to 30 and it told me 4,000 seconds.

Turns out I had a glitch in my math that I was using to generate my stats, or rather my SQL statement that was doing the math to generate my stats. Because of that my average time literally went from 280 seconds for each message to 89 seconds and it’s still going down slowly.

So what that means is that for the past week I’ve been trying to optimize the system with bad numbers. Which explains why things that I thought would help didn’t do as well as I thought they would.

But now, officially I think I have the back-end optimized the way it needs to be and I’ll be working on adding more features again.

Over the past two days I’ve been experimenting with two distinctly different ways to handle message queues. Basically the easiest way for me to describe them is a Push vs Pull concept.

Originally I designed the system with a Push method. I would get a new message and I would push it through the profilers and then out the client.

While this method was working, it had some problems when I tried pushing to a server that went down. So I’d have to re-profile and then that became a problem when I was re-profiling the same message on several machines. I was also ending up with multiple copies of the good messages being sent out when they were completed.

The Pull method that I have switched to I designed a throttle so that only one server can pull one section at a time and all servers check the queue every 1 to 5 seconds. This makes sure that each server is only processing one thing at a time and all servers are working on something different.

I have a lot of code from the push method still in place, so some of the system looks redundant at times, but in reality this make the system more fault tolerant.

All in all I am much happier with the current pull system I have implemented.