I was beginning to look over my shoulder to see if advancing glaciers were going to overrun me. It seemly has taken forever, but my rewrite last year of the Socorro processors has finally made it to production. The switchover happened on Thursday afternoon (while I was at the dentist) from the old processor (Processor2008) to the new processor (Processor2012).
The new processor was written as a Fetch-Transform-Save Configman app. This just means that the steps to process a Firefox crash were encoded in a modular manner and, through configuration, attached to the FTS framework. Many of the Socorro apps follow that basic FTS model allowing them all to share the same multi-threading framework and storage schemes.
What's the advantage of doing this?
First, sharing code between apps allows us to reduce the shear size of the codebase that we have to maintain. No more repeating similar mechanisms between apps. This also allows the steps to accomplish a task in a Socorro application to be isolated and reduced down to a single action. For example, in the processor, there is now a single method called 'convert_raw_crash_to_processed_crash'. That method knows nothing about storage mechanisms, queuing, or threading. Its target action is its only concern. The external framework handles all the other details.
Second, with that focus on a single action, the methods can be optimized. Here's some stats comparing last weekend with Procssor2008 and this weekend with Processor2012:
Average Time from submission to completed processing:
Processor2008: 40 seconds
Processor2012: 16 seconds
Average Time for actual conversion from raw to processed:
Processor2008: 2.7 seconds
Processor2012: 1.7 seconds
I really don't think I need to say anything more. I'm just smiling today.
disclaimer: Processor2012 has only been running for a few days, long term stats will have to be collected to get a more focused picture of the performance characteristics.