In a Socorro meeting in September in Washington DC, we started a discussion about how
Socorro could be offered as a service. This is posting is a sketch of
my vision for such a service.
Would the developers with apps in the Mozilla Marketplace benefit from stats and aggregate reports from Socorro? Socorro processes and reports on binary crashes from the Firefox platform itself. Is that useful information to the Web developer? Well, they might want it to just see what Firefox problems they're triggering so that they may recode around our problems.
It might be more valuable for app developers to have Javascript stack information about their unhandled exceptions. Socorro doesn't do this right now. An unhandled Javascript exception is not something that brings down Firefox and triggers a breakpad crash submission. So the first step in getting a version of Socorro to help Web developers is to create and install a hook into our Firefox platform to submit an unhandled Javascript exception with stack information to a waiting instance of Socorro.
Since Socorro is so modular, we could yank out the breakpad binary crash handling and replace it with some Javascript stack analysis code. That implies that these crashes are submitted to a Socorro instance substantially different than the one that we use. I'll save the speculation as to what Javascript stack processing would look like for a future posting.
In this case, offering Socorro as a service means just offering a minimally configured Socorro (no Hadoop/HBase, file system storage only) in, perhaps, a VM. Otherwise, it's up to the web developer to host their own Socorro. If offering Socorro to Web developers means reporting on Javascript unhandled exceptions, then modifications will be required for both the Firefox platform and Socorro.
Does Mozilla itself have interest in a Javascript aware Socorro? If Mozilla is dog fooding their own platform and writing Marketplace apps (or the default apps such as the phone app), then I could see a Javascript aware Socorro as a valuable tool in tracking our own app problems.
Assuming that the regular binary crash information is of interest to app developers, how could Socorro be offered as a service? Our instance of Socorro shouldn't be burdened with all the aggregation and overhead of maintaining stats on potentially thousands of WebApps. I'm thinking about expanding Socorro in a different dimension: cooperating multiple instances of Socorro.
Imagine this: an app developer has his own instance of Socorro that only receives crashes associated with his own app. This instance of Socorro is either hosted by us on a VM, or hosted on some external system arranged by the app developer himself.
Whoa Nelly, can we share binary crash dumps with external developers? This is a critical question that is likely answered with a resounding, "no". The binary contains potentially "sensitive information" about the user. Privacy concerns would prevent us from sending the binary to third parties. Therefore the rest of this missive should be considered just a self indulgent speculation of which an implementation will never see the light of day.
How does the privately owned Socorro get crashes? Our Socorro will continue to receive all the crashes from our products. If Socorro senses that a crash happened in a specific WebApp, during our own save-the-crash-phase of Socorro processing, we just forward the crash on to the registered URL for that WebApp's private instance of Socorro. Once thrown over the wall, our Socorro continues with our normal crash processing. It is up to the WebApp developer to use his private Socorro to the benefit his own app.
Where does the private Socorro live? It really doesn't matter. We could offer the hosting. They could host it on Amazon or their own servers. If they want a full blown HBase enabled Socorro, they can host on their own cluster. If they only need a tiny instance, a fully functional Socorro can be configured sans HBase and run on a VM.
How does our Socorro know about the private Socorro so that it can forward crashes? This can be implemented in several ways. If a WebApp has a GUID and that GUID is included in the Breakpad metadata submitted with the crash, we could look up a registered crash submission URL and use it for forwarding the crash in exactly the same manner that our products submit crashes to us.
An alternative that I find more attractive, is having the private Socorro crash submission url included in the breakpad crash submission metadata. If the key is present in the metadata, then our master Socorro would just submit the crash directly to that URL. That way there are no lookups and we're not having to maintain a registry. It would be up to the app developer to insure the URL is up to date and correct. A failed submission is lost.
The changes to Socorro to enable this crash forwarding scheme are minimal. The Socorro CrashStorage hierarchy already has the ability to do an http submit of crashes (this is used in the testing app, SubmitterApp). There would be only minor modifications required, likely handled smoothly by inheritance. Including that class in the CrashMoverApp configuration is all that is needed.
So why would the WebApp developer want our Firefox binary crash reports?
It would also might be possible to hook into the Javascript unhandled exception code to do the crash report submission without involving breakpad at all. In such a case, the submission could by-pass our master Socorro entirely and directly submitting to the private Socorro.
This also leads to the idea that the app developer could skip Socorro entirely, roll their own crash reporting system that'll just receive crashes using the Socorro styled submission technique.