Yesterday, I wrote about
the use of the class TransactionExecutor
. Today, I'm going to propose a new way of
using it within Socorro's HBase crashstorage subsystem.
When used with Postgres, the
TransactionExecutor treats all of the individual steps within a
transaction collectively as if they were together a single atomic
operation. This is the exactly how we want a database transaction
treated. Lets say we have a transaction that consists of steps A, B,
C, D. We start a transaction and succeed in steps A and B. However,
C fails. The TransactionExecutor rolls back the transaction,
essentially undoing steps A and B. Then it retries the whole thing
from the beginning after sleeping.
That idea isn't ideal for HBase. Since
HBase doesn't support relational database-like transactions, the same
strategy of treating multiple steps as one atomic transaction doesn't
work as well. Just like the previous example, lets say that for
HBase we have four steps A, B, C and D. We start a “transaction”
and complete steps A and B before we get a failure in step C. If the
transaction executor is used exactly like it is in Postgres, we
rollback the transaction. But HBase doesn't support transactions, so
steps A and B are not actually undone. The TransactionExecutor then
sleeps and after waking, retries the whole thing from the beginning.We end up redoing steps A and B that
have already been done.
What are the consequences of this? It is my
understanding that rows in tables in HBase are static. We don't
really change them, we just make a new version of the row. The old
version of the row still sticks around (the number of old rows that
HBase saves is configurable). By starting the transaction over from
the beginning, we're increasing the amount of space that HBase needs
to save the same information.
Honestly, this isn't much of a problem.
During a failure, having multiple copies of the same information will
happen only to a single row (or in our case, one row per actively
writing thread during the HBase problem). That's not a lot of extra
storage wasted in comparison to the huge size of our data storage.
However, if we care to, we can resolve this
duplication problem by using the TransactionExecutor in a different
manner. Rather than treating all the steps A, B, C, D as if they are
collectively atomic, we could use the TransactionExecutor
individually on each step. If we don't fail until step C, then only
step C will be retried. We can avoid the duplication of steps A and
B.
For purity sake, it makes sense to use the TransactionExecutor on operations that are truly atomic. In our application with HBase, it really doesn't matter too much and would be likely more trouble than it is worth to change.