Well, as promised I am following up on my blog that talked in general about transactions. Today I want to discuss transactional Cache with persistence.
First what does it mean? For me this is transaction that requires ACID like properties for in-memory data grids. They are committed in memory first and returned to the application then simultaneously go to disk and databases for persistence meeting ACID qualities.
What is required? The ability to scale distributed transactions across two to hundreds of nodes and then to disk and databases. This is no trivial feat. In fact, the complexity of first making it work is challenging enough. How will performance be? We all know 2PC and XA will not scale on performance in this architecture and in fact degrades. Sure you can throw lots of hardware at the problem but it becomes very expensive to run and more expensive to maintain. Which brings me to the next level of complexity. What happens when a node goes down or a database goes down? How do you dynamically reallocate resources and not lose transactions in the process? What if your site goes down due to some catastrophic event? How does one handle uptime in worse case scenarios? As they say the devil is in the details.
Then the question is who can address this complex problem getting performance and scale across let’s say 10 to 100’s of nodes? It is not a big community of developers. If it were there would have been an open source initiative already in place? Some developers will try to write it themselves yet is that really the best thing for the business or would an out of the box solution be better? Businesses have to be aware of risk of a one off solution that requires constant attention and maintenance. And putting too many developers on this may create less than an ideal environment to solve the problem. It takes dedication of a few very knowledgeable developers with tenacity, resilience, and time such as a few years to work through the use cases and edge cases. No trivial matter here yet very important work.
Solving this problem for in-memory data grids allows any mission critical application to move to higher performance and scale at extreme affordability. The cost of the business transaction could be a fraction of what it is today! With a few bright people it will and can happen. This single-minded focus and effort on transaction management will bring costs down dramatically as well as flexibility for one business. It will also open the in-memory market from several hundred million today to billions in a few years. And with it Big Data will be redefined as well.