About Tom Henn

Tom Henn, our CEO, President, and co-founder, brings 30 years of experience to CloudTran from a variety of successful companies. Prior to starting CloudTran, Tom was a consultant for many Silicon Valley start-ups, primarily in the cloud computing, open source, and data management space. He was CEO of Centerboard, a provider of data center virtualization software. Prior to this, Tom was the CEO of Cloudscape, a developer of database and synchronization tools for enterprise Java applications, and VP Worldwide Sales of Red Brick Systems, a provider of database management system for data warehouse applications.

Transactional Data Cache

Well, as promised I am following up on my blog that talked in general about transactions.  Today I want to discuss transactional Cache with persistence.

First what does it mean?  For me this is transaction that requires ACID like properties for in-memory data grids. They are committed in memory first and returned to the application then simultaneously go to disk and databases for persistence meeting ACID qualities.

What is required? The ability to scale distributed transactions across two to hundreds of nodes and then to disk and databases.   This is no trivial feat.  In fact, the complexity of first making it work is challenging enough.  How will performance be?  We all know 2PC and XA will not scale on performance in this architecture and in fact degrades.  Sure you can throw lots of hardware at the problem but it becomes very expensive to run and more expensive to maintain.  Which brings me to the next level of complexity.  What happens when a node goes down or a database goes down?  How do you dynamically reallocate resources and not lose transactions in the process?   What if your site goes down due to some catastrophic event? How does one handle uptime in worse case scenarios?  As they say the devil is in the details.

Then the question is who can address this complex problem getting performance and scale across let’s say 10 to 100’s of nodes?  It is not a big community of developers.  If it were there would have been an open source initiative already in place? Some developers will try to write it themselves yet is that really the best thing for the business or would an out of the box solution be better?   Businesses have to be aware of risk of a one off solution that requires constant attention and maintenance. And putting too many developers on this may create less than an ideal environment to solve the problem.  It takes dedication of a few very knowledgeable developers with tenacity, resilience, and time such as a few years to work through the use cases and edge cases.  No trivial matter here yet very important work.

Solving this problem for in-memory data grids allows any mission critical application to move to higher performance and scale at extreme affordability.  The cost of the business transaction could be a fraction of what it is today! With a few bright people it will and can happen.  This single-minded focus and effort on transaction management will bring costs down dramatically as well as flexibility for one business.   It will also open the in-memory market from several hundred million today to billions in a few years.   And with it Big Data will be redefined as well.

The Transactional Supply Chain

Transaction management is much like FedEx or UPS managing movement of parcels from one location to another starting with payment, tagging with an ID, routing, monitoring/tracking and confirmation of delivery.  The package moves through multiple distribution centers and is confirmed at each location it travels through. When it is delivered, a final confirmation happens.

So using this analogy, let’s start with defining what is the Transactional Supply Chain for in-memory and cloud architectures.  First, it can include a variety of transactions, ACID Transactions for payment, commerce, financial commitment, etc., NO-ACID or what I will call “less than ACID” transactions for messaging, logging, updating catalogues.  Overall, there are several types of transactions, and there is not a one-fit-all scenario — a transaction is specific to the need based upon the business.  In addition, there are multiple data stores where persistence has to happen. For this example, let’s say 16 data stores/databases located across the world.

In the Transactional Supply Chain, let’s assume we have a Transactional Bus that manages transactions for purposes of placing an order, invoicing, updating inventory, shipping and messaging to the buyer.  The buyer may participate in social commerce where he or she brags about buying an item by sending a message (transaction) to others they know. If these others like it, they go and buy it, and we start the transactional bus moving down the transactional supply chain over and over again.  The information about these buyers can be aggregated in real time and sent to a big data analytics engine, a data warehouse, for example, so that analysis will help to understand both individual customers as well as demographic groups, creating more business.  Then once again a new buyer engages, which at some point triggers an ACID commerce transaction to confirm the order and payment, updates inventory, invoicing, shipping information, and tracks this transaction until confirmation of delivery.

The notion that one type of transaction can handle all of this has changed to what type of transaction is best used for in-memory grids and cloud architectures.  That said, whether it starts or ends with an ACID transaction, ACID transactions are an essential part of the transactional supply chain. Without this ACID transaction option we would be writing brittle transaction management code to maintain and monitor—very time consuming and very costly.

Well, if there were an out-of-the-box solution for distributed ACID transactions that could deliver high scalability and exception performance, much of the bally ho above would be unnecessary.  This would be a technology solution that business managers and developers obviously have a crying need for.  So for distributed transactions, if you could do them in a highly scalable way, with consistent performance for commit time, and quickly (e.g., milliseconds) in a scale out/grid environment, whether in-house or in a private, public or hybrid cloud… would this seem a much more reasonable choice?

So the world of transactions is NOT about one single transaction type. It is about what kind of transaction serves the best purpose for the business intent such as socializing, commerce, ordering, etc.  It is about a Transactional Supply Chain where a single database or data source is no longer enough!! Frankly, it is one of the biggest bottlenecks. The choking point for the web and worse for the grid, and even worse for the cloud is the database. Hence having a Transactional Bus to manage the transaction supply chain and the in-transit data movement to any and all data sources located around the world is essential.  Much like an application server layer there needs to be a data server layer.  A transactional bus independent of databases with transaction management and monitoring unbundled for flexibility, performance and scale to meet business needs seems so logical. Then why has someone not done it? Well, as you know, it is a very complex problem and some want to waddle in the “it can’t be done” camp.  Well, it can be done.

As we move to in-memory data grids and cloud architectures, the need for a Transactional Supply Chain will become a requirement and reality.  It will be a critically important business asset. Businesses will have much more flexibility to change and grow, the limits on transaction management will be lifted, and the transactional supply chain will include the full business life cycle of the transaction including such aspects as distributed ACID transactions uniting in-memory and disk for persistence, active to active transactional duplication, workflow management, aggregation for business intelligence, data policy management, active-state transaction tracking for alerts, customer tracking, and more.  These topics and more will be covered in future blogs.

It is an exciting time, and it all starts with the wonderful new way for businesses to be nimble transacting their business free of technology silos, free of limitations on transactions, and free to build your business with no limits!