The ‘in-memory’ future

Lately, it is hard to read an IT business magazine without encountering an article on how ‘in-memory’ is going to change everything. While the recent developments around in-memory analytics and processing are indeed very promising, it is worth having a look at the underlying technologies, before jumping on the bandwagon. The discussions on in-memory often include one or more of the following:

Big data
Flash memory
In-memory analytics
In-memory transactional processing.

Big data

‘Big data’ is itself a buzzword. 5GB of relational banking transactional data a week is not big data but on the other hand, CERN’s Large Hadron Collider’s experimental results, at 6GB/second, are much more in the big data category. At its core, big data refers to data that is too large, too fast and/or too varied to be processed with traditional data-processing methods. It is not just about size, big data implies that there is a lot of data, that it is accumulating quickly and that it is unstructured. While big data may be processed using in-memory techniques, as a result of the size of the data involved, it is not likely to be stored in memory. Aggregations may be stored in memory, temporarily or permanently, but not the data itself.

Flash memory

When talking about in-memory, it is important to distinguish what kind of memory is being spoken of. Flash memory is a high-speed, non-volatile storage technology. It is what is used in Solid State Hard Drives (SSDs). While significantly faster than traditional magnetic media, flash memory is still a storage technology. However, it can be used to extend a server’s main memory as in the case of SQL Server 2014’s Buffer Pool Extensions. Generally, flash memory is used to reduce or remove the IO latency from applications that store and retrieve large amounts of data, like relational database engines. Since flash storage in the form of SSDs appear as regular drives, the applications using them need no special code or changes to handle them.

In-memory analytics

In-memory analytics allows for very fast querying and analysis of data. It encompasses applications such as Tableau and the PowerPivot plugin for Microsoft’s Excel. In-memory analytics will often store data in columnar format, as opposed to the traditional row-based storage seen in relational databases. Columnar storage enables a very high compression rate, allowing for high speed analytics. The high compression means that large data sets can be worked on in-memory. Typically, the data is stored on durable storage and processed in-memory.

In-memory transactional processing

With in-memory OLTP engines, the data is stored in volatile memory and persisted at intervals to durable storage (flash memory or traditional magnetic drives). In-memory OLTP allows for extremely fast transactional processing, making it suitable for applications that need very high insert rates and to execute huge numbers of small transactions. Examples of this are stock market trading, telephone exchanges and on-line stores. Solutions in this area include the Hekaton feature in Microsoft SQL Server 2014 (renamed as In-memory OLTP) and Oracle’s TimesTen In-Memory Database engine.

Right solution in the right place

Whichever form of in-memory solution is being considered, it is critically important that it not be implemented as a ‘solution looking for a problem’. The problems the various in-memory solutions solve are very different from one another. There is no gain from deploying in-memory solutions for the sake of doing so. Any solution must be carefully considered in the light of what problem it is expected to solve and what gains it is expected to give. Like with most areas of technology, there is unfortunately no magic solution for all problems to be found here.

Related Articles

Customer data privacy:
Trust or trapped
MVP vs big bang release
Effective prioritisation:
Not all requirements are created equal