Organising Data in the Cloud is No Mean Feat

Data management is a complex process, made tougher due to the humongous amount of data being generated each day. Databases were developed to make sense of this data but these need to evolve continuously with the increasing demands being made on them. Following a few best practices will also help to manage data well.

Databases came into the limelight during the initial growth of application programming. As data grew in volume, a second generation of database architecture evolved called NoSQL databases. These offered strong support in architecting a data warehouse or data lake-based application design.

First-generation database systems (e.g., Oracle, MySQL) have a lot of flexibility in terms of developer access for querying and accessing/interpreting the data as compared to second-generation (NoSQL) database systems like Hadoop and MongoDB. On the other hand, second-generation database systems are designed to address the limitations of RDBMS in terms of handling volume and multi-branched structural data.

To cater to online transaction processing (OLTP) of first-generation databases and the structural accessibility for large volume data handled by second-generation database systems, hybrid database systems called NewSQL have emerged as third-generation database systems.

NewSQL database systems try to address the shortcomings of second-generation database systems by combining some of the advantages of first-generation systems like relation between entity models, ACID transaction handling capability, etc. Popular NewSQL database systems are VoltDB, ClusterixDB, CockroachDB and NeoDB.

Polyglot databases in cloud platforms

A data store is an integral component of any application development as it is the persistent store for handling all the data required for processing logic. Database solutions are moving towards polyglot databases, which combine multiple types of databases for an enterprise architecture. Polyglot databases are typically used in microservices architecture as they use multi-types of database systems in an integrated solution.