Learn about the primary categories of cloud data services and their key use cases.
In this first installment of our cloud data services series, we present you with a practical categorization and concise overview of cloud data solutions.
We have highlighted use cases and include notable examples from the top cloud services providers namely, Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, Snowflake and Databricks.
Our focus is on core data management services in the data value chain, excluding adjacent capabilities, such as Business Intelligence (BI), Artificial Intelligence (AI) & Machine Learning (ML), DataOps, Orchestration and Security.
#1. Relational Database Management System (RDBMS)
The row-oriented relational database has long been the dominant data management solution.
Consider a Relational Database When You Require:
- Data Integrity: Structured table schemas, modeling requirements and relationships between tables.
- Transaction Consistency: Atomic, consistent, isolated and durable (ACID-compliant) transaction processing.
- SQL: Powerful query language for data retrieval and manipulation.
- Ecosystem Maturity: Comprehensive tooling and community.
Relational Database Limitations:
- Horizontal Scalability: Limited ability to scale out.
- Schema Inflexibility: Fixed schemas and relationships.
- Query Performance: Time-consuming table joins.
- Unstructured Data Handling: Limited capabilities for managing semi-structured and unstructured data.
Examples of Cloud Relational Database Services:
- Amazon RDS & Amazon Aurora
- Google Cloud SQL & Google Cloud Spanner
- Azure Database & Azure SQL Database
#2. Column-Oriented Database / Columnar Database / Data Warehouse-as-a-Service (DWaaS)
Column-oriented databases optimize analytical performance by storing and retrieving data by columns instead of rows.
Consider a Columnar Database for:
- Analytical Workloads: Large-scale data analytics and complex queries.
- High Query Performance: Fast execution on large datasets.
- Massive Scalability: Separation of storage and compute, as well as horizontal scaling.
- Elasticity: Adapt to changing workloads and optimize cost.
- SQL: Familiar and versatile language.
- Concurrency: Support high number of concurrent users and queries.
- Data Compression: Columnar compression techniques.
Columnar Database Limitations:
- Write Performance: Slower writes than row-oriented databases.
- Consistency: Eventual consistency may cause results lag.
Examples of Cloud Columnar Database Solutions:
- Amazon Redshift & Amazon Athena (with Parquet or ORC)
- Google BigQuery & Google AlloyDB
- Azure Synapse Analytics & Azure Databricks (with Parquet, ORC or Delta)
- Snowflake
- Databricks Lakehouse
#3. Data Lake
A data lake is a platform for storing and processing raw data on distributed, parallelized infrastructure, with two core capabilities:
- Storage: Holds unstructured, semi-structured and structured data.
- Compute: Processes and analyzes data.
Consider Data Lake Adoption for:
- Flexibility: Handle disparate data formats and types.
- Customization: Tailor clusters to specific requirements.
- Versatility: Leverage open-source tooling, such as those from the Hadoop ecosystem and others like Presto, Flink, Hudi and TensorFlow.
Data Lake Challenges:
- Complexity: Requires specialized expertise in distributed systems and tooling.
- Transactional Consistency: Often lack strong transactional consistency guarantees. Consider using Apache Spark in combination with Delta Lake for ACID-compliance.
- Data Governance: Managing data quality, cataloging and lineage can be a challenge.
Examples of Cloud Data Lake Services:
- Storage: Amazon S3, Google Cloud Storage & Azure Blob Storage (Azure Data Lake Storage Gen2)
- Storage & Compute: Amazon EMR, Google Dataproc & Azure HDInsight
- Compute: AWS Glue
#4. Key-Value Storage
Key-value storage is a type of NoSQL database that pairs unique keys with values.
Key-value tables excel in fast, simple lookups, inserts, updates and deletes using keys.
Consider Key-Value Storage for:
- High Performance: Fast read and write operations; in-memory options for sub-millisecond latency.
- Scalability: Horizontal scaling for growing data volumes and user loads.
- Simplicity: Straightforward data model without table relationships.
- Specific Use Cases: Read-heavy workloads, session management, caching and configuration data.
Key-Value Store Limitations:
- Limited Querying Capabilities: Largely restricted to key-based lookups.
- No Built-in Support for Relationships: Less suitable for complex data models.
- Data Redundancy: Potential introduction of redundant data which complicates data consistency.
- Inadequate for Complex Analytics: Not designed for analytical queries or aggregations.
Examples of Cloud Key-Value Services:
- Amazon DynamoDB, Amazon Keyspaces & Amazon ElastiCache
- Google Cloud Datastore, Google Cloud Firestore & Google Cloud Memorystore
- Azure Table Storage, Azure Cosmos DB & Azure Cache
#5. Document-Oriented Database
A document-oriented database stores collections of key-value pairs as JSON in a document.
Document databases are ACID-compliant within a single document and, for several document databases, across multi-document transactions.
Consider a Document Database for:
- Flexibility: Schema-less design for evolving data structures.
- Scalability: Horizontal scaling, suited for high-traffic applications.
- Performance: Fast querying of indexed JSON documents.
- ACID Compliance: Ensure consistency of document transactions.
Document Database Limitations:
- Query Limitations: Limited support for complex multi-document joins and aggregations.
- Data Redundancy: Potential duplicate data across documents increases storage requirements and management complexity.
Examples of Cloud Document-Oriented Services:
- Amazon DynamoDB & Amazon DocumentDB
- Azure CosmosDB
- Google Datastore & Google Firestore
Interested in Getting Started with Cloud Data Services?
We can help.
Our team of expert, cloud-certified data architects and data engineers will partner closely with you to design, build and deploy a high-performance and cost-efficient combination of cloud services to meet your specific needs.
Get started with us today to start seeing the benefits of massive scalability, agility, resiliency and stability that cloud data services enable for you.
#6. Wide Column Storage
Wide column storage is similar to document storage, in that both patterns map a collection of values to a key. However, in a wide column table, each value has its own column.
Wide column tables enable extensible records by linking an arbitrary number of columns to a row.
Consider Wide Column Storage for:
- Scalability: Ideal for massive data volumes and heavy write loads.
- Flexibility: Dynamic addition of columns without schema modification.
- Performance: Efficient row-based queries and wide column retrieval.
Wide Column Storage Limitations:
- Query Limitations: Limited support for complex queries and joins.
- Inconsistent Data Model: Potential for varying structures across rows.
Examples of Cloud Wide Column Services:
- Amazon DynamoDB & Amazon Keyspaces
- Azure Table Storage & Azure CosmosDB
- Google Cloud Bigtable
#7. Streaming
Streaming systems perform real-time (milliseconds latency) to near-real-time (single-digit minutes latency) collection, processing and/or analytics of events.
Streaming systems enable decoupled, asynchronous and scalable event management between event production (E.g., Ecommerce Order Submission) and event consumption (E.g., Order Status Correspondence, Order Fulfillment, Inventory Management).
Consider Streaming for:
- Real-Time/Near-Real-Time: Immediate processing and analysis of events.
- Decoupling: Asynchronous event management for production and consumption.
- Scalability: Efficient handling of large-scale, high-velocity data streams.
Streaming Challenges:
Complex Infrastructure: Requires robust architecture and monitoring. Complexity now largely mitigated when using cloud managed services, particularly serverless offerings.
Examples of Cloud Streaming and Stream Processing Services:
- Amazon Kinesis, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon EMR (Spark), Amazon Glue Streaming & AWS Lambda
- Azure Event Hubs, Azure Stream Analytics, Azure HDInsight (Spark or Kafka) & Azure Functions
- Google Cloud Pub/Sub, Google Dataproc (Spark), Google Dataflow & Google Cloud Functions
#8. Graph Database
Leverage a graph database to manage the relationships (edges) between entities (nodes/vertices).
Graph databases treat relationships between entities as first-class citizens.
Therefore, the relationships between entities drive much of the value in a graph database.
Graph databases are particularly powerful for datasets that prioritize relationships between entities, such as social networking.
Consider a Graph Database for:
- Relationship-Driven Data: Highly connected entities with descriptive, persistent relationships. E.g., social networks, recommendation engines, fraud detection.
- Complex Data Navigation: Fast traversal of connected entities, cyclic relationships and self-referenced entities.
- Schema Flexibility: Gracefully handles evolving data structures, many-to-many relationships and dynamically changing relationships, especially for tree-structured data.
Graph Database Limitations:
- Scalability Limitations: Potential scalability challenges with massive-scale data sets and high-velocity updates.
- Query Expertise: Requires learning and adapting to graph-specific query languages.
- Resource-Intensive: High memory and compute demands for large-scale graphs with write and read requirements on both entities and relationships.
Examples of Cloud Graph Processing Services:
- Amazon Neptune & Neo4j AuraDB on AWS
- Neo4j AuraDB on Google Cloud Platform
- Azure Cosmos DB (Apache Gremlin) & Neo4j AuraDB on Microsoft Azure
#9. Search Databases
Search databases enable real-time indexing, retrieval and analytics of data, spanning unstructured text, semi-structured text, structured data and other formats like numerical and geospatial data.
Consider a Search Database for:
- Machine Data Observability: For enhanced visibility into machine-generated data, such as logs, metrics and traces.
- Fast Full-Text Search: Low-latency indexing for monitoring and analytics on text data.
- Ad Hoc Queries: Execute simple to complex, exact to fuzzy searches across text documents.
- Real-time Analytics: Powerful tooling when instantaneous insights from streaming data are required.
Search Database Limitations:
- Imprecise Results: May not be suitable for workloads demanding exact results.
- Complex Transactions: Lack of support for workloads requiring multiple operations.
- Query Performance: As data size and query complexity increase, search performance may degrade.
- Data Consistency: Immediate consistency is not guaranteed, which may impact certain workload scenarios.
Examples of Cloud Search Services:
- Amazon OpenSearch
- Azure AI Search & Elasticsearch (Elastic Cloud) for Azure
- Elasticsearch (Elastic Cloud) for Google Cloud
Conclusion
In the rapidly evolving cloud landscape, understanding and selecting the right cloud data services for your organization's needs is crucial.
We have briefly touched on the primary categories of cloud data services and their key use cases.
Stay tuned for future articles in which we will dive into each of the cloud data service categories in greater detail.
Interested in Getting Started with Cloud Data Services?
You are in great hands with us.
Contact us today to get started with cutting-edge cloud data solutions to streamline your operations and enable rapid innovation and growth for your business.
Our team of cloud-certified experts has extensive data engineering experience building advanced cloud data platforms for leading companies and innovative startups.
Schedule a cloud data strategy and architecture consultation with us today to empower your business with cutting-edge cloud, data and AI solutions that put you ahead of the curve and future-proof your business.
