Navigate the World of Cloud Data Services: An Overview for Tech Executives

Learn about the primary categories of cloud data services and their key use cases.

In this first installment of our cloud data services series, we present you with a practical categorization and concise overview of cloud data solutions.

We have highlighted use cases and include notable examples from the top cloud services providers namely, Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, Snowflake and Databricks.

Our focus is on core data management services in the data value chain, excluding adjacent capabilities, such as Business Intelligence (BI), Artificial Intelligence (AI) & Machine Learning (ML), DataOps, Orchestration and Security.

#1. Relational Database Management System (RDBMS)

The row-oriented relational database has long been the dominant data management solution.

Consider a Relational Database When You Require:

• Data Integrity: Structured table schemas, modeling requirements and relationships between tables.
• Transaction Consistency: Atomic, consistent, isolated and durable (ACID-compliant) transaction processing.
• SQL: Powerful query language for data retrieval and manipulation.
• Ecosystem Maturity: Comprehensive tooling and community.

Relational Database Limitations:

• Horizontal Scalability: Limited ability to scale out.
• Schema Inflexibility: Fixed schemas and relationships.
• Query Performance: Time-consuming table joins.
• Unstructured Data Handling: Limited capabilities for managing semi-structured and unstructured data.

Examples of Cloud Relational Database Services:

• Amazon RDS & Amazon Aurora
• Google Cloud SQL & Google Cloud Spanner
• Azure Database & Azure SQL Database

#2. Column-Oriented Database / Columnar Database / Data Warehouse-as-a-Service (DWaaS)

Column-oriented databases optimize analytical performance by storing and retrieving data by columns instead of rows.

Consider a Columnar Database for:

• Analytical Workloads: Large-scale data analytics and complex queries.
• High Query Performance: Fast execution on large datasets.
• Massive Scalability: Separation of storage and compute, as well as horizontal scaling.
• Elasticity: Adapt to changing workloads and optimize cost.
• SQL: Familiar and versatile language.
• Concurrency: Support high number of concurrent users and queries.
• Data Compression: Columnar compression techniques.
  

Columnar Database Limitations:

• Write Performance: Slower writes than row-oriented databases.
• Consistency: Eventual consistency may cause results lag.

Examples of Cloud Columnar Database Solutions:

• Amazon Redshift & Amazon Athena (with Parquet or ORC)
• Google BigQuery & Google AlloyDB
• Azure Synapse Analytics & Azure Databricks (with Parquet, ORC or Delta)
• Snowflake
• Databricks Lakehouse

#3. Data Lake

A data lake is a platform for storing and processing raw data on distributed, parallelized infrastructure, with two core capabilities:

1. Storage: Holds unstructured, semi-structured and structured data.
2. Compute: Processes and analyzes data.

Consider Data Lake Adoption for:

• Flexibility: Handle disparate data formats and types.
• Customization: Tailor clusters to specific requirements.
• Versatility: Leverage open-source tooling, such as those from the Hadoop ecosystem and others like Presto, Flink, Hudi and TensorFlow.
  

Data Lake Challenges:

• Complexity: Requires specialized expertise in distributed systems and tooling.
• Transactional Consistency: Often lack strong transactional consistency guarantees. Consider using Apache Spark in combination with Delta Lake for ACID-compliance.
• Data Governance: Managing data quality, cataloging and lineage can be a challenge.

Examples of Cloud Data Lake Services:

• Storage: Amazon S3, Google Cloud Storage & Azure Blob Storage (Azure Data Lake Storage Gen2)
• Storage & Compute: Amazon EMR, Google Dataproc & Azure HDInsight
• Compute: AWS Glue
  

#4. Key-Value Storage

Key-value storage is a type of NoSQL database that pairs unique keys with values.

Key-value tables excel in fast, simple lookups, inserts, updates and deletes using keys.

Consider Key-Value Storage for:

• High Performance: Fast read and write operations; in-memory options for sub-millisecond latency.
• Scalability: Horizontal scaling for growing data volumes and user loads.
• Simplicity: Straightforward data model without table relationships.
• Specific Use Cases: Read-heavy workloads, session management, caching and configuration data.
  

Key-Value Store Limitations:

• Limited Querying Capabilities: Largely restricted to key-based lookups.
• No Built-in Support for Relationships: Less suitable for complex data models.
• Data Redundancy: Potential introduction of redundant data which complicates data consistency.
• Inadequate for Complex Analytics: Not designed for analytical queries or aggregations.

Examples of Cloud Key-Value Services:

• Amazon DynamoDB, Amazon Keyspaces & Amazon ElastiCache
• Google Cloud Datastore, Google Cloud Firestore & Google Cloud Memorystore
• Azure Table Storage, Azure Cosmos DB & Azure Cache

#5. Document-Oriented Database

A document-oriented database stores collections of key-value pairs as JSON in a document.

Document databases are ACID-compliant within a single document and, for several document databases, across multi-document transactions.

Consider a Document Database for:

• Flexibility: Schema-less design for evolving data structures.
• Scalability: Horizontal scaling, suited for high-traffic applications.
• Performance: Fast querying of indexed JSON documents.
• ACID Compliance: Ensure consistency of document transactions.
  

Document Database Limitations:

• Query Limitations: Limited support for complex multi-document joins and aggregations.
• Data Redundancy: Potential duplicate data across documents increases storage requirements and management complexity.
  

Examples of Cloud Document-Oriented Services:

• Amazon DynamoDB & Amazon DocumentDB
• Azure CosmosDB
• Google Datastore & Google Firestore

Interested in Getting Started with Cloud Data Services?

We can help.

Our team of expert, cloud-certified data architects and data engineers will partner closely with you to design, build and deploy a high-performance and cost-efficient combination of cloud services to meet your specific needs.

Get started with us today to start seeing the benefits of massive scalability, agility, resiliency and stability that cloud data services enable for you.

#6. Wide Column Storage

Wide column storage is similar to document storage, in that both patterns map a collection of values to a key. However, in a wide column table, each value has its own column.

Wide column tables enable extensible records by linking an arbitrary number of columns to a row.

Consider Wide Column Storage for:

• Scalability: Ideal for massive data volumes and heavy write loads.
• Flexibility: Dynamic addition of columns without schema modification.
• Performance: Efficient row-based queries and wide column retrieval.
  

Wide Column Storage Limitations:

• Query Limitations: Limited support for complex queries and joins.
• Inconsistent Data Model: Potential for varying structures across rows.
  

Examples of Cloud Wide Column Services:

• Amazon DynamoDB & Amazon Keyspaces
• Azure Table Storage & Azure CosmosDB
• Google Cloud Bigtable

#7. Streaming

Streaming systems perform real-time (milliseconds latency) to near-real-time (single-digit minutes latency) collection, processing and/or analytics of events.

Streaming systems enable decoupled, asynchronous and scalable event management between event production (E.g., Ecommerce Order Submission) and event consumption (E.g., Order Status Correspondence, Order Fulfillment, Inventory Management).

Consider Streaming for:

• Real-Time/Near-Real-Time: Immediate processing and analysis of events.
• Decoupling: Asynchronous event management for production and consumption.
• Scalability: Efficient handling of large-scale, high-velocity data streams.

Streaming Challenges:

Complex Infrastructure: Requires robust architecture and monitoring. Complexity now largely mitigated when using cloud managed services, particularly serverless offerings.

Examples of Cloud Streaming and Stream Processing Services:

• Amazon Kinesis, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon EMR (Spark), Amazon Glue Streaming & AWS Lambda
• Azure Event Hubs, Azure Stream Analytics, Azure HDInsight (Spark or Kafka) & Azure Functions
• Google Cloud Pub/Sub, Google Dataproc (Spark), Google Dataflow & Google Cloud Functions

#8. Graph Database

Leverage a graph database to manage the relationships (edges) between entities (nodes/vertices).

Graph databases treat relationships between entities as first-class citizens.

Therefore, the relationships between entities drive much of the value in a graph database.

Graph databases are particularly powerful for datasets that prioritize relationships between entities, such as social networking.

Consider a Graph Database for:

• Relationship-Driven Data: Highly connected entities with descriptive, persistent relationships. E.g., social networks, recommendation engines, fraud detection.
• Complex Data Navigation: Fast traversal of connected entities, cyclic relationships and self-referenced entities.
• Schema Flexibility: Gracefully handles evolving data structures, many-to-many relationships and dynamically changing relationships, especially for tree-structured data.

Graph Database Limitations:

• ​​Scalability Limitations: Potential scalability challenges with massive-scale data sets and high-velocity updates.
• Query Expertise: Requires learning and adapting to graph-specific query languages.
• Resource-Intensive: High memory and compute demands for large-scale graphs with write and read requirements on both entities and relationships.

Examples of Cloud Graph Processing Services:

• Amazon Neptune & Neo4j AuraDB on AWS
• Neo4j AuraDB on Google Cloud Platform
• Azure Cosmos DB (Apache Gremlin) & Neo4j AuraDB on Microsoft Azure

#9. Search Database

Search databases enable real-time indexing, retrieval and analytics of data, spanning unstructured text, semi-structured text, structured data and other formats like numerical and geospatial data.

Consider a Search Database for:

• Machine Data Observability: For enhanced visibility into machine-generated data, such as logs, metrics and traces.
• Fast Full-Text Search: Low-latency indexing for monitoring and analytics on text data.
• Ad Hoc Queries: Execute simple to complex, exact to fuzzy searches across text documents.
• Real-time Analytics: Powerful tooling when instantaneous insights from streaming data are required.

Search Database Limitations:

• Imprecise Results: May not be suitable for workloads demanding exact results.
• Complex Transactions: Lack of support for workloads requiring multiple operations.
• Query Performance: As data size and query complexity increase, search performance may degrade.
• Data Consistency: Immediate consistency is not guaranteed, which may impact certain workload scenarios.

Examples of Cloud Search Services:

• Amazon OpenSearch
• Azure AI Search & Elasticsearch (Elastic Cloud) for Azure
• Elasticsearch (Elastic Cloud) for Google Cloud

#10. Vector Database

Vector databases enable intelligent, contextually-aware information retrieval of similar entities.

A prerequisite for storing an entity in a vector database is to create a vector embedding of the entity using an embedding model. An embedding model converts the entity into a numeric representation of semantic meaning.

A vector embedding is a representation of your entity as a series of floating point values in a high-dimensional vector space, each dimension referring to a semantic attribute of your entity. Hence, the term vector database.

A vector database enables storage, indexing and querying of your embeddings.

Consider a Vector Database for:

• Semantic Search: Also called similarity search. Perform searches by similar meaning and context, even when wording is different.
• Fuzzy Search: Also called lexical search. Perform searches by similar text, such as similar spellings or misspellings.
• Unstructured and Semi-Structured Data: Work with any kind of data. Index and search high-dimensional data, such as documents, images, audio and video.
• Retrieval-Augmented Generation (RAG) for Generative AI: RAG is a powerful method to enhance the relevance and accuracy of large language model (LLM) outputs by augmenting your prompts with relevant entities from a vector database. RAG enables decoupling the reasoning capabilities (How AI thinks) from the memory capabilities (What AI knows) for generative AI. Think of RAG as short-term memory for your LLM, similar to RAM on your laptop.
• Recommendation Systems: Build powerful and efficient recommendation systems for content, products and people.
  
• Clustering / Segmentation: Group or identify semantically similar entities.
  
• Anomaly Detection: Identify outliers from a semantic grouping of similar entities.
  

Vector Database Considerations:

• Probabilistic Search: Vector databases excel in semantic and relevance-based search. While vector databases can perform exact search, they might not always guarantee the precision of a traditional relational database.
• Full Index Rebuilds: Vector database indexes typically do not yet support incremental rebuilds of an index, when adding new embeddings to the index. Therefore, vector indexes typically require a full rebuild for new embeddings, which adds operational complexity and cost to your solution.
  
• Chunking: Chunking is the process of extracting only relevant parts of your input entity prior to embedding and storage in your vector database. Chunking is important to minimize noise and maximize the relevance and accuracy of vector searches. Furthermore, chunking enables more performant and cost-efficient information retrieval from your vector database. However, chunking adds another layer of operational complexity and cost to your solution.
  
• New to Market: Vector databases are relatively new, in terms of market adoption. Therefore, while vector database capabilities are evolving quickly, their capabilities are not yet as mature as other more established data management patterns, like the SQL database.
  
• Cost: Storing, indexing and querying large volumes of high-dimensional vectors is compute-intensive and incurs material costs for your cloud deployments. Serverless offerings like Pinecone Serverless are more cost-efficient through separation of storage from compute; and a pay-per-use pricing model.
  

Examples of Vector Databases:

Vector-Native Databases

• Pinecone
• Chroma
  
• Weaviate
  
• Qdrant
  
• Milvus
  
• Zilliz
  
• Cloudflare Vectorize
  

Datastores with Vector Search

• Amazon OpenSearch Serverless Vector Engine
• Amazon RDS / Aurora PostgreSQL pgvector
  
• Google Vertex AI Vector Search
  
• Elasticsearch Vector Search
  
• Oracle AI Vector Search
  
• Azure SQL Database Vector Search (via Azure AI Search)
  
• MongoDB Atlas Vector Search
  
• Databricks Vector Search
  
• Snowflake Cortex
  

Conclusion

In the rapidly evolving cloud landscape, understanding and selecting the right cloud data services for your organization's needs is crucial.

We have briefly touched on the primary categories of cloud data services and their key use cases.

Stay tuned for future articles in which we will dive into each of the cloud data service categories in greater detail.

Interested in Getting Started with Cloud Data Services?

You are in great hands with us.

Contact us today to get started with cutting-edge cloud data solutions to streamline your operations and enable rapid innovation and growth for your business.

Our team of cloud-certified experts has extensive data engineering experience building advanced cloud data platforms for leading companies and innovative startups.

Schedule a cloud data strategy and architecture consultation with us today to empower your business with cutting-edge cloud, data and AI solutions that put you ahead of the curve and future-proof your business.