From POC to Production: Near Real-Time Supply Chain Analytics with Databricks SQL 

A large logistics and supply chain organization operates hundreds of physical warehouse sites.
These warehouses rely on industry-standard Warehouse Management Systems (WMS) such as
Blue Yonder, Manhattan, and similar applications to manage daily operations.

These systems continuously capture operational events such as:

• Inventory movements and adjustments
• Orders and order line items
• Picks, packs, shipments, and returns

While the operational systems were working well, business users and operations teams needed
near real-time visibility into what was happening on the warehouse floor, but insights often arrived
minutes—or even hours—after the actual events.

In a fast-moving supply chain environment, that delay mattered.

The Data Challenge: Near Real-Time Analytics at Scale

Each warehouse management system maintains a well-defined relational data model.
For every warehouse site, there are hundreds of tables capturing transactional and reference data.

The analytics requirements were clear:

• Data needed to be replicated using CDC (Change Data Capture) every 1–5 minutes.
• Replication had to happen across all warehouse sites without compromising analytics performance.
• Analytics needed to reflect changes within minutes.

The solution had to scale across the following:

• Hundreds of tables
• Multiple warehouse locations and sites
• Continuous data changes
• High-concurrency analytical workloads

Traditional batch ETL pipelines were no longer sufficient.

Target Platform: A Lakehouse-Based Data Architecture

The organization had already standardized on a Databricks-based Lakehouse paradigm
for its central data platform.

Key components included:

• • Qlik Replicate for CDC ingestion from warehouse systems
  • Databricks and Apache Spark for ingestion and ETL
  • Delta Lake as the open, transactional storage layer

One architectural question remained open:
Should near real-time analytics be served from an external cloud data warehouse (Snowflake), or directly from the Lakehouse using Databricks SQL?

Both options were viable. Rather than assume, the team decided to validate the choice through a structured Proof of Concept (POC).

The POC Approach: Databricks SQL vs Snowflake

The goal of the POC was not to “pick a winner” in theory, but to understand which platform worked best for this specific use case.

POC Objectives

• Validate end-to-end near real-time CDC ingestion
• Compare performance on fresh, continuously changing data
• Evaluate cost behavior under realistic workloads
• Assess operational complexity at scale

POC Setup

To ensure a fair comparison:

• The same source systems were used
• The same CDC streams were ingested
• The same data volumes and schemas were replicated
• The same ingestion tool (Qlik Replicate) was used
• Identical analytical queries and BI workloads were executed

Two paths were evaluated:

1. CDC ingestion into Delta Lake with analytics via Databricks SQL Serverless
2. CDC ingestion into a cloud data lake, followed by replication into Snowflake for analytics

What Worked Well on Both Platforms

The POC confirmed that both Databricks SQL and Snowflake are strong, enterprise-grade platforms.

Both demonstrated:

• Excellent horizontal scalability
• Clear separation of compute and storage
• Familiar and expressive SQL for analytics
• Strong support for CDC ingestion patterns
• Reliability under sustained workloads

At a high level, either platform could meet the functional requirements.

The difference showed up in how the architecture behaved over time.

Where Databricks SQL Stood Out

Cost Efficiency at Scale

With Databricks SQL:

• Analytics ran directly on Delta Lake
• There was no duplicate storage of data
• Compute was consumed only when queries executed
• Performance was at par with Databricks SQL Serverless compute

In contrast, the Snowflake approach introduced:

• Additional end-to-end costs
• Extra compute to move data back into the Lakehouse for other use cases
• Ongoing synchronization between the data lake and warehouse

As data volumes and warehouse sites scaled, the cost difference became increasingly visible.

Interoperability with Open File Format Delta Lake

Databricks SQL is natively optimized for Delta Lake, and Delta Lake as an open table format is interoperable across the modern data ecosystem.

• Low-latency queries on continuously changing data
• Efficient handling of CDC-driven merges by Qlik Replicate
• Consistent performance even as data freshness increased

This proved especially important for operational dashboards querying data that had just arrived.

Unified Analytics on a Single Platform

Using Databricks SQL allowed the organization to:

• Ingest data
• Transform it
• Serve near real-time analytics

—all on the same Lakehouse platform.

There was no need for:

• Additional data movement
• Separate warehouse pipelines
• Duplicate governance models

This simplification reduced both operational overhead and architectural complexity.

The Final Decision

Based on the POC results, the organization chose to use
Databricks SQL Serverless as the data warehouse layer on the Lakehouse.

The decision was driven by:

• Lower total cost of ownership
• Strong performance on Delta Lake
• Unified analytics capabilities
• Reduced system complexity

Snowflake remained a capable platform, but for this near real-time Lakehouse-centric use case,
Databricks SQL was a better architectural fit.

Final Architecture: Near Real-Time Analytics on the Lakehouse

The production architecture followed a clean, streamlined flow:

1. CDC ingestion from WMS databases using Qlik Replicate
2. Data landed and managed in Delta Lake
3. Databricks SQL Serverless served:
   • BI dashboards
   • Operational analytics
   • Near real-time KPIs

Technical Considerations and Best Practices

Some practical learnings from the implementation:

• CDC-driven MERGE Patterns require careful tuning
• Late-arriving data must be handled explicitly
• Delta Lake features such as schema evolution, time travel, and ACID guarantees proved critical for reliability
• Databricks SQL Serverless handled concurrency well without manual cluster management

Key Takeaways

• Near real-time analytics does not require a separate traditional data warehouse
• POCs are essential for making informed architectural decisions
• Open file formats matter as data volumes grow
• Cost efficiency becomes clearer at scale
• Unified platforms simplify both analytics and governance