Databricks makes real-time mode generally available in Lakeflow pipelines

Databricks has made real-time mode generally available in Lakeflow pipelines, its declarative pipeline framework built on Spark Declarative Pipelines. The October 29, 2026 entry in the Databricks platform release notes states that the feature is now generally available on both classic compute and serverless compute, and that it targets operational workloads needing end-to-end latency as low as five milliseconds, with fraud detection given as the reference example.

The same day, Databricks published an engineering blog post describing real-time mode as an evolution of the Structured Streaming engine. Instead of processing a stream as a sequence of small micro-batches, real-time mode schedules every stage of a query at once and passes records between stages as soon as they are produced. Databricks claims the approach is up to 100 times faster than a traditional micro-batch architecture and, in its own benchmarks on customer feature-computation workloads, up to 92% faster than Apache Flink. Both comparisons are vendor figures.

For data engineering teams, the change means a pipeline that feeds a fraud model, a personalization service or an alerting system can now stay inside Lakeflow, with the same declarative definitions and monitoring as the batch and streaming pipelines around it, without moving the low-latency path to a separate stream-processing stack.

··········

HOW DATABRICKS REAL-TIME MODE WORKS IN LAKEFLOW PIPELINES.

The documentation describes real-time mode as a specialized continuous trigger. It runs long batches, five minutes by default, and processes data inside each batch as it arrives at the source. Progress is checkpointed and metrics are published between batches, so the batch length controls how often the pipeline records its state and how long a replay takes after a failure.

Because all stages of the query run at the same time, the compute behind the pipeline must provide enough task slots to cover every task in a batch. Data moves between stages through a streaming shuffle rather than waiting for a stage to finish. Databricks positions micro-batch processing, which remains the default trigger, for analytical ETL, transformations and medallion architectures where seconds or minutes of latency are acceptable, and real-time mode for sub-second operational decisions. The Structured Streaming documentation also notes that real-time mode needs dedicated compute resources and that micro-batch recovers faster because it checkpoints more often.

··········

REQUIREMENTS FOR REAL-TIME MODE IN LAKEFLOW PIPELINES.

The Lakeflow documentation, last updated on October 29, 2026, lists a short set of requirements. The pipeline must run in continuous mode, real-time mode must be switched on in the pipeline's Spark configuration, and each low-latency flow must be defined as an update flow with a real-time trigger that writes to a sink. Kinesis polling mode requires Databricks Runtime 19 or above. The page does not mention editions or pricing tiers.

........

Item	Documented status
Release status	Generally available (October 29, 2026)
Compute	Classic and serverless
Pipeline mode	Continuous
Default batch length	5 minutes
Lowest latency cited	5 milliseconds end to end
Kinesis polling	Databricks Runtime 19 or above
Pricing	Not disclosed in the announcement

........

Two capabilities remain in Beta: running multiple real-time flows in one pipeline without task-slot contention, and adaptive repartitioning of the source stage. Both require serverless pipelines on the preview channel. On classic compute and on the current serverless channel, the documentation recommends one real-time flow per pipeline, because additional flows compete for task slots and raise latency.

··········

ENABLING REAL-TIME MODE IN A LAKEFLOW PIPELINE.

The pipeline settings need two values: continuous execution and the real-time Spark flag. In the pipeline JSON the documentation shows them as follows.

{
  "continuous": true,
  "spark_conf": {
    "spark.databricks.streaming.realTimeMode.enabled": "true"
  }
}

In the pipeline code, written in Python, the sink is defined with dp.create_sink() or with the dp.foreach_sink decorator for custom destinations, and the flow is declared with the dp.update_flow decorator pointing its target at that sink. The flow carries its own trigger settings in spark_conf.

{
  "pipelines.trigger": "RealTime",
  "pipelines.trigger.interval": "5 minutes"
}

The interval value sets the checkpoint interval and is optional, since five minutes is the default. Shorter intervals improve recoverability but add checkpointing overhead; longer intervals reduce overhead but lengthen recovery and delay the metrics. The documentation gives all examples in Python and JSON and provides no SQL syntax for real-time flows, so teams that define their pipelines only in SQL will need Python for this path.

··········

SUPPORTED SOURCES, SINKS AND OPERATORS.

Real-time mode in Lakeflow pipelines reads from and writes to message buses. Delta tables are supported neither as a source nor as a sink, which keeps real-time flows separate from the lakehouse tables that most analysts query.

........

Connector	Source	Sink
Apache Kafka	Yes	Yes
Azure Event Hubs (Kafka connector)	Yes	Yes
AWS MSK	Yes	Yes
Amazon Kinesis	Yes	No
ForEach sink	Not applicable	Yes
Delta	No	No

........

Custom destinations such as Lakebase, Redis or other OLTP stores are reached through the ForEach sink, where a writer implements open, process and close methods. Databricks lists a native Lakebase sink with built-in batching, retries and Unity Catalog-managed authentication among its planned improvements, together with packing several real-time flows into one pipeline and automatic partition tuning.

On the operator side, the Lakeflow page lists selection and projection, Scala and Python UDFs with limitations, sum, count, max, min and avg aggregations, tumbling and sliding windows, dropDuplicates, broadcast table joins, transformWithState, union with limitations, and forEach. Session windows, dropDuplicatesWithinWatermark, stream-to-stream joins, flatMapGroupsWithState, mapPartitions, forEachBatch and transformWithStateInPandas are not supported. With transformWithState, handleInputRows runs once per row, event-time timers are unavailable and processing-time timers fire at the end of a long batch if no data has arrived. For pandas UDFs, the documentation suggests setting spark.sql.execution.arrow.maxRecordsPerBatch to 1 for the lowest latency, or to 100 or more when throughput matters more.

··········

SIZING TASK SLOTS FOR A REAL-TIME FLOW.

On classic compute, the cluster must have enough task slots for all stages to run concurrently. The examples in the documentation add one slot per source partition to one slot per shuffle partition in each later stage: a single-stage stateless flow with maxPartitions set to 8 needs 8 slots, a two-stage stateful flow with 20 shuffle partitions needs 28, and a three-stage flow with two 20-partition shuffles needs 48. When maxPartitions is not set, the Kafka topic's partition count applies.

The figures below apply the same rule to an illustrative Kafka topic with 12 partitions, with shuffle partitions set to match the input partitions as the aggregation example in the documentation does.

........

Illustrative flow shape	Source tasks	Shuffle tasks	Task slots needed
Stateless filter and projection	12	0	12
One stateful aggregation	12	12	24
Two shuffle stages	12	24	36

........

The checkpoint interval has a similar arithmetic. At the default of five minutes a flow checkpoints 12 times per hour and 288 times per day; at one minute the counts rise to 60 and 1,440, with more overhead, while at ten minutes they fall to 6 and 144, with longer replays after a failure. Databricks recommends benchmarking the interval against the actual workload.

On the current serverless channel, Databricks sizes the compute, but source task counts stay fixed, so maxPartitions for Kafka, or maxPartitions and shardsPerTask for Kinesis, should be set for peak traffic. On the preview channel, Databricks reserves task slots per flow and adjusts source task counts automatically when those options are left unset; shuffle partitions are still not adjusted.

··········

MONITORING LATENCY IN REAL-TIME PIPELINES.

Latency metrics appear in StreamingQueryProgress under latencies, readable through a StreamingQueryListener or the query's lastProgress. Three measures are reported at p50, p90, p95 and p99: processingLatencyMs, from the moment the flow reads a record to the moment it is fully processed; sourceQueuingLatencyMs, from the write to the message bus to the first read by the flow; and e2eLatencyMs, from production at the source to full processing. Databricks reports that its own Feature Store reaches a p99 of 200 milliseconds from Kafka to the online store on real-time mode.

Teams already using Databricks for semantic modeling, such as the relationships between Unity Catalog metric views released the day after, will find real-time mode on the opposite end of the platform: it serves applications and models that act on events, while dashboards and AI assistants that query governed tables, including the ChatGPT Work data agent connected to Databricks, work on Delta tables, which real-time flows neither read nor write.

··········

DATABRICKS REAL-TIME MODE: WHAT TO CHECK BEFORE ADOPTING IT.

Confirmed by Databricks: general availability in Lakeflow pipelines on classic and serverless compute, continuous mode as a prerequisite, the configuration keys above, Kafka, Event Hubs and MSK as sources and sinks, Kinesis as a source only, and the ForEach sink for custom stores. The latency comparisons with micro-batch and Flink are Databricks' own figures.

What depends on the workspace: the serverless channel in use, which decides whether multiple flows per pipeline and automatic source repartitioning are available; the number of task slots on classic clusters; the Databricks Runtime version for Kinesis polling; and whether the pipeline code can be written in Python. What is not disclosed in the announcement: specific pricing for real-time flows beyond the reference to serverless billing by cores used, and a date for the native Lakebase sink. Before moving a production flow, teams should verify that every operator in the query appears on the supported list, size the slots or partitions for peak traffic, and measure e2eLatencyMs at p99 on their own data.

··········

FOLLOW US FOR MORE.

·····

DATA STUDIOS

·····

datastudios.org