Spark Structured Streaming, Kafka, Connect and Real-Time Pipelines

structured streaming

Spark Structured Streaming models incoming data as an incrementally updated table and executes supported DataFrame operations as a fault-tolerant streaming.

unbounded table model

Structured Streaming treats an input stream as an unbounded table and maintains incremental results as new rows are appended.

event time and processing time

Event time is when the source event occurred, while processing time is when the engine handled it

watermark

A watermark estimates how far event time has progressed and bounds how long state is retained for late records.

output mode

A streaming output mode chooses whether each trigger emits only new rows, changed result rows or the complete result table where supported.

checkpoint

A streaming checkpoint persists offsets, progress and state-store metadata so a query can resume consistently after driver failure.

Kafka broker topic partition

A Kafka broker stores replicated topic partitions, each an ordered append-only log whose offsets identify record positions.

producer and consumer group

A Kafka producer chooses a topic partition for records, while consumers in one group divide those partitions so each partition has one active group member.

Kafka offset and delivery semantics

An offset identifies a record position in a Kafka partition; commit and retry behavior determines at-most-once, at-least-once or exactly-once processing.

Spark Kafka integration

Spark’s Kafka connector maps Kafka records into streaming DataFrame rows and coordinates source offsets with checkpointed query progress.

Apache Kafka

What is Kafka?

Apache Kafka is a distributed event streaming platform for:

• High-throughput, low-latency message streaming
• Real-time data pipelines
• Decoupling of data producers and consumers

Kafka Core Concepts

PRODUCER (writes data)     KAFKA CLUSTER      CONSUMER (reads data)
+-----------+              +----------+        +----------+
| App       |   produces   | Topic    |  reads | Analytics|
| Database  |──────────────| (logs)   |────────| ML model |
| IoT device|              | Partition|        | Dashboard|
|           |              | 1, 2, 3  |        |          |
+-----------+              +----------+        +----------+
                            Brokers (servers)

Kafka Use Cases

Kafka vs Traditional Messaging

Big Data in the Cloud

AWS Big Data Services

Google Cloud Big Data

Azure Big Data