Hasan OzganHasan Ozgan
← back to projects

Data Streaming Pipeline

Real-time event-driven streaming pipeline implementing Pekko Streams → Kafka → Apache Flink → Apache Iceberg/MinIO data lake architecture.

Completed
Repository
scalajavapekko-streamskafkaflinkapache-icebergminioparquetstreamingdata-engineering

Real-Time Data Streaming Pipeline

A production-grade end-to-end streaming data pipeline demonstrating modern event-driven architectures and lakehouse ingestion: Pekko Streams → Kafka → Apache Flink → Apache Iceberg / MinIO.

Architecture & Pipeline Stages

The pipeline decouples ingestion, validation, stream transformation, and windowed aggregation across four dedicated processing phases:

  1. Simulation Producer (Scala 3 & Pekko Streams): Generates high-volume synthetic data files (CSV, JSON, LOG, TXT) with customizable arrival distributions into MinIO raw storage.
  2. Source Reader (Scala 3 & Pekko Streams): Continuously monitors object storage buckets, translates incoming files into structured FileEvent messages, publishes them to Kafka (raw-events-topic), and archives processed objects.
  3. Transformer & Validator (Scala 3 & Pekko Streams): Consumes raw events, downloads file contents, extracts cryptographic checksums and metadata, enriches event schemas, and routes valid records to transformer-topic while dispatching anomalies to a Dead Letter Queue (DLQ).
  4. Windowed Aggregator (Java 17 & Apache Flink): Consumes transformed streams and executes 5-minute sliding window aggregations grouped by event type, file type, and processing status.

Data Lake & Storage Layer

  • Apache Iceberg REST Catalog: Aggregated window statistics are committed to Apache Iceberg tables in columnar Apache Parquet format, partitioned by window_start.
  • MinIO Object Storage: Serves as the S3-compatible data lake backing both raw file ingestion and the Iceberg warehouse.
  • Kafka Event Backbone: Multi-partitioned topics (raw-events-topic, transformer-topic, dead-letter-queue) with consumer group scaling.

Technology Stack

  • Stream Ingestion & Transformation: Scala 3, Apache Pekko Streams, Akka/Pekko HTTP
  • Stream Processing & Aggregation: Java 17, Apache Flink 2.1
  • Messaging & Event Streaming: Apache Kafka 4.1.1, Kafdrop UI
  • Table Format & Lakehouse: Apache Iceberg REST Catalog, Apache Parquet
  • Storage & Infrastructure: MinIO (S3), Docker Compose, Nix