data-pipeline

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

Change data capture for a variety of databases. Please log issues at https://github.com/debezium/dbz/issues.

Flink CDC is a streaming data integration tool

BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

zerocode-tdd is a community-developed, free, open-source, outcome-driven automated testing framework for Data Pipelines, ETL, REST API, Kafka(Data Streams), Databases and Load scenarios. all defined in simple JSON or YAML — with zero coding.

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

Harness for building data engineering agents designed around human control, correctness, and safety.

DataY 是一套轻量、高性能的数据平台，一个 JAR、自研流批一体处理引擎，自研分布式调度框架，在 2 核 4G 的机器上 就能跑通 数据集成 → 数据建模 → 数据开发 → 任务调度 → 数据应用 的完整链路。

By Smart Shaped s.r.l. (https://www.smartshaped.com/)

Kafka Streams made easy with a YAML file

cron replacement to schedule complex data workflows

Library for describing data transformation pipelines by compositing simple reusable components.

Data pipeline using Apache Kafka, Apache Spark and HDFS

SeaTunnel is a distributed, high-performance data integration platform for the synchronization and transformation of massive data (offline & real-time).

A composable, stage-based streaming ETL framework with built-in violation detection, delta analysis, and data lake layer processing. Designed for compliance monitoring, data quality enforcement, and supply chain traceability.

⚡ DataLink is a lightweight data integration framework build on top of DataX, Spark and Flink

A real-time data pipeline using Kafka, Spark, and Cassandra for processing and storing credit card expenses. Includes a Spring Boot application for retrieving personnel data from MySQL, storing images in S3, and displaying employee details with expense reports on a web interface.

An end to end data pipeline with Kafka Spark Streaming Integration

A source-available JVM pipeline kernel for policy-aware, benchmarkable operational data movement.

Real estate market intelligence app with native Android and responsive web clients, powered by Redfin data, Python/Flask, Java/Gradle, Cloudflare Pages, and GitHub Actions.