DataOps

Kubernetes-native workflow automation platform for complex, mission-critical data and ML processes at scale. It has been battle-tested at Lyft, Spotify, Freenome, and others and is truly open-source.

Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging.

The open standard for data logging

Kafka Docker for development. Kafka, Zookeeper, Schema Registry, Kafka-Connect, Landoop Tools, 20+ connectors

✨ Argilla: Open-source platform empowering teams to build better language models through human feedback

Modern columnar data format for ML implemented in Rust. Convert from parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB, Polars, Pyarrow, with more integrations coming..

Cloud Native DataOps & AIOps Platform | 云原生数智运维平台

Open-source data observability for analytics engineers.

Streaming reference architecture for ETL with Kafka and Kafka-Connect. You can find more on http://lenses.io on how we provide a unified solution to manage your connectors, most advanced SQL engine for Kafka and Kafka Streams, cluster monitoring and alerting, and more.

Extract & Load /with joy/ — CLI & version control for ELT without limitations. No more black box. Let your creativity flow.

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Support agile DataOps Based on DataX and Flink-CDC, Chunjun with Web-UI

A list of tools for annotating data, managing annotations, etc.

Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.

The open-source pipeline and storage engine for security.

DataOps for the Modern Data Warehouse on Microsoft Azure. https://aka.ms/mdw-dataops.

📙 Awesome Data Catalogs and Observability Platforms.

SQLMesh is a DataOps framework that brings the benefits of DevOps to data teams. It enables data scientists, analysts, and engineers to efficiently run and deploy data transformations written in SQL or Python.

Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.

Build, run and manage your data pipelines with Python or SQL on any cloud