dataengineering

Here are 251 public repositories matching this topic...

datafold / data-diff

Efficiently diff rows across two different databases.

mysql postgres data-science sql database postgresql snowflake data-engineering rdbms oracle-database data-quality dataengineering trino dataquality data-quality-monitoring databricks-sql

Updated Sep 15, 2022
Python

open-metadata / OpenMetadata

Star

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

Updated Sep 15, 2022
TypeScript

zinggAI / zingg

Star

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Updated Sep 7, 2022
Java

chaos-genius / chaos_genius

Star

ML powered analytics engine for outlier detection and root cause analysis.

Updated Sep 12, 2022
Python

meltano / meltano

Star

Your open source DataOps Platform Infrastructure to let you manage all the data tools in your stack in one place, and turn them into your ideal end-to-end data platform

Updated Sep 15, 2022
Python

Datavault-UK / dbtvault

Star

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

sql etl snowflake datawarehousing dbt elt datawarehouse datalake dataengineering datavault datavault20 data-vault

Updated Sep 15, 2022

awslabs / aws-orbit-workbench

Star

A Data Platform built for AWS, powered by Kubernetes.

kubernetes aws jupyter analytics gpu jupyterhub data-analysis redshift mach workbench datalake dataengineering eks eks-cluster orbit-workbench

Updated Aug 24, 2022
Python

awslabs / aws-ddk

Star

An open source development framework to help you build data workflows and modern data architecture on AWS.

python aws dataops dataengineering

Updated Sep 1, 2022
Python

prodmodel / prodmodel

Star

Build, test, deploy, iterate - Dev and prod tool for data science pipelines

productivity data-science data production build-automation datascience build-tool build-system dataengineering dataeng

Updated Jun 21, 2022
Python

kislerdm / data-engineering-interviews

Star

Data engineering interviews Q&A for data community by data community

python linux opensource sql kafka spark interview-questions dataengineering

Updated Jun 7, 2020
Python

Finance-And-ML / US-Stock-Prediction-Using-ML-And-Spark

Star

Predict stock price based on financial news feeds

scala spark stock-price-prediction tableau webscraping dataengineering

Updated Apr 6, 2018
Jupyter Notebook

olist / work-at-olist-data

Star

Apply for a job at Olist's Data Team: https://olist.gupy.io/

python data r sql analytics julia pandas datascience dataset machinelearning dataengineering

Updated Mar 4, 2022

Spratiher9 / SparkDataset

Star

Instant search for and access to many datasets in Pyspark.

python data benchmark data-mining r spark python3 pyspark dataset standard quickstart data-analysis datasets instantsearch benchmark-framework dataengineering easy-access-application

Updated Nov 5, 2021
Python

Wittline / apache-spark-docker

Sponsor

Star

Dockerizing an Apache Spark Standalone Cluster

docker apache-spark hive docker-compose pyspark hdfs hadoop-cluster hue hadoop-docker dataengineering hive-metastore dataengineer

Updated Jun 29, 2022
VBA

WaylonWalker / kedro-static-viz

Sponsor

Star

kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.

python data datapipeline dataengineering kedro kedro-plugin

Updated Jul 20, 2022
Python

sarahmk125 / airflow-docker-metrics

Star

data grafana orchestration statsd-metrics dataengineering workflow-orchestration airflow-metrics statsd-format airflow-containers

Updated Apr 21, 2022
Python

CynthiaKoopman / Forecasting-Solar-Energy

Star

Forecasting Solar Power: Analysis of using a LSTM Neural Network

data-science lstm forecasting machinelearning deeplearning solar-energy renewable-energy dataengineering neuralnetworks solarpower electricity-grid forecasting-solar-power

Updated Feb 7, 2020
Jupyter Notebook

abhishek-ch / data-machinelearning-the-boring-way

Star

Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.

kubernetes machine-learning datascience dataengineering data-infrastructure mlops

Updated May 15, 2022
Python

Wittline / data-engineer-challenge

Sponsor

Star

Challenge Data Engineer

docker docker-compose postgresql data-engineering data-pipeline dataengineering fastapi

Updated Jun 13, 2022
Python

WaylonWalker / kedro-action

Sponsor

Star

A GitHub Action to lint, test, build-docs, package, and run your kedro pipelines. Supports any Python version you'll give it (that is also supported by pyenv).

actions datapipeline dataengineering kedro

Updated Aug 30, 2022
Shell

Improve this page

Add a description, image, and links to the dataengineering topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the dataengineering topic, visit your repo's landing page and select "manage topics."

Learn more