#

dataengineering

Here are 261 public repositories matching this topic...

datafold / data-diff

Efficiently diff data in or across relational databases

mysql postgres data-science sql database postgresql snowflake data-engineering rdbms oracle-database data-quality dataengineering trino dataquality data-quality-monitoring databricks-sql

Updated Oct 20, 2022
Python

OpenMetadata

open-metadata / OpenMetadata

Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.

Updated Oct 20, 2022
TypeScript

zingg

zinggAI / zingg

Scalable identity resolution, entity resolution, data mastering and deduplication using ML

Updated Oct 19, 2022
Java

meltano / meltano

Your CLI for ELT+. It's open source, flexible, scales to your needs. Confidently move, transform, and test your data using tools you know with a data engineering workflow you’ll love.

Updated Oct 20, 2022
Python

dbtvault

Datavault-UK / dbtvault

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered trademark of dbt Labs)

sql etl snowflake datawarehousing dbt elt datawarehouse datalake dataengineering datavault datavault20 data-vault

Updated Oct 19, 2022

awslabs / aws-orbit-workbench

A Data Platform built for AWS, powered by Kubernetes.

kubernetes aws jupyter analytics gpu jupyterhub data-analysis redshift mach workbench datalake dataengineering eks eks-cluster orbit-workbench

Updated Sep 19, 2022
Python

aws-ddk

awslabs / aws-ddk

An open source development framework to help you build data workflows and modern data architecture on AWS.

python aws dataops dataengineering

Updated Oct 20, 2022
Python

prodmodel / prodmodel

Build, test, deploy, iterate - Dev and prod tool for data science pipelines

productivity data-science data production build-automation datascience build-tool build-system dataengineering dataeng

Updated Jun 21, 2022
Python

data-engineering-interviews

kislerdm / data-engineering-interviews

Data engineering interviews Q&A for data community by data community

python linux opensource sql kafka spark interview-questions dataengineering

Updated Jun 7, 2020
Python

Finance-And-ML / US-Stock-Prediction-Using-ML-And-Spark

Predict stock price based on financial news feeds

scala spark stock-price-prediction tableau webscraping dataengineering

Updated Apr 6, 2018
Jupyter Notebook

olist / work-at-olist-data

Apply for a job at Olist's Data Team: https://olist.gupy.io/

python data r sql analytics julia pandas datascience dataset machinelearning dataengineering

Updated Mar 4, 2022

apache-spark-docker

Wittline / apache-spark-docker

Sponsor

Dockerizing an Apache Spark Standalone Cluster

docker apache-spark hive docker-compose pyspark hdfs hadoop-cluster hue hadoop-docker dataengineering hive-metastore dataengineer

Updated Jun 29, 2022
VBA

Spratiher9 / SparkDataset

Instant search for and access to many datasets in Pyspark.

python data benchmark data-mining r spark python3 pyspark dataset standard quickstart data-analysis datasets instantsearch benchmark-framework dataengineering easy-access-application

Updated Oct 6, 2022
Jupyter Notebook

kedro-static-viz

WaylonWalker / kedro-static-viz

Sponsor

kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.

python data datapipeline dataengineering kedro kedro-plugin

Updated Oct 6, 2022
Python

sarahmk125 / airflow-docker-metrics

data grafana orchestration statsd-metrics dataengineering workflow-orchestration airflow-metrics statsd-format airflow-containers

Updated Apr 21, 2022
Python

CynthiaKoopman / Forecasting-Solar-Energy

Forecasting Solar Power: Analysis of using a LSTM Neural Network

data-science lstm forecasting machinelearning deeplearning solar-energy renewable-energy dataengineering neuralnetworks solarpower electricity-grid forecasting-solar-power

Updated Feb 7, 2020
Jupyter Notebook

minhadona / Data_Engineer_Interview_Challenges

Found a data engineering challenge or participated in a selection process ? Share with us!

Updated Oct 17, 2022
Python

abhishek-ch / data-machinelearning-the-boring-way

Build & Learn Data Engineering,Machine Learning over Kubernetes. No Shortcut approach.

kubernetes machine-learning datascience dataengineering data-infrastructure mlops

Updated May 15, 2022
Python

noahgift / data-engineering-and-dataops

Sponsor

Duke MIDS: Data Engineering and DataOps Course

data-science data cloud course book dataops software-engineering duke dataengineering mlops

Updated Sep 15, 2022
HTML

Wittline / data-engineer-challenge

Sponsor

Challenge Data Engineer

docker docker-compose postgresql data-engineering data-pipeline dataengineering fastapi

Updated Jun 13, 2022
Python

Improve this page

Add a description, image, and links to the dataengineering topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the dataengineering topic, visit your repo's landing page and select "manage topics."