A list of useful resources to learn Data Engineering from scratch
-
Updated
Jan 13, 2021
#
data-pipeline
Here are 215 public repositories matching this topic...
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
data-science
machine-learning
data-visualization
data-engineering
cloud-computing
data-analysis
data-processing
data-pipeline
-
Updated
Oct 14, 2020 - Jupyter Notebook
A lightweight stream processing library for Go
redis
kafka
pipeline
etl
websocket
data-stream
pipelines
streams
stream-processing
pulsar
aerospike
kafka-streams
data-pipeline
streaming-data
nats-streaming
stream-processor
apache-pulsar
-
Updated
Jan 8, 2021 - Go
Smarter data pipelines for audio.
-
Updated
Jan 22, 2021 - Python
A list about Apache Kafka
infrastructure
kafka
apache-spark
stream-processing
apache-kafka
kafka-streams
data-processing
data-pipeline
streaming-data
-
Updated
Jan 21, 2021
Streaming reactive and dataflow graphs in Python
python
streaming
kafka
stream
asynchronous
websockets
python3
lazy-evaluation
data-pipeline
reactive-data-streams
python-data-streams
-
Updated
Jan 17, 2021 - Python
Content for architecting a data science platform for products using Luigi, Spark & Flask.
-
Updated
Jan 27, 2020 - Jupyter Notebook
Visualise your Kedro data pipelines.
react
open-source
data-visualization
hacktoberfest
data-pipeline
mckinsey
kedro
kedro-plugin
quantumblack
-
Updated
Jan 22, 2021 - JavaScript
-
Updated
Jan 11, 2021 - Vue
Tools for ASR Corpus Generation from Online Video
-
Updated
Feb 10, 2019 - Python
aeksco
commented
Apr 22, 2020
There's a warning note in README.md detailing:
Warning - the AnalyzeDocument process from AWS Textract costs $50 per 1,000 PDF pages. Be careful when deploying this CDK stack as you could unintentionally rack up an expensive AWS bill quickly if you're not paying attention.
This might not be enough - if a user finds this project and doesn't read the documentation, they could inadvertently
Bulk Stash is a docker rclone service to sync, or copy, files between different storage services. For example, you can copy files either to or from a remote storage services like Amazon S3 to Google Cloud Storage, or locally from your laptop to a remote storage.
docker
sync
docker-image
s3
google-cloud-storage
google-cloud
rclone
docker-service
amazon-web-services
data-pipeline
oracle-cloud
storage-service
sftp-synchronisation
docker-rclone
-
Updated
Sep 21, 2020 - Shell
1
wybiral
commented
Jul 1, 2018
The serial package being used supports multiple options beyond the baud rate which might be useful to expose as optional arguments.
We should add more options to the serial protocol. See comment below.
Serverless Data Pipeline powered by Kinesis Firehose, API Gateway, Lambda, S3, and Athena
-
Updated
Oct 22, 2018 - Python
Example end to end data engineering project.
python
redis
elasticsearch
airflow
kafka
big-data
mongodb
scraping
django-rest-framework
s3
data-engineering
minio
kafka-connect
hacktoberfest
data-pipeline
debezium
-
Updated
Jan 8, 2021 - Python
一个基于浏览器环境的数据采集SDK
-
Updated
Dec 31, 2020 - JavaScript
vitaliihonta
commented
Feb 1, 2019
Allow trembita integration with grafana and prometheus for monitoring pipeline performance and visualisation of pipeline itself
Streaming data changes to a Data Lake with Debezium and Delta Lake pipeline
-
Updated
Jun 18, 2020 - HTML
Framework which helps you to make parallel/distributed calculations using data pipelines
-
Updated
Oct 22, 2020 - Python
Copy Pandas DataFrames and HDF5 files to PostgreSQL database
postgres
sqlalchemy
database
etl
postgresql
pandas
python3
scientific-computing
hdf5
data-pipeline
pytables
hdf
sqlalchemy-database
-
Updated
Oct 19, 2020 - Python
A Machine Learning and Real-Time Data Analytics Workshop
-
Updated
Jan 25, 2020 - Jupyter Notebook
(deprecated) A fast and memory-efficient Python data engineering framework for machine learning.
-
Updated
Sep 30, 2019 - Python
Network traffic data pipeline for real-time predictions and building datasets for deep neural networks
redis
flask
machine-learning
django
ai
spring
vue
zap
tensorflow
django-rest-framework
swagger
keras
react-redux
ip
ethernet
network-analysis
data-pipeline
network-security
machine-learning-defense
ai-defense
-
Updated
Sep 4, 2020 - Python
Saisoku is a Python module that helps you build complex pipelines of batch file/directory transfer/sync jobs.
python
sync
pipeline
scheduling
s3
orchestration-framework
rclone
luigi
tornado
data-transfer
file-transfer
transfer-files
data-synchronization
data-pipeline
sync-directories
luigi-pipeline
transfer-server
directory-transfer
-
Updated
Sep 12, 2020 - Python
A web service for disambiguating and canonically storing entities.
-
Updated
Jul 3, 2019 - Python
Kestra, the modern, scalable orchestrator & scheduler open source platform.
workflow
pipeline
workflow-engine
scheduler
orchestration
dataflow
data-pipeline
workflow-automation
kestra
-
Updated
Jan 22, 2021 - Java
Improve this page
Add a description, image, and links to the data-pipeline topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-pipeline topic, visit your repo's landing page and select "manage topics."
Write unit test coverage for
SafeDatasetandSafeDataLoader, along with the functions inutils.py.