data-processing

Here are 725 public repositories matching this topic...

onceupon / Bash-Oneliner

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

linux shell bash terminal system hardware grep data-processing variables xargs xwindow one-liners linux-administration oneliner-commands shell-oneliner

Updated Sep 19, 2022

johnkerl / miller

Star

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

Updated Dec 28, 2022
Go

activeloopai / deeplake

Star

Data Lake for Deep Learning. Build, manage, query, version, & visualize datasets. Stream data real-time to PyTorch/TensorFlow. https://activeloop.ai

python data-science machine-learning ai computer-vision deep-learning tensorflow cv jupyter-notebook image-processing ml pytorch datasets data-processing data-version-control datalake data-centric mlops lakehouse

Updated Dec 28, 2022
Python

NVIDIA / DALI

Star

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

python machine-learning deep-learning neural-network mxnet gpu image-processing pytorch gpu-tensorflow data-processing data-augmentation audio-processing paddle image-augmentation fast-data-pipeline

Updated Dec 28, 2022
C++

TomWright / dasel

Sponsor

Star

Select, put and delete data from JSON, TOML, YAML, XML and CSV files with a single tool. Supports conversion between formats and can be used as a Go package.

config go cli golang yaml toml parser json query xml configuration update selector data-structures data-wrangling devops-tools data-processing yaml-processor json-processing

Updated Dec 27, 2022
Go

asyml / texar

Star

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

python machine-learning natural-language-processing deep-learning tensorflow machine-translation text-generation data-processing bert text-data dialog-systems gpt-2 texar xlnet casl-project

Updated Aug 26, 2021
Python

dashbitco / broadway

Star

Concurrent and multi-stage data ingestion and data processing with Elixir

elixir broadway concurrent data-processing genstage data-ingestion

Updated Dec 10, 2022
Elixir

unionai-oss / pandera

Star

A light-weight, flexible, and expressive statistical data testing library

testing schema validation data-validation pandas-dataframe assertions pandas testing-tools data-processing dataframes data-cleaning hypothesis-testing data-verification pandas-validation data-check data-assertions dataframe-schema pandas-validator

Updated Dec 20, 2022
Python

microsoft / DialoGPT

Star

Large-scale pretraining for dialogue

machine-learning dialogue text-generation pytorch transformer data-processing text-data gpt-2 dialogpt

Updated Oct 17, 2022
Python

python-bonobo / bonobo

Star

Extract Transform Load for Python 3.5+

automation parallelization python3 data-processing bonobo extract-transform-load

Updated Dec 8, 2022
Python

GoogleCloudPlatform / data-science-on-gcp

Star

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

data-science machine-learning data-visualization data-engineering cloud-computing data-analysis data-processing data-pipeline

Updated Dec 20, 2022
Jupyter Notebook

GoogleCloudPlatform / DataflowJavaSDK

Star

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

data-science data-mining big-data data-analysis google-cloud-dataflow data-processing

Updated Nov 25, 2020

asyml / texar-pytorch

Star

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

python machine-learning natural-language-processing deep-learning machine-translation text-generation pytorch data-processing bert text-data dialog-systems roberta gpt-2 texar xlnet casl-project texar-pytorch

Updated Apr 14, 2022
Python

hstreamdb / hstream

Star

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

iot distributed-systems haskell streaming real-time sql database kafka scale stream-processing distributed-database realtime-database data-processing financial-analysis streaming-data materialized-view iot-database hstreamdb streaming-database

Updated Dec 28, 2022
Haskell

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.