Skip to content

/scio

A Scala API for Apache Beam and Google Cloud Dataflow.
scala bigquery google-cloud beam dataflow batch streaming data ml scio
  1. Scala 77.0%
  2. Java 22.4%
  3. Other 0.6%
Branch: master
Find File
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Download ZIP

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop...

If nothing happens, download GitHub Desktop and try again.

Launching Xcode...

If nothing happens, download Xcode and try again.

Launching Visual Studio...

If nothing happens, download the GitHub extension for Visual Studio and try again.

@nevillelyh
nevillelyh fix #1831, Avro map string key cast exception (#1832) 
* fix #1831, Avro map string key cast exception

* Update scio-avro/src/test/scala/com/spotify/scio/avro/types/ConverterProviderTest.scala

Co-Authored-By: nevillelyh <neville.lyh@gmail.com>
Latest commit 9a49f60 Apr 10, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci use golang image for CircleCI deploy, fix #1772 (#1779) Mar 29, 2019
project
scalafix
scio-avro/src fix #1831, Avro map string key cast exception (#1832) Apr 10, 2019
scio-bench
scio-bigquery/src
scio-bigtable/src Enable -Ywarn-value-discard (#1815) Apr 8, 2019
scio-cassandra Remove the usage of Future around ScioContext and Tap's (#1666) Mar 29, 2019
scio-core/src
scio-elasticsearch
scio-examples/src
scio-extra/src
scio-jdbc/src
scio-jmh/src/test
scio-macros/src/main/scala/com/spotify/scio Add schema and row coders (#1698) Mar 29, 2019
scio-parquet/src
scio-repl/src
scio-schemas/src/main Add coder for java Enum (#1434) Oct 8, 2018
scio-spanner/src
scio-tensorflow/src Enable -Ywarn-value-discard (#1815) Apr 8, 2019
scio-test/src
scripts
site/src/paradox Enable -Ywarn-value-discard (#1815) Apr 8, 2019
.gitignore
.jvmopts
.scalafix.conf Add scalafix rules to scio itself (#1594) Dec 21, 2018
.scalafmt.conf Add scalafmt (#1372) Sep 18, 2018
CONTRIBUTING.md
LICENSE Add license file Feb 19, 2016
NOTICE
README.md
apache.sbt
build.sbt
codecov.yaml Disable codecov github status (#1545) Nov 19, 2018
scalastyle-config.xml
version.sbt

README.md

Scio

Build Status codecov.io GitHub license Maven Central Join the chat at https://gitter.im/spotify/scio

Scio Logo

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]
Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Features

  • Scala API close to that of Spark and Scalding core APIs
  • Unified batch and streaming programming model
  • Fully managed service*
  • Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable
  • HDFS, JDBC, TensorFlow TFRecords, Cassandra, Elasticsearch and Parquet I/O
  • Interactive mode with Scio REPL
  • Type safe BigQuery
  • Integration with Algebird and Breeze
  • Pipeline orchestration with Scala Futures
  • Distributed cache

* provided by Google Cloud Dataflow

Quick Start

Use our giter8 template to quickly setup a project:

sbt new spotify/scio.g8

Compile it:

sbt pack

Run the included word count example:

target/pack/bin/word-count --output=wc

Inspect the results:

cat wc/part-00000-of-00001.txt

Documentation

Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.

Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.

Artifacts

Scio includes the following artifacts:

  • scio-core: core library
  • scio-test: test utilities, add to your project as a "test" dependency
  • scio-avro: add-on for Avro, included in scio-core but can also be used standalone
  • scio-bigquery: add-on for BigQuery, included in scio-core but can also be used standalone
  • scio-bigtable: add-on for Bigtable
  • scio-cassandra2: add-on for Cassandra 2.x
  • scio-cassandra3: add-on for Cassandra 3.x
  • scio-elasticsearch2: add-on for Elasticsearch 2.x
  • scio-elasticsearch5: add-on for Elasticsearch 5.x
  • scio-elasticsearch6: add-on for Elasticsearch 6.x
  • scio-extra: extra utilities for working with collections, Breeze, etc.
  • scio-hdfs: add-on for HDFS IO
  • scio-jdbc: add-on for JDBC IO
  • scio-parquet: add-on for Parquet
  • scio-tensorflow: add-on for TensorFlow TFRecords IO and prediction

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

You can’t perform that action at this time.