Skip to content

/ scio

A Scala API for Apache Beam and Google Cloud Dataflow.
scala bigquery google-cloud beam dataflow batch streaming data ml scio
Scala Java Other
  1. Scala 74.1%
  2. Java 25.1%
  3. Other 0.8%
Branch: master
Find file
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Open in Desktop Download ZIP

Latest commit

@scala-steward
scala-steward Update sbt-scalafix to 0.9.16 (#3014)
Latest commit 2dd3556 May 27, 2020

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.circleci Update scala to 2.13.2 (#2951) May 11, 2020
project Update sbt-scalafix to 0.9.16 (#3014) May 27, 2020
scalafix Update sbt-scalafix to 0.9.16 (#3014) May 27, 2020
scio-avro/src Fix deprecation warns (#2966) May 13, 2020
scio-bench Remove scalastyle (#2426) Nov 20, 2019
scio-bigquery/src Use BigQuery client project consistently (#2995) May 21, 2020
scio-bigtable/src Fix deprecation warns (#2966) May 13, 2020
scio-cassandra/cassandra3/src Fix deprecation warns (#2966) May 13, 2020
scio-core/src Add SCollection#groupMap and SCollection#groupMapReduce operations (#… May 20, 2020
scio-elasticsearch Fix deprecation warns (#2966) May 13, 2020
scio-examples/src implement SMB "least bucket replication" for SortedBucketSource (#2953) May 15, 2020
scio-extra/src Fix deprecation warns (#2966) May 13, 2020
scio-jdbc/src Apply javafmt (#2679) Feb 25, 2020
scio-jmh Fix deprecation warns (#2966) May 13, 2020
scio-macros/src/main/scala/com/spotify/scio
scio-parquet/src Fix deprecation warns (#2966) May 13, 2020
scio-repl/src Fix deprecation warns (#2966) May 13, 2020
scio-schemas/src/main Fix deprecation warns (#2966) May 13, 2020
scio-smb/src propagate targetParallelism from SortedBucketIO (#2997) May 21, 2020
scio-spanner/src Fix deprecation warns (#2966) May 13, 2020
scio-sql/src/main/scala/com/spotify/scio/sql Fix deprecation warns (#2966) May 13, 2020
scio-tensorflow/src Fix deprecation warns (#2966) May 13, 2020
scio-test/src Add SCollection#groupMap and SCollection#groupMapReduce operations (#… May 20, 2020
scripts Fix deprecation warns (#2966) May 13, 2020
site/src/paradox Add BigQuery section to 0.9.0 migration guide (#2991) May 20, 2020
.gitignore Update .gitignore Dec 6, 2019
.jvmopts set slf4j-simple log level to warn (#2278) Oct 1, 2019
.scala-steward.conf Ignore scala-steward bigdataoss updates (#2962) May 11, 2020
.scalafix.conf Add scalafix rules to scio itself (#1594) Dec 21, 2018
.scalafmt.conf Update scalafmt-core to 2.5.3 (#3011) May 26, 2020
CODE_OF_CONDUCT.md Add COC to project root Jul 12, 2019
CONTRIBUTING.md fix typed BQ examples build, #2272 (#2284) Oct 3, 2019
LICENSE Add license file Feb 19, 2016
NOTICE add NOTICE Feb 23, 2016
README.md add comment on scio-extra support level Jan 15, 2020
apache.sbt Use https for apache-snapshots Aug 30, 2019
build.sbt Update caffeine to 2.8.4 (#2999) May 21, 2020
codecov.yaml
version.sbt Setting version to 0.9.1-SNAPSHOT Apr 17, 2020

README.md

Scio

Build Status codecov.io GitHub license Maven Central Scaladoc Join the chat at https://gitter.im/spotify/scio Scala Steward badge

Scio Logo

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o] Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Features

  • Scala API close to that of Spark and Scalding core APIs
  • Unified batch and streaming programming model
  • Fully managed service*
  • Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable
  • JDBC, TensorFlow TFRecords, Cassandra, Elasticsearch and Parquet I/O
  • Interactive mode with Scio REPL
  • Type safe BigQuery
  • Integration with Algebird and Breeze
  • Pipeline orchestration with Scala Futures
  • Distributed cache

* provided by Google Cloud Dataflow

Quick Start

Download and install the Java Development Kit (JDK) version 8.

Use our giter8 template to quickly create a new Scio job repository:

sbt new spotify/scio.g8

Switch to the new repo (default scio-job) and build it:

cd scio-job
sbt pack

Run the included word count example:

target/pack/bin/word-count --output=wc

List result files and inspect content:

ls -l wc
cat wc/part-00000-of-00001.txt

Documentation

Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.

Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam's Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.

Artifacts

Scio includes the following artifacts:

  • scio-core: core library
  • scio-test: test utilities, add to your project as a "test" dependency
  • scio-avro: add-on for Avro, can also be used standalone
  • scio-bigquery: add-on for BigQuery, can also be used standalone
  • scio-bigtable: add-on for Bigtable
  • scio-cassandra*: add-ons for Cassandra
  • scio-elasticsearch*: add-ons for Elasticsearch
  • scio-extra: extra utilities for working with collections, Breeze, etc., best effort support
  • scio-jdbc: add-on for JDBC IO
  • scio-parquet: add-on for Parquet
  • scio-tensorflow: add-on for TensorFlow TFRecords IO and prediction

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0

You can’t perform that action at this time.