Dataflow SDK for Java Release Notes

Added support for accessing PipelineOptions in the Dataflow worker.
Removed one of the type parameters in PCollectionView, which may require simple changes to user's code that uses PCollectionView.
Changed side input API to apply per window. Calls to sideInput() now return values only in the specific window corresponding to the window of the main input element, and not the whole side input PCollectionView. Consequently, sideInput() can no longer be called from startBundle and finishBundle of a DoFn.
Added support for viewing a PCollection as a Map when used as a side input. See View.asMap().
Renamed custom source API to use term "bundle" instead of "shard" in all names. Additionally, term "fork" is replaced with "dynamic split".
Custom source Reader now requires implementing new method start(). Existing code can be fixed by simply adding this method that just calls advance() and returns its value. Additionally, code that uses the Reader should be updated to use both start() and advance(), instead of advance() only.

Initial Alpha version of the Dataflow SDK for Java with support for streaming pipelines.
Added determinism checker in AvroCoder to make it easier to interoperate with GroupByKey.
Added support for accessing PipelineOptions in the worker.
Added support for compressed sources.

Removed the dependency on the gcloud core component version 2015.02.05 or newer.

Caution: depends on the gcloud core component version 2015.02.05 or newer.

Included streaming pipeline runner, which, for now, requires additional whitelisting.
Renamed several windowing-related APIs in a non-backward-compatible way.
Added support for custom sources, which you can use to read from your own input formats.
Introduced worker parallelism: one task per processor.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 3.0 License, and code samples are licensed under the Apache 2.0 License. For details, see our Site Policies.