Skip to content

/spark

Apache Spark
python scala r java big-data jdbc sql spark
Scala Java Python HiveQL R TSQL Other
  1. Scala 72.4%
  2. Java 9.5%
  3. Python 7.2%
  4. HiveQL 4.3%
  5. R 2.7%
  6. TSQL 1.1%
  7. Other 2.8%
Branch: master
Find file
Clone or download

Clone with HTTPS

Use Git or checkout with SVN using the web URL.

Download ZIP

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching GitHub Desktop

If nothing happens, download GitHub Desktop and try again.

Launching Xcode

If nothing happens, download Xcode and try again.

Launching Visual Studio

If nothing happens, download the GitHub extension for Visual Studio and try again.

@Ngone51 @cloud-fan
Ngone51 and cloud-fan [SPARK-30098][SQL] Use default datasource as provider for CREATE TABL… 
…E syntax

### What changes were proposed in this pull request?

In this PR, we propose to use the value of `spark.sql.source.default` as the provider for `CREATE TABLE` syntax instead of `hive` in Spark 3.0.

And to help the migration, we introduce a legacy conf `spark.sql.legacy.respectHiveDefaultProvider.enabled` and set its default to `false`.

### Why are the changes needed?

1. Currently, `CREATE TABLE` syntax use hive provider to create table while `DataFrameWriter.saveAsTable` API using the value of `spark.sql.source.default` as a provider to create table. It would be better to make them consistent.

2. User may gets confused in some cases. For example:

```
CREATE TABLE t1 (c1 INT) USING PARQUET;
CREATE TABLE t2 (c1 INT);
```

In these two DDLs, use may think that `t2` should also use parquet as default provider since Spark always advertise parquet as the default format. However, it's hive in this case.

On the other hand, if we omit the USING clause in a CTAS statement, we do pick parquet by default if `spark.sql.hive.convertCATS=true`:

```
CREATE TABLE t3 USING PARQUET AS SELECT 1 AS VALUE;
CREATE TABLE t4 AS SELECT 1 AS VALUE;
```
And these two cases together can be really confusing.

3. Now, Spark SQL is very independent and popular. We do not need to be fully consistent with Hive's behavior.

### Does this PR introduce any user-facing change?

Yes, before this PR, using `CREATE TABLE` syntax will use hive provider. But now, it use the value of `spark.sql.source.default` as its provider.

### How was this patch tested?

Added tests in `DDLParserSuite` and `HiveDDlSuite`.

Closes #26736 from Ngone51/dev-create-table-using-parquet-by-default.

Lead-authored-by: wuyi <yi.wu@databricks.com>
Co-authored-by: yi.wu <yi.wu@databricks.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Latest commit 58be82a Dec 6, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.github [MINOR][INFRA] Use GitHub Action Cache for `build` Nov 24, 2019
R [SPARK-29777][SPARKR] SparkR::cleanClosure aggressively removes a fun… Nov 19, 2019
assembly Revert "Prepare Spark release v3.0.0-preview-rc2" Oct 31, 2019
bin [SPARK-28525][DEPLOY] Allow Launcher to be applied Java options Jul 30, 2019
build [SPARK-30121][BUILD] Fix memory usage in sbt build script Dec 5, 2019
common [SPARK-30129][CORE] Set client's id in TransportClient after successf… Dec 5, 2019
conf [SPARK-29032][CORE] Add PrometheusServlet to monitor Master/Worker/Dr… Sep 13, 2019
core [SPARK-30009][CORE][SQL][FOLLOWUP] Remove OrderingUtil and Utils.nanS… Dec 5, 2019
data [SPARK-22666][ML][SQL] Spark datasource for image format Sep 5, 2018
dev [SPARK-30142][TEST-MAVEN][BUILD] Upgrade Maven to 3.6.3 Dec 6, 2019
docs [SPARK-30098][SQL] Use default datasource as provider for CREATE TABL… Dec 6, 2019
examples [SPARK-29126][PYSPARK][DOC] Pandas Cogroup udf usage guide Oct 31, 2019
external [SPARK-29957][TEST] Reset MiniKDC's default enctypes to fit jdk8/jdk11 Dec 6, 2019
graph Revert "Prepare Spark release v3.0.0-preview-rc2" Oct 31, 2019
graphx [SPARK-29877][GRAPHX] static PageRank allow checkPoint from previous … Nov 28, 2019
hadoop-cloud Revert "Prepare Spark release v3.0.0-preview-rc2" Oct 31, 2019
launcher [SPARK-29733][TESTS] Fix wrong order of parameters passed to `assertE… Nov 3, 2019
licenses-binary [SPARK-29308][BUILD] Update deps in dev/deps/spark-deps-hadoop-3.2 fo… Oct 13, 2019
licenses [SPARK-27557][DOC] Add copy button to Python API docs for easier copy… May 1, 2019
mllib-local Revert "Prepare Spark release v3.0.0-preview-rc2" Oct 31, 2019
mllib [SPARK-24666][ML] Fix infinity vectors produced by Word2Vec when numI… Dec 6, 2019
project [SPARK-29348][SQL] Add observable Metrics for Streaming queries Dec 3, 2019
python [SPARK-30113][SQL][PYTHON] Expose mergeSchema option in PySpark's ORC… Dec 4, 2019
repl [SPARK-30012][CORE][SQL] Change classes extending scala collection cl… Dec 3, 2019
resource-managers [SPARK-30111][K8S] Apt-get update to fix debian issues Dec 4, 2019
sbin [SPARK-28164] Fix usage description of `start-slave.sh` Jun 26, 2019
sql [SPARK-30098][SQL] Use default datasource as provider for CREATE TABL… Dec 6, 2019
streaming [SPARK-29477] Improve tooltip for Streaming tab Dec 3, 2019
tools Revert "Prepare Spark release v3.0.0-preview-rc2" Oct 31, 2019
.gitattributes [SPARK-3870] EOL character enforcement Oct 31, 2014
.gitignore [SPARK-30084][DOCS] Document how to trigger Jekyll build on Python AP… Dec 4, 2019
CONTRIBUTING.md [MINOR][DOCS] Tighten up some key links to the project and download p… May 21, 2019
LICENSE [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ Nov 3, 2019
LICENSE-binary [MINOR][BUILD] Fix an incorrect path in license-binary file Nov 13, 2019
NOTICE [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ Nov 3, 2019
NOTICE-binary [SPARK-29674][CORE] Update dropwizard metrics to 4.1.x for JDK 9+ Nov 3, 2019
README.md [SPARK-28473][DOC] Stylistic consistency of build command in README Jul 23, 2019
appveyor.yml [SPARK-29991][INFRA] Support Hive 1.2 and Hive 2.3 (default) in PR bu… Nov 30, 2019
pom.xml [SPARK-30142][TEST-MAVEN][BUILD] Upgrade Maven to 3.6.3 Dec 6, 2019
scalastyle-config.xml [SPARK-30030][INFRA] Use RegexChecker instead of TokenChecker to chec… Nov 25, 2019

README.md

Apache Spark

Spark is a unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. It also supports a rich set of higher-level tools including Spark SQL for SQL and DataFrames, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for stream processing.

https://spark.apache.org/

Jenkins Build AppVeyor Build PySpark Coverage

Online Documentation

You can find the latest Spark documentation, including a programming guide, on the project web page. This README file only contains basic setup instructions.

Building Spark

Spark is built using Apache Maven. To build Spark and its example programs, run:

./build/mvn -DskipTests clean package

(You do not need to do this if you downloaded a pre-built package.)

You can build Spark using more than one thread by using the -T option with Maven, see "Parallel builds in Maven 3". More detailed documentation is available from the project site, at "Building Spark".

For general development tips, including info on developing Spark using an IDE, see "Useful Developer Tools".

Interactive Scala Shell

The easiest way to start using Spark is through the Scala shell:

./bin/spark-shell

Try the following command, which should return 1,000,000,000:

scala> spark.range(1000 * 1000 * 1000).count()

Interactive Python Shell

Alternatively, if you prefer Python, you can use the Python shell:

./bin/pyspark

And run the following command, which should also return 1,000,000,000:

>>> spark.range(1000 * 1000 * 1000).count()

Example Programs

Spark also comes with several sample programs in the examples directory. To run one of them, use ./bin/run-example <class> [params]. For example:

./bin/run-example SparkPi

will run the Pi example locally.

You can set the MASTER environment variable when running examples to submit examples to a cluster. This can be a mesos:// or spark:// URL, "yarn" to run on YARN, and "local" to run locally with one thread, or "local[N]" to run locally with N threads. You can also use an abbreviated class name if the class is in the examples package. For instance:

MASTER=spark://host:7077 ./bin/run-example SparkPi

Many of the example programs print usage help if no params are given.

Running Tests

Testing first requires building Spark. Once Spark is built, tests can be run using:

./dev/run-tests

Please see the guidance on how to run tests for a module, or individual tests.

There is also a Kubernetes integration test, see resource-managers/kubernetes/integration-tests/README.md

A Note About Hadoop Versions

Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported storage systems. Because the protocols have changed in different versions of Hadoop, you must build Spark against the same version that your cluster runs.

Please refer to the build documentation at "Specifying the Hadoop Version and Enabling YARN" for detailed guidance on building for a particular distribution of Hadoop, including building for particular Hive and Hive Thriftserver distributions.

Configuration

Please refer to the Configuration Guide in the online documentation for an overview on how to configure Spark.

Contributing

Please review the Contribution to Spark guide for information on how to get started contributing to the project.

You can’t perform that action at this time.