apache / spark

35 Pull requests opened by 24 people

• [SPARK-34144][SQL] Exception thrown when trying to write LocalDate and Instant values to a table

  #31264 opened Jan 20, 2021

• [SPARK-21195][CORE] MetricSystem should pick up dynamically registered metrics in sources

  #31267 opened Jan 21, 2021

• [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions

  #31268 opened Jan 21, 2021

• [SPARK-31634][SQL] "show tables like" support for SQL wildcard characters (% and _)

  #31272 opened Jan 21, 2021

• [WIP][Spark-34152][SQL] Make CreateViewStatement.child to be LogicalPlan's children so that it's resolved in analyze phase

  #31273 opened Jan 21, 2021

• [SPARK-34167][SQL]Reading parquet with IntDecimal written as a LongDecimal blows up

  #31284 opened Jan 22, 2021

• [SPARK-34199][SQL] Block `table.*` inside function to follow ANSI standard and other SQL engines

  #31286 opened Jan 22, 2021

• [SPARK-33601][SQL] Group exception messages in catalyst/parser

  #31293 opened Jan 22, 2021

• [SPARK-34205][SQL][SS] Add pipe to Dataset to enable Streaming Dataset pipe

  #31296 opened Jan 22, 2021

• [SPARK-34193][CORE] TorrentBroadcast block manager decommissioning race fix

  #31298 opened Jan 22, 2021

• [SPARK-32866][K8S] Fix docker cross-build

  #31299 opened Jan 22, 2021

• [SPARK-34214][SQL][PYTHON] Expose regexp_extract_all in PySpark

  #31306 opened Jan 24, 2021

• SPARK-33933][SQL][3.0] Materialize BroadcastQueryStage first to try to avoid broadcast timeout in AQE

  #31307 opened Jan 24, 2021

• [SPARK-33542][SQL][FOLLOWUP] Group exception messages in catalyst/catalog

  #31312 opened Jan 25, 2021

• [SPARK-34221][WEBUI] Ensure if a stage fails in the UI page, the corresponding error message can be displayed correctly.

  #31314 opened Jan 25, 2021

• [SPARK-33599][SQL][FOLLOWUP] Group exception messages in catalyst/analysis

  #31316 opened Jan 25, 2021

• [SPARK-34222][SQL] Enhance boolean simplification rule

  #31318 opened Jan 25, 2021

• [SPARK-34226][SQL] Reduce RepartitionOperation num partitions to its child max row

  #31324 opened Jan 25, 2021

• [SPARK-34182][AVRO] Improve error messages when matching Catalyst-to-Avro schemas

  #31333 opened Jan 25, 2021

• [SPARK-34234][SQL] Remove TreeNodeException that didn't work

  #31337 opened Jan 26, 2021

• [SPARK-34237][SQL] Add more metrics (fallback, spill) to object hash aggregate

  #31340 opened Jan 26, 2021

• [SPARK-34238][SQL] Unify output of SHOW PARTITIONS and pass output attributes properly

  #31341 opened Jan 26, 2021

• [SPARK-34242]use getPartitionByNames to filter partition in special case to avoid partition scan

  #31343 opened Jan 26, 2021

• [SPARK-34245][CORE] Ensure Master removes executors that failed to send finished state

  #31348 opened Jan 26, 2021

• [SPARK-34246][SQL] New type coercion syntax rules in ANSI mode

  #31349 opened Jan 26, 2021

• [SPARK-34252][SQL] Subquery with view in aggregate's grouping expression fails during the analysis check

  #31352 opened Jan 26, 2021

• [SPARK-34253][SQL] Object hash aggregate should not fallback if no more input rows

  #31353 opened Jan 26, 2021

• [SPARK-34256][ML] VectorSlicer refine numFeatures checking and toString method

  #31354 opened Jan 27, 2021

• [SPARK-34255][SQL] Support partitioning with static number on required distribution and ordering on V2 write

  #31355 opened Jan 27, 2021

• [SPARK-34257][SQL] Improve performance for last_value over unbounded window frame

  #31356 opened Jan 27, 2021

• [SPARK-34212][SQL][FOLLOWUP] Refine the behavior of reading parquet non-decimal fields as decimal

  #31357 opened Jan 27, 2021

• [SPARK-34261][SQL] Avoid side effect if create exists temporary function

  #31358 opened Jan 27, 2021

• [SPARK-34083][SQL][3.1] Using TPCDS original definitions for char/varchar colums

  #31359 opened Jan 27, 2021

• [SPARK-34260][SQL] Fix UnresolvedException when creating temp view twice

  #31360 opened Jan 27, 2021

• [SPARK-34262][SQL] Refresh cached data of v1 table in `ALTER TABLE .. SET LOCATION`

  #31361 opened Jan 27, 2021

46 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-34104][SPARK-34105][CORE][K8S] Maximum decommissioning time & allow decommissioning for excludes

  #31249 commented on Jan 25, 2021 • 49 new comments

• [SPARK-34157][SQL] Unify output of SHOW TABLES and pass output attributes properly

  #31245 commented on Jan 27, 2021 • 42 new comments

• [SPARK-26346][BUILD][SQL] Upgrade Parquet to 1.11.1

  #26804 commented on Jan 26, 2021 • 37 new comments

• [WIP][SPARK-33449][SQL] Add File Metadata cache support for Parquet and Orc

  #30483 commented on Jan 27, 2021 • 32 new comments

• [SPARK-34165][SQL] Add count_distinct as an option to Dataset#summary

  #31254 commented on Jan 27, 2021 • 15 new comments

• [SPARK-34168] [SQL] Support DPP in AQE when the join is Broadcast hash join at the beginning

  #31258 commented on Jan 25, 2021 • 13 new comments

• [SPARK-33867][SQL] Instant and LocalDate values aren't handled when generating SQL queries

  #31148 commented on Jan 26, 2021 • 9 new comments

• [SPARK-33979][SQL] Reorder predicate

  #31024 commented on Jan 24, 2021 • 8 new comments

• [SPARK-29375][SPARK-28940][SPARK-32041][SQL] Whole plan exchange and subquery reuse

  #28885 commented on Jan 26, 2021 • 7 new comments

• [SPARK-32953][PYTHON] Add Arrow self_destruct support to toPandas

  #29818 commented on Jan 21, 2021 • 7 new comments

• [SPARK-31801][API][SHUFFLE] Register map output metadata

  #30763 commented on Jan 27, 2021 • 7 new comments

• [SPARK-34054][CORE] BlockManagerDecommissioner code cleanup

  #31102 commented on Jan 26, 2021 • 7 new comments

• [SPARK-32194][PYTHON] Add more refined pyspark exceptions

  #31238 commented on Jan 22, 2021 • 7 new comments

• [SPARK-33055][PYTHON][SQL] Add Python CalendarIntervalType

  #29935 commented on Jan 22, 2021 • 6 new comments

• [SPARK-33995][SQL] Expose make_interval as a Scala function

  #31073 commented on Jan 26, 2021 • 6 new comments

• [SPARK-7768][SQL] Revise user defined types (UDT)

  #16478 commented on Jan 22, 2021 • 5 new comments

• [SPARK-28227][SQL] Support projection, aggregate/window functions, and lateral view in the TRANSFORM clause

  #29087 commented on Jan 25, 2021 • 5 new comments

• [SPARK-32995][SQL] CostBasedJoinReorder optimizer rule should be idempotent

  #29871 commented on Jan 26, 2021 • 5 new comments

• [SPARK-33739] [SQL] Jobs committed through the S3A Magic committer don't track bytes

  #30714 commented on Jan 26, 2021 • 5 new comments

• [SPARK-32165][SQL] Ensure Spark only initiates SharedState once across SparkSessions

  #31053 commented on Jan 20, 2021 • 5 new comments

• [SPARK-34136][PYTHON][SQL] Add support for complex literals in PySpark

  #31207 commented on Jan 27, 2021 • 5 new comments

• [SPARK-34141][SQL] Remove side effect from ExtractGenerator

  #31213 commented on Jan 26, 2021 • 5 new comments

• [SPARK-33308][SQL] Refactor current grouping analytics

  #30212 commented on Jan 25, 2021 • 4 new comments

• [SPARK-24818][CORE] Support delay scheduling for barrier execution

  #30650 commented on Jan 26, 2021 • 4 new comments

• [SPARK-33971][SQL] Eliminate distinct from more aggregates

  #30999 commented on Jan 26, 2021 • 4 new comments

• [SPARK-34089][CORE] MemoryConsumer should respect TaskMemoryManager's MemoryMode when the mode isn't explicitly set

  #31152 commented on Jan 20, 2021 • 4 new comments

• [SPARK-34087][SQL] Make ExecutionListenerManager.listenerBus removable.

  #31231 commented on Jan 21, 2021 • 4 new comments

• [SPARK-32288][UI] Add exception summary for failed tasks in stage page

  #29082 commented on Jan 26, 2021 • 3 new comments

• [SPARK-33594][SQL] Forbid binary type as partition column

  #30542 commented on Jan 25, 2021 • 3 new comments

• [SPARK-33678][SQL] Product aggregation function

  #30745 commented on Jan 24, 2021 • 3 new comments

• [SPARK-33832][SQL] Add an option in AQE to mitigate skew even if it c…

  #30829 commented on Jan 27, 2021 • 3 new comments

• [SPARK-33489][PYSPARK][WIP]Add support for converting null from & to Arrow…

  #30832 commented on Jan 22, 2021 • 3 new comments

• [SPARK-26399][WEBUI][CORE] Add new stage-level REST APIs and parameters

  #31204 commented on Jan 27, 2021 • 3 new comments

• [SPARK-32446][SHS] Add percentile distribution REST API of peak memory metrics for all executors

  #29247 commented on Jan 26, 2021 • 2 new comments

• [SPARK-33122][SQL] Remove redundant aggregates in the Optimzier

  #30018 commented on Jan 26, 2021 • 2 new comments

• [SPARK-33274][SS] Fix job hang in cp mode when total cores less than total kafka partition

  #30175 commented on Jan 26, 2021 • 2 new comments

• [SPARK-32617][K8S][TESTS] Configure kubernetes client based on kubeconfig settings in kubernetes integration tests

  #30751 commented on Jan 22, 2021 • 2 new comments

• [SPARK-26836][SQL] Supporting Avro schema evolution for partitioned Hive tables

  #31133 commented on Jan 22, 2021 • 1 new comment

• [SPARK-33933][SQL] Materialize BroadcastQueryStage first to avoid broadcast timeout in AQE

  #31167 commented on Jan 21, 2021 • 1 new comment

• [SPARK-34108][SQL] Caching with permanent view doesn't work in certain cases

  #31182 commented on Jan 21, 2021 • 1 new comment

• [SPARK-34146][SQL][THRIFTSERVER] "*" should not throw Exception in SparkGetSchemasOperation

  #31217 commented on Jan 22, 2021 • 1 new comment

• [SPARK-33630][SQL] Support SHOW command as table valued function

  #31257 commented on Jan 20, 2021 • 1 new comment

• [SPARK-28325][SQL]Support ANSI SQL: SIMILAR TO ... ESCAPE syntax

  #27432 commented on Jan 26, 2021 • 0 new comments

• [SPARK-32314][SHS] Add config to control whether log old format of stacktrace

  #29113 commented on Jan 26, 2021 • 0 new comments

• [SPARK-33747][CORE] Avoid calling unregisterMapOutput when the map stage is being rerunning.

  #30716 commented on Jan 22, 2021 • 0 new comments

• [SPARK-34033][R] SparkR Daemon Initialization

  #31162 commented on Jan 22, 2021 • 0 new comments