data-pipelines

Issue from the Dagster Slack

[dagster_shell] defer environ copy to solid run time

This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1623181007055200?thread_ts=1623181007.055200&cid=

🚨🚨 Feature Request

We need description, citation, license, and version meta info to be added to the dataset.

Is your feature request related to a problem?

Some datasets need this info inside them for legal reasons.

If your feature will improve HUB

Easy to implement, won't hurt for sure.

Description of the possible solution

Currently, we have all metadata store

Describe the solution you'd like
Add types to Python codebase

What does your solution aim to solve?
Make it more explicit what arguments functions take and therefore make it easier to extend consume or refactor code.

Bigquery external table API supports auto schema detection for external table source by a boolean property - ref.

If the target field names are the same as the source fields in the Google Spreadsheet, it should be better not to supply any schema rather than sp

Backend probly should return something like HTTP 409 if entity already exists.

Request (run twice):

{
  "name": "DBT data source",
  "oddrn": "//dbt/localhost",
  "description": "DBT local environment data source",
  "connection_url": "https://dbt-adapter:8080",
  "active": false
}

Logs:

2021-09-01 14:49:37.044 ERROR 1 --- [oundedElastic-2] a.w.r.e.AbstractErrorWebExce