data-engineering

The Mixed Time-Series chart type allows for configuring the title of the primary and the secondary y-axis.
However, while only the title of the primary axis is shown next to the axis, the title of the secondary one is placed at the upper end of the axis where it gets hidden by bar values and zoom controls.

How to reproduce the bug

1. Create a mixed time-series chart
2. Configure axi

Description

Occasionally I get the following error when running sub-flows using create_flow_run and wait_for_flow_run

Task "wait_for_flow": Exception encountered during task execution!
Traceback (most recent call last):
  File "/root/miniconda3/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 876, in get_task_run_state
    value = prefect.utilities.executor

Describe the bug
data docs columns shrink to 1 character width with long query

To Reproduce
Steps to reproduce the behavior:

1. make a batch from a long query string
2. run validation
3. render result to data docs
4. See screenshot
   <img width="1525" alt="Data_documentation_compiled_by_Great_Expectations" src="https://user-images.githubusercontent.com/928247/103230647-30eca500-4

Tell us about the problem you're trying to solve

If a source incorrectly declares its schema (e.g; it says the "ID" column is a number when it's really a string) then we only find out about that when the destination fails upon encountering one such record. This has two problems:

1. It incurs unnecessary cost on the destination system e.g: you sync 1 billion records just fine, then the 1-bill

Under the hood, Benthos csv input uses the standard encoding/csv packages's csv.Reader struct.

The current implementation of csv input doesn't allow setting the LazyQuotes field.

We have a use case where we need to set the LazyQuotes field in order to make things work correctly.

Is your feature request related to a problem? Please describe.
Currently in feature_store.yaml, we can only specify a region for DynamoDB provider. As a result, it requires an actual DynamoDB to be available when we want to do local development/testing or integration testing in a sandbox environment.

Describe the solution you'd like
A way to solve this is to let user pass an endpoint

When we show data for a metric, we currently don't include the current day's worth of data. For users just getting set up, they may only have events from today, and want to test out if the query is working, and by excluding events from 'today', they can't see results.

TODO:

• In packages/back-end/src/services/experiments.ts on line 329, instead of using the current date as the value

when loading the import_tasks_from: file.yaml, we are not validating its contents, if the file is empty, yaml.safe_load will return None, which will throw a cryptic error message:

https://github.com/ploomber/ploomber/blob/beb625cc977bcd34481608a91daddc5493e0983c/src/ploomber/spec/dagspec.py#L326

Fix:

• If yaml.safe_load returns None, replace it with an empty list
• If returns somet

(1) Add docstrings to methods
(2) Covert .format() methods to f strings for readability
(3) Make sure we are using Python 3.8 throughout
(4) zip extract_all() in ingest_flights.py can be simplified with a Path parameter

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return Sp

Background

This thread is borne out of the discussion from #968 , in an effort to make documentation more beginner-friendly & more understandable.
One of the subtasks mentioned in that thread was to go through the function docstrings and include a minimal working example to each of the public functions in pyjanitor.

Criteria reiterated here for the benefit of discussion:

It sh

Follow the implementation example of ingestion/tests/integration/ometa/test_ometa_database_service_api.py to implement the testing of the Python client for PipelineService.