data-pipeline

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

The filter of Date supports convert from Timestamp

Usage Scenario

date {
source_field = "timestamp_field"
target_field = "date"
source_time_format = "TIME

It would be nice if doit could use as many processes as there are CPUs instead of only a hardcoded number.

We're using marshmallow to parse whylogs config from YAML

However, Pydantic is much more powerful as it allows users to set config via various mechanims, from YAML, JSON to Environment settings.

We should consider moving to pydantic

Environment

• klio* version(s):
  • klio-cli: v1.0.5

Description of the bug

When

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

Support copy into queries

The following packages must be updated in package.json:

• @aws-cdk/aws-dynamodb
• @aws-cdk/aws-events
• @aws-cdk/aws-events-targets
• @aws-cdk/aws-iam
• @aws-cdk/aws-lambda
• @aws-cdk/aws-lambda-event-sources
• @aws-cdk/core

The latest published version on NPM is 1.109.0 - all of these packages should be updated to at least 1.100.0 if possible.

Don't worry about event types, just have it forward messages over HTTP in the SSE protocol.

Read more: https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events