etl

We should do something like https://blog.questionable.services/article/kubernetes-deployments-configmap-change/ to ensure that if the pod sweeper has a different config map the underlying pod gets rolled.

Under the hood, Benthos csv input uses the standard encoding/csv packages's csv.Reader struct.

The current implementation of csv input doesn't allow setting the LazyQuotes field.

We have a use case where we need to set the LazyQuotes field in order to make things work correctly.

I am trying to implement retry logic for Aws Aurora Mysql.

In Linq2db there is retyrpolicy already implemented except there is no way to pass retrypolicy to DataContext.
So I have extended the DataContext as follows.

public class RetryingDataContext : DataContext
{
protected IRetryPolicy _retryPolicy;
public RetryingDataContext(IDataProvider dataProvider, string c

The functions in this file should be factored out to a separate utility lib as they are reused in bitcoin-etl https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/misc_utils.py

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return Sp

Sending a rest call to delete a job specification throws 404 where as grpc call works fine. Steps to reproduce

curl -X DELETE "http://localhost:9100/v1/project/my-project/namespace/kush/helloworld" -H  "accept: application/json"

Right now, these aren't caught until we try to gob-encode. Consider failing faster in type-checking to avoid too much confusion/loss when it works with local execution.

If i have an input yaml connecting to jdbc source like:

inputs:
  somedb:
    jdbc:
      connectionUrl: jdbc:jtds:redact
      user: someuser
      password: somepass

the log has

2020-12-08 17:01:28,076 [main] INFO  com.yotpo.metorikku.Job - these are the config inputs: Some(Map(.....somepass

ideally this should be printed as ****** in the log