* backend
* format & tests
* rename backend
* migrations and ingestion
* format & tests
* format & tests
* tests
* format & tests
* tests
* updated ui side of changes
* addressing comment
* fixed failing unit test
* fix test list
* added e2e test, and fixed existing test
---------
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
When using using the assumeRole connection option with jobs involving
AWS services, the default 1 hour of boto might not cut it and the job
fails, one way of solving this is to refresh the credentials when they
expire, this ensures there is always valid credentials for the job
regardless for how long it runs.
Co-authored-by: Abdallah Serghine <abdallah.serghine@olx.pl>
Co-authored-by: IceS2 <pjt1991@gmail.com>
* MINOR: User search should only look in name & displayname
* py_format
* pyformat
---------
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
This aims at fixing the s3 ingestion for parquet files, current behaviour is that
the pipeline will break if it encounters a file that is not valid parquet in the
the container, this is not great as containers might container non parquet files
on purpose like for example _SUCCESS files created by spark.
For that do not fail the whole pipeline when a single container fails, instead
count it as a failure and move on with the remainder of the containers, this is
already an improvement by ideally the ingestion should try a couple more files
under the given prefix before given up, additionally we can allow users to specify
file patterns to be ignored.
Co-authored-by: Abdallah Serghine <abdallah.serghine@olx.pl>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
* fix(data-diff): added nd5 handling for bigquery
- added MD5 handling for bigquery
- use URL instead of Engine because it requires less steps and less prone to failure
* added e2e test for data diff with sampling in bigquery
* Get last completed job run
* formatting
---------
Co-authored-by: 😺Leo Luo <leo.luo@mavenclinic.com>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
