Aditya Radhakrishnan
c20a47f34c
feat(operation): display the reported time for last updated in the UI ( #4800 )
2022-05-02 16:00:29 -07:00
Shirshanka Das
a9ad138172
feat(ingest): docs - overhaul source connector docs to make it code driven ( #4798 )
...
Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
2022-05-02 00:18:15 -07:00
Shirshanka Das
c15b44a1de
fix(ingest): databricks - hive ingestion should not fail on table comment ( #4787 )
2022-04-29 13:56:42 -07:00
mayurinehate
c34a1ba735
fix(s3): improved handling for corner cases ( #4774 )
2022-04-29 12:25:41 -07:00
vanmeete
74d6d35881
feat(ingestion): add Pulsar source ( #4721 )
2022-04-29 15:57:02 +05:30
Aseem Bansal
f099aeb550
fix(m1): tweak m1 preflight ( #4771 )
2022-04-29 10:29:47 +02:00
Aseem Bansal
ee6413f0ab
feat(snowflake): add option to disable checking role grants ( #4760 )
...
* feat(ingest): add option to disable checking role grants
2022-04-29 10:14:05 +02:00
Aseem Bansal
37c1fce964
fix(snowflake): improve debug log for external tables ( #4772 )
2022-04-29 10:10:46 +02:00
Jordan Wolinsky
bbac4a7a11
feat(ingestion): glue/s3 - Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job ( #4689 )
2022-04-29 10:09:06 +02:00
Shirshanka Das
d0eb772301
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … ( #4786 )
2022-04-28 22:31:19 -07:00
John Joyce
91f166c60b
fix(ingest): fwk - fix how we import DataHub actions ( #4784 )
2022-04-28 21:18:15 -07:00
Ravindra Lanka
8583f397f1
Fix breaking changes from GE 0.15.3 that are affecting our Python3.6 smoke_tests. ( #4779 )
2022-04-28 15:50:17 -07:00
Aseem Bansal
9d1168317c
fix(lint): lib upgrade caused ( #4773 )
2022-04-28 20:35:20 +05:30
Aditya Radhakrishnan
4458e6261c
docs(ingestion): glue - clarify that table regex patterns should be fully-qualified ( #4747 )
2022-04-27 21:28:02 -07:00
Aditya Radhakrishnan
a7d76e43b5
fix(azure_ad): make redirect and graph_url optional parameters and update docs ( #4754 )
2022-04-27 15:19:49 -07:00
mayurinehate
33d6842ab0
fix(tableau): miscellaneous tableau fixes for lineage, browse path, non-embedded datasets ( #4724 )
...
* fix(tableau): add config whether to emit aspects for external datasets
other changes:
- do not set browse path in absence of datasource or project name
- remove unused nodes from tableau metadata query
* fix(tableau): remove redundant (transitive) lineage edges between tables, datasource, sheet
other changes:
- update subtypes for datasource to be more specific
* fix(tableau): fix browse paths for custom sql and embedded datasource
other changes:
- do not set browse path if any intermediate folder level in browse path is empty
* docs(tableau): update tableau doc
2022-04-27 11:20:03 -07:00
Paweł Iwiński
810b6a4655
feat(ingestion): aws - Custom endpoint url and proxies in S3. ( #4708 )
...
This feature will allow user to specify custom endpoint url with custom proxies to connect to dedicated S3 bucket not associated with the amazon aws.
2022-04-27 14:13:08 +02:00
Jordan Wolinsky
c6424f7a8a
fix(ingest): aws - When using aws_profile, grab temporary credentials from the session. ( #4751 )
...
* allow for temporary credentials generated when using an aws_profile. Mostly used for SSO and temporary credentials
2022-04-27 11:48:57 +02:00
Danilo Peixoto
d2a6bc06dc
feat(ingest): feast - add support for Feast 0.18, deprecate older integration ( #4094 )
2022-04-26 14:35:02 -07:00
cccs-eric
abf8d62cf5
fix(azure_ad): silently discard other Azure AD object types ( #4693 ) ( #4704 )
2022-04-26 13:56:46 -07:00
Sebo Kim
958b52f2f4
fix(ingest): bigquery - Fix BigQuery Datetime/Timestamp type column partition table profile bug ( #4658 )
...
* fix BigQuery Datetime type column partition table profile bug
* inplace datetime replace
* extract out 'if' blocks and write a unit-test
* parse logic inside get_partition_range func
2022-04-26 16:54:19 +02:00
John Joyce
95487555c5
feat(cli): Eagerly load datahub actions CLI commands ( #4748 )
2022-04-26 19:00:41 +05:30
Eddie (Xiao Di) Li
01a5b13a7a
feat(Cassandra): Add Cassandra implementation of EntityService ( #3286 )
2022-04-25 17:29:24 -07:00
Shirshanka Das
7e9e9ddca0
fix(cli): telemetry - make config file processing more robust ( #4738 )
2022-04-25 11:05:24 -07:00
Tamas Nemeth
474b0ba61e
feat(ingest): dbt - add query tag mapping and match template ( #4744 )
2022-04-25 10:56:45 -07:00
Shirshanka Das
a55073036d
fix(cli): align default sink env variables across all cli commands ( #4739 )
2022-04-25 10:19:17 -07:00
Shirshanka Das
a518e3d13e
feat(cli): improve error reporting, make sink config optional ( #4718 )
2022-04-24 17:12:21 -07:00
John Joyce
b1b1898752
Revert "fix(timelineCli): fix naming for timeline cli ( #4729 )" ( #4731 )
...
This reverts commit 7bcc2d9f62a7513b8bd8bc74d6eea0c85f07e245.
2022-04-22 15:15:47 -07:00
RyanHolstien
7bcc2d9f62
fix(timelineCli): fix naming for timeline cli ( #4729 )
2022-04-22 13:33:17 -07:00
John Joyce
325c9b0f08
Adding new models to python codegen ( #4726 )
2022-04-22 10:12:58 -07:00
Shirshanka Das
3353451561
fix(transformers): add snapshot aspects from dataset into base_transformer ( #4719 )
2022-04-22 08:46:55 -07:00
Kevin Hu
df514cfd6e
feat(ingest): mysql - add decimal128 custom type ( #4624 )
2022-04-21 16:43:29 -07:00
mayurinehate
9e30a9cc81
fix(glue): fix error for custom connector if ignore_unsupported_conne… ( #4667 )
2022-04-21 11:33:04 -07:00
BZ
bbfc902950
fix(ingestion): glue - delete CatalogId parameter from get_jobs api call ( #4646 )
2022-04-21 09:30:01 +02:00
Aseem Bansal
c66ef7c1fe
fix(snowflake): deprecate config, update examples ( #4644 )
...
* fix(snowflake): deprecate config, update examples
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-20 15:21:09 -07:00
Aditya Radhakrishnan
15e90f6dd0
feat(ingest) - update identity sources to add flags for masking sensitive work units ( #4711 )
2022-04-20 14:21:08 -07:00
Tamas Nemeth
bb2b8515ff
fix(ingest) bigquery: Moving bigquery temporary credential deletion to atexit ( #4701 )
2022-04-20 13:35:16 -07:00
Tamas Nemeth
b217c8cd4e
fix(looker): Pydantic validation error for Looker TransportOptions on python 3.8 ( #4705 )
...
* Fix for pydantic validation error for Looker TransportOptions on python 3.8
2022-04-20 22:27:56 +02:00
Aseem Bansal
4b7f407e26
fix(bigquery): error due to not handling date properly ( #4702 )
2022-04-20 18:14:33 +02:00
Aseem Bansal
bb0a87ae74
fix(snowflake): remove extra lineage edges in reports, change badly named config variable ( #4595 )
...
* fix(snowflake): remove extra lineage edges in reports
2022-04-20 07:03:54 -07:00
Aseem Bansal
98d4fd4ea9
fix(cli): rest emitter should override config and env variables ( #4622 )
...
* fix(cli): rest emitter should override env variables
* fix(cli): change to not update env variables, small refactor
* fix bug
2022-04-18 07:31:01 -07:00
Atul Saurav
e8e0067f23
fix(cli):Supress printing variables to logs during ingestion failure ( #4566 )
...
Currently when ingestion while running pipeline, stackprinter prints all vars to logs.
This may contain sensitive information. To prevent this from happening, a optional `safe`
flag is added to cli. If this flag is set while running ingestion, no variables are logged in
case of unexpected failures.
2022-04-15 10:30:48 -07:00
Aseem Bansal
8e2bd00059
chore(ingestion): update example recipes ( #4660 )
2022-04-14 16:09:19 -07:00
Maggie Hays
2534221d5d
remove source summary table ( #4670 )
2022-04-14 16:28:05 -05:00
Tamas Nemeth
61dc6e8723
fix(ingestion): airflow - import emitters indirectly to avoid unneeded dependency ( #4668 )
2022-04-14 10:22:16 -07:00
Fernanda de Camargo
d508f5c036
fix(ingestion): tableau - validate datasource before creating its upstream ( #4613 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2022-04-13 23:08:01 -07:00
Aseem Bansal
73d69510f8
fix(sqlparser): fix sqlparser breaking due to # sign ( #4662 )
...
* fix(sqlparser): fix sqlparser breaking due to # sign
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-13 17:15:38 -07:00
Arun Vasudevan
5aa3da5c9c
feat(ingestion) dbt: Fixing issue with strip_user_ids_from_email and adding owner_extraction_pattern ( #4587 )
...
* Fixing issue with strip_user_ids_from_email and adding owner_extraction_pattern
Co-authored-by: BZ <93607724+BoyuanZhangDE@users.noreply.github.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-13 16:58:36 -07:00
Aseem Bansal
53d357b4eb
fix(bigquery-usage): missing dependency ( #4661 )
2022-04-13 14:29:31 +02:00
Aseem Bansal
5a59d5a1dd
fix(ingestion): Adding missing init.py ( #4659 )
2022-04-13 11:02:57 +02:00
