3904 Commits

Author SHA1 Message Date
Aseem Bansal
7498f4bf54
doc(kafka): add permissions required for confluent cloud (#3850) 2022-01-12 14:50:58 -08:00
iasoon
988baeb467
feat(ingest): metabase - allow configuring how database engines get mapped to platforms (#3869) 2022-01-12 00:52:55 -08:00
Kevin Hu
5f701ebb3c
fix(cli): disable telemetry in CLI tests (#3877) 2022-01-12 00:25:42 -08:00
iasoon
130d5ff635
docs(ingest): azure-ad - correct default value for username attr (#3861) 2022-01-11 19:37:28 -08:00
Rickard Cardell
2f7e49b399
feat(ingest): datahub-rest - Make datahub-rest client more robust by configurable retries. (#3826) (#3860) 2022-01-11 19:35:06 -08:00
iasoon
6f7c2120fd
fix(ingest): metabase - fix crashes on missing values (#3859) 2022-01-10 15:40:52 -08:00
Harshal Sheth
0171735ae0
feat(build): include correct version in metadata-ingestion docker image (#3857) 2022-01-10 15:38:19 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources (#3845) 2022-01-07 17:27:09 -08:00
iasoon
0a6ec819cf
fix(ingest): dbt - handle database =None for executors like spark (#3851) 2022-01-07 15:50:29 -08:00
Paolo Genissel
744dee5206
fix(ingest): bigquery-usage - fix remove_extras to remove all partitions (#3842) 2022-01-07 12:30:45 -08:00
Aseem Bansal
cfe3c54d02
fix(ingest): dbt - fix warning due to struct type not being mapped (#3846) 2022-01-07 12:15:36 -08:00
Aseem Bansal
c4e401ee48
fix(azure AD): fix problem with missing key causing failures in ingestion (#3824) 2022-01-07 06:22:49 -08:00
Ravindra Lanka
9f80e5487d
fix(ingest): metabase - fix the datetime parsing issue (#3831)
fixes #3803
2022-01-05 16:34:05 -08:00
Swaroop Jagadish
480ad24282
feat(ingest): dbt - add support for dbt meta mapping (#3832) 2022-01-05 16:32:05 -08:00
Kevin Hu
5b369447f4
fix(ingest): glue - ignore custom connectors (#3805) 2022-01-05 15:12:50 -08:00
Tamas Nemeth
0f20d60bfc
feat(profiler): using approximate queries for profiling (#3752) 2022-01-05 12:42:15 -08:00
Tamas Nemeth
e5d8313788
feat(ingest): sql parsing - Replacing sqlmetadata sql parser lib with sqlineage parser lib (#3806) 2022-01-05 12:22:59 -08:00
Tamas Nemeth
286b42663c
fix(ingest): redshift - handle string type for Glue-backed tables (#3811) 2022-01-04 01:43:41 -08:00
Ravindra Lanka
81bc173630
fix(ingest): Make url an optional field of the DefaultConfig for business glossary source. (#3817) 2022-01-04 01:42:01 -08:00
MugdhaHardikar-GSLab
c65609a84a
feat(datahub-client): add Java REST emitter (#3781) 2022-01-02 22:48:38 +05:30
Tamas Nemeth
450cdc1739
feat(snowflake-usage): Generate email address if not exists (#3791) 2021-12-29 08:14:18 -08:00
Aseem Bansal
895af09af2
docs(kafka): add example for ingestion from confluent cloud (#3789) 2021-12-28 07:17:03 -08:00
Tamas Nemeth
5df5150e51
feat(ingest): Add config option to set Bigquery credential in source config (#3786) 2021-12-27 05:48:45 -08:00
Sergio Gómez Villamor
0f8458ad74
doc(transforms): adds doc for simple_add_dataset_properties transformer (#3790) 2021-12-27 05:46:09 -08:00
cccs-eric
2645881b08
fix(superset): fix Superset chart ingestion with an empty metric label (#3792) (#3793) 2021-12-27 05:44:37 -08:00
Tamas Nemeth
c2459056e2
docs(redshift): Updating Redshift permission prerequisites in doc (#3777) 2021-12-22 14:15:09 -08:00
Sergio Gómez Villamor
3fa22e1138
feat(transform): adds simple add dataset properties transform (#3778) 2021-12-22 08:51:38 -08:00
Hyun Min Choi
e76d40b57a
fix(lkml): bump lkml version up to 1.1.2 to support sql_preamble expression (#3757) 2021-12-21 12:13:13 -08:00
Aseem Bansal
77e3641a50
fix(ingest): fix compatibility with google composer (#3774) 2021-12-20 12:47:56 -08:00
Tamas Nemeth
2ee1a78f4e
fix(ingestion): Fix for same schema foreign key reference (#3769) 2021-12-20 07:11:55 -08:00
John Joyce
110efa68b9
docs(snowflake): Adding documentation about required Snowflake Privileges (#3770) 2021-12-19 12:01:53 -08:00
Aseem Bansal
2770eb6813
docs(ingestion): Add details of sensitive info handling (#3767) 2021-12-19 11:37:32 -08:00
Tamas Nemeth
599edd22ae
fix(ingest): profiling - disable expensive profilers by default (#3759) 2021-12-17 17:17:25 -08:00
Hyun Min Choi
e73a30dc81
feat(ingest): bigquery - add support for parsing exported audit logs (#3680) 2021-12-17 17:05:21 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method (#3746) 2021-12-16 20:07:38 -08:00
Ravindra Lanka
bd69e736ba
feat(Stateful Ingestion-2/3): Client side changes for checkpointing a source job state. (#3763)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2021-12-16 20:06:33 -08:00
jawadqu
b4eaa9e09e
feat(ingestion): Mode retry wait logic to avoid hitting Mode API rate limit (#3761) 2021-12-16 19:17:12 -08:00
Harshal Sheth
df2cb94ed8
feat(ingest): profiling add upper bound on combined query size (#3762) 2021-12-16 17:34:46 -08:00
Harshal Sheth
c6f3ddf077
build(ingest): restrict latest mypy version (#3756) 2021-12-16 08:48:15 -08:00
Tamas Nemeth
1c1561c497
feat(ingest): skipping emitting metadata for duplicate tables from ingestion (#3753) 2021-12-15 09:15:03 -08:00
Sergio Gómez Villamor
c59c63e90d
feat: enables dbt metadata files to be loaded from URIs (#3739) 2021-12-15 09:11:39 -08:00
Harshal Sheth
adf9d2ead7
test(ingest): fix pytest warning for class starting with Test (#3745) 2021-12-14 22:44:42 -08:00
varunbharill
70d068892e
fix(ingest): snowflake honor allow/deny pattern for lineage and usage. (#3748) 2021-12-14 20:42:29 -08:00
Gabe Lyons
2e2ed34250
feat(ingest): snowflake-usage add knob for direct objects accessed vs base objects accessed (#3744) 2021-12-14 18:07:55 -08:00
John Joyce
40963b0635
fix(ingest): remove data platform isalpha check as it complains about s3 (#3742) 2021-12-14 17:39:15 -08:00
varunbharill
5f80e7a4b2
fix(ingest): changing datahub-graph to use underlying session connection. (#3743) 2021-12-14 17:28:27 -08:00
Gabe Lyons
3fd3313544
Revert "feat(graph): Make Dgraph a proper Neo4j alternative (#3578)" (#3740) 2021-12-14 10:49:03 -08:00
Harshal Sheth
f24440eff3
fix(ingest): count profiled tables separately in report (#3731) 2021-12-13 23:06:49 -08:00
jawadqu
578590e795
feat(ingestion) : Add Metabase Source Connector (#3602)
Co-authored-by: Jawad Qureshi <jqureshi@petabloc.com>
2021-12-13 23:02:47 -08:00
Harshal Sheth
3b7fd24740
feat(ingest): cleanup deprecated datahub.integrations.airflow.* imports (#3732) 2021-12-13 21:34:09 -08:00