3904 Commits

Author SHA1 Message Date
Kirill Popov
3acd25ba1d
feat(ingest/metabase): detect source table for cards sourced from other cards (#8577) 2023-08-28 13:02:41 -04:00
Harshal Sheth
7b66c32b70
feat(ingest): support writing configs to files (#8696) 2023-08-28 09:55:50 -07:00
Akhil
ada40e6edd
fix(superset): get alternate platform value if sqlalchemy_uri param is missing (#8667) 2023-08-28 15:03:43 +05:30
Mayuri Nehate
cc94ffbf6c
fix(ingest): stateful redundant run skip handler (#8467) 2023-08-28 15:03:31 +05:30
Mayuri Nehate
47571e6257
fix(examples): fix typo in business glossary bootstrap yml (#8703) 2023-08-28 15:02:47 +05:30
Andrew Sikowitz
d169ee8ae5
config(ingest/rest): Update rest sink defaults to retry more often (#8729) 2023-08-25 20:27:35 -04:00
Hyejin Yoon
04ecf4f75a
docs(docs): add native versioning (#8714) 2023-08-25 14:10:13 -07:00
Andrew Sikowitz
6659ff26ef
feat(ingest/sql-queries): Add sql queries source, SqlParsingBuilder, sqlglot_lineage performance optimizations (#8494)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-08-24 10:35:46 -04:00
Mayuri Nehate
e285da3e75
feat(ingest/snowflake): tables from snowflake shares as siblings (#8531) 2023-08-24 10:23:07 -04:00
Adriano Vega Llobell
d6a935e3ca
fix(ingestion/kafka-connect): update retrieval of database name in Debezium SQL Server (#8608) 2023-08-24 14:49:04 +05:30
RChygir
43d48ddde4
feat(ingest/mssql): load jobs and stored procedures (#5363) 2023-08-24 14:48:03 +05:30
Alexander
bcef25acd3
feat(ingest/looker): Record observed lineage timestamps for Looker and LookML sources (#7735) 2023-08-24 14:47:04 +05:30
Andrew Sikowitz
22c35f1a23
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off (#8659) 2023-08-24 14:46:06 +05:30
Harshal Sheth
090f8af8e4
ci(ingest): make ingestion caching rules correct (#8685) 2023-08-23 22:06:28 -07:00
Harshal Sheth
a97548ce46
fix(ingest/powerbi): add sqlglot python dep (#8704) 2023-08-23 22:05:53 -07:00
Andrew Sikowitz
01ae5d96da
fix(ingest/ldap): Handle case when 'objectClass' not in attrs (#8658) 2023-08-23 15:58:34 -04:00
Andrew Sikowitz
8141e2d649
remove(ingest/snowflake): Remove legacy snowflake lineage (#8653)
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-08-23 15:57:46 -04:00
Andrew Sikowitz
68abf9c6a1
build(ingest): Bump pydantic pin (#8660) 2023-08-23 16:55:51 +05:30
siddiquebagwan-gslab
8ee58af0c2
feat(ingestion/powerbi): support multiple tables as upstream in native SQL parsing (#8592) 2023-08-23 14:38:58 +05:30
Andrew Sikowitz
4116716a15
logs(ingestion/unity): Hide stack trace on sql parse failure logs (#8657) 2023-08-23 14:38:10 +05:30
Andrew Sikowitz
439cf4d7dc
test(ingest/vertica): Skip integration test failing CI; support arm Macs (#8694) 2023-08-22 16:27:46 -04:00
Mayuri Nehate
3681e1a128
docs(ingest/kafka-connect): add details on platform instance mapping (#8654) 2023-08-18 18:51:14 +05:30
Harshal Sheth
c5a2e8923e
feat(ingest/dbt-cloud): use job-based graphql queries (#8647) 2023-08-17 13:29:10 -04:00
Harshal Sheth
130f908a49
fix(ingest/snowflake): fix usage enum bug (#8649) 2023-08-17 13:27:19 -04:00
Alexander
c0addf6eef
feat(ingest/bigquery): add tag to BigQuery clustering columns (#8495)
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-08-17 12:44:15 -04:00
Jinlin Yang
6748aecdc0
fix(ingest/s3): emit data_platform_instance aspect if the config has platform_instance (#8585) 2023-08-17 10:40:54 +05:30
david-leifker
40caf35812
feat(datahub-ingestion): refactor datahub ingestion slim images (#8515) 2023-08-17 10:40:17 +05:30
Harshal Sheth
2f9b5a8273
fix(ingest): avoid mutable defaults in powerbi dataclass (#8609) 2023-08-17 10:36:52 +05:30
kr_Deepankar
23ac9062fe
feat(ingestion/ldap): flag to ingest ldap users with email instead of username (#8606) 2023-08-16 10:33:18 +05:30
kr_Deepankar
94ce753bb0
fix(ingest/kafka): use SchemaReference properties instead of dict access (#8615)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-16 10:33:09 +05:30
skrydal
baae3d261d
fix(ingest/okta): fix event_loop RuntimeError with nested asyncio (#8637) 2023-08-16 10:32:57 +05:30
Andrew Sikowitz
526e626146
feat(ingest): Add DataHub source (#8561)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-15 17:49:20 -04:00
Tamas Nemeth
ec693d744a
feat(ingest/s3): Adding option to pass in any spark config property to s3 source (#8621) 2023-08-15 14:43:25 +02:00
Harshal Sheth
9eb432cda0
fix(ingest/airflow): make inlets work again (#8631) 2023-08-15 14:42:43 +02:00
Mayuri Nehate
ddcd5109dc
feat(ingest): allow relative start time config (#8562)
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-08-14 17:48:06 -07:00
mohdsiddique
b3251577e1
docs(ingsetion/csv-enricher): fix sample csv mentioned in Docstrings (#8432)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-14 10:18:54 -07:00
alplatonov
11fdfcf956
Fix(ingestion/clickhouse) move to two tier sqlalchemy (#8300)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-11 16:11:40 -04:00
Mayuri Nehate
7bb4e7b90d
docs(ingest): update s3 and gcs doc with concept mapping (#8575) 2023-08-11 11:01:15 -07:00
Meenakshi Kamalaseshan Radha
93cc10ebfc
fix(ingest): Add client_certificate_path for rest client cert instead of ca_certif… (#8581)
Co-authored-by: Kamalaseshan Radha <mkamalas@LAMU02DN212MD6R.uhc.com>
2023-08-10 15:43:06 +05:30
alplatonov
b58f9bb396
Feat(ingest/ldap)fix list index out of range error (#8525)
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
2023-08-09 13:13:27 -04:00
Harshal Sheth
ebf42e2702
fix(ingest): use hive pure_sasl variant (#8570) 2023-08-09 13:04:36 -04:00
Harshal Sheth
fd9121737d
fix(ingest/file): remove entity_type_counts and aspect_counts (#8586) 2023-08-09 13:01:12 -04:00
Aseem Bansal
2b0952195b
doc(ingest/log): failure log at pipeline level document (#8591) 2023-08-09 10:18:14 +05:30
Ellie O'Neil
4b3e566be8
docs(lineage): Add description to make_lineage_mce (#8596) 2023-08-08 16:06:47 -07:00
Mayuri Nehate
b4e104f190
fix(ingest/snowflake): maintain qualified name casing, do not lowercase (#8574) 2023-08-04 10:43:22 -07:00
Harshal Sheth
ef15861d0d
fix(ingest): handle redaction of configs with int keys (#8545) 2023-08-04 13:02:04 -04:00
Aseem Bansal
dac89fb1fb
feat(ingest): allow lower freq profiling based on date of month/day of week (#8489) 2023-08-04 10:13:48 +05:30
mohdsiddique
6a36118b4f
feat(ingestion/snowflake): use user email-id in urn generation for top users stat (#8513)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-03 08:30:50 +05:30
mohdsiddique
05ef7db45e
fix(ingetion/mssql): convert dataset urns to lowercase (#8551)
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
2023-08-02 13:44:05 -07:00
Harshal Sheth
3428bcaaad
fix(ingest): add tableau sqlglot dep (#8552) 2023-08-02 15:18:06 -03:00