Andrew Sikowitz
8c1fa04c87
fix(ingest/snowflake): Allow SnowflakeObjectAccessEntry.objectId to be None ( #7601 )
...
Co-authored-by: Pedro Silva <pedro@acryl.io>
2023-03-16 12:55:52 +01:00
mohdsiddique
0a36826407
fix(ingest/powerbi): support each expression in m-query function invocation ( #7541 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-15 09:06:29 -07:00
Harsha Mandadi
bf36c935fa
feat(ingest/s3): support path_specs of different S3 buckets in the same recipe ( #7514 )
2023-03-14 21:55:57 -07:00
Andrew Sikowitz
8dd7a85533
refactor(ingest): Use shared connection wrapper over connection cache ( #7570 )
2023-03-14 15:09:37 -07:00
Mayuri Nehate
70c9b20c6d
feat(ingest/kafka-connect): add stateful ingestion and platform instance support ( #7526 )
2023-03-14 13:59:54 -07:00
John Joyce
18f387c6ea
fix(cli): Adding exit code to correctly return failure or success ( #7520 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-03-13 13:32:40 -07:00
Teppo Naakka
23830423f8
feat(ingest): powerbi # store powerbi entity descriptions ( #7154 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-03-13 13:31:21 -07:00
Harshal Sheth
fbfe43b1cb
feat(ingest): fix edge cases + interface cleanup for file-system APIs ( #7533 )
2023-03-13 13:14:53 -07:00
Shirshanka Das
26cf0a71ab
fix(test): suppress s3 golden file test for specific paths ( #7551 )
2023-03-12 10:43:02 -07:00
Harshal Sheth
b82afa89f1
feat(ingest): enable joins across FileBackedDicts + add FileBackedList ( #7506 )
2023-03-09 15:22:03 -08:00
Harshal Sheth
91a8e5ea16
feat(ingest/powerbi): support PowerBI parameter references ( #7523 )
2023-03-09 11:14:55 -08:00
J Feldman
aa4228734c
feat(ingest/looker): upgrade to Looker API from 3.1 to 4.0 ( #7411 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2023-03-07 16:49:37 -08:00
Arun Vasudevan
e93a17e917
feat(ingest/postgres): support estimated row counts in profiling ( #7476 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-07 11:08:09 -08:00
Peter Szalai
7207c348b8
feat(cli): finalizing quickstart config commit hash ( #7509 )
2023-03-07 10:47:45 -08:00
mohdsiddique
de719663ff
feat(ingestion): powerbi # support Google BigQuery table lineage ( #7502 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-07 09:07:00 -08:00
Peter Szalai
1d33392761
feat(cli): introduce remote config for quickstart ( #7424 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-03-07 13:14:24 +01:00
Mayuri Nehate
406b11a9ed
feat(ingest/GX): add urn lowercasing option for GX assertions ( #7472 )
2023-03-06 20:42:23 -08:00
Harshal Sheth
01ee351c4c
fix(ingest): prevent logging from blowing up on TypeErrors ( #7497 )
2023-03-03 14:36:55 -08:00
mohdsiddique
29d171106b
feat(ingest/tableau): project path and container support ( #7426 )
...
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
2023-03-02 16:53:19 -08:00
Aseem Bansal
1adbc2cab0
chore(ci): upgrade GE version ( #7290 )
2023-03-02 10:47:38 -08:00
Harshal Sheth
49029943f9
fix(ingest): remove extraneous platform configs ( #7454 )
2023-03-02 01:10:35 -08:00
Andrew Sikowitz
8101f0d47a
feat(ingest): Introduce FileBackedDict for offloading data to disk ( #7461 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Also includes minor refactoring to the bigquery connector
2023-03-01 19:09:51 -05:00
Shirshanka Das
17e85979dd
refactor(ingest): subtypes - standardize ( #7437 )
2023-02-28 13:11:07 -08:00
Harshal Sheth
639bbcfa86
chore(ingest/glue): cleanup deprecated underlying_platform config ( #7449 )
2023-02-28 10:41:54 -08:00
nachiket-juneja
e07cd2090b
Feat/s3 ingestion enhancement to update schema from latest partition ( #7410 )
...
Co-authored-by: Prashant Singh Thakur <prashant.thakur@nucleusteq.com>
2023-02-28 08:58:28 +01:00
Harshal Sheth
3b8b5e8aa4
chore(ingest): cleanup unused files/vars in tests ( #7450 )
2023-02-28 08:07:34 +01:00
Tamas Nemeth
62e33e03a3
fix(ingest/unity): Use assigned metastore if not metastore listed in unity catalog ( #7446 )
2023-02-28 08:06:28 +01:00
Tamas Nemeth
14a660428e
fix(ingest/bigquery): Querying table metadata details in batch properly ( #7429 )
2023-02-27 11:10:24 +01:00
Shirshanka Das
221b1ae801
fix(ingest): lookml - add support for includes, extends, view_name i… ( #7428 )
2023-02-24 12:05:21 -08:00
Shirshanka Das
95750317e1
refactor(ingest): lookml - fix up golden files in normalized form ( #7423 )
2023-02-24 00:10:18 -08:00
Andrew Sikowitz
0532cc9056
fix(ingest/bigquery) Filter upstream lineage by list of existing tables ( #7415 )
...
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
- Creates global stores table_refs and view_upstream_tables when extracting lineage
- Moves lineage processing to the end, after schema processing
- Adds `project_ids` config option to specify multiple projects to ingest; adds corresponding tests
- Changes `created` timestamps to `auditStamp` on `UpstreamClass`; uses VIEW type for lineage identified through view ddl parsing
2023-02-23 19:40:00 -05:00
Tamas Nemeth
4c1bf18f9a
feat(ingest/bigquery) - Emit cross-project usage from gcp logs ( #7364 )
2023-02-22 18:53:35 -05:00
Andrew Sikowitz
e82e284982
fix(ingest/kafka): Remove topic from kafka browse path ( #7398 )
2023-02-22 18:38:08 -05:00
Mayuri Nehate
d436ab9f9b
feat(ingest/kafka-connect): add config to lowercase urns, do not emit… ( #7393 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-02-22 11:42:44 -08:00
Andrew Sikowitz
2764c44977
fix(ingest): Do not require platform_instance for stateful ingestion ( #7397 )
2023-02-21 21:27:44 -05:00
서재권(Data Platform)
3068e7f0b1
fix(ingest/oracle) add database name to oracle urn name ( #7016 )
2023-02-21 13:50:24 -05:00
Andrew Sikowitz
8fd2cc5f20
fix(ingest/snowflake): Improve memory usage of metadata extraction ( #7349 )
2023-02-20 14:46:10 +01:00
Aseem Bansal
986086ae00
test(cli): add check for missing init files ( #7378 )
2023-02-20 18:41:12 +05:30
Shirshanka Das
07e4d0696f
feat(ingest): json-schema - add json schema support for files and kaf… ( #7361 )
2023-02-19 08:43:13 -08:00
Teppo Naakka
702221089d
feat(powerbi): add chart entities to similar browsepath as dashboards ( #7293 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-02-17 13:38:48 -08:00
Aseem Bansal
372f673aef
chore(ci): mark tests correctly ( #7337 )
2023-02-15 16:32:53 +05:30
skrydal
8207e4637a
fix(ingest/tableau): make Tableau ingestor resilient to timeout exceptions ( #7333 )
2023-02-15 11:21:31 +01:00
mohdsiddique
3a095f960f
feat(ingestion): powerbi # Configurable Admin API ( #7055 )
...
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2023-02-14 09:58:34 -08:00
Mayuri Nehate
2cffec9452
fix(check upgrade): update logic to compare server and client version ( #7238 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-02-13 13:09:38 -08:00
Andrew Sikowitz
8901498582
fix(transformers): pattern add domain transformer - enable replace_existing ( #7317 )
2023-02-13 12:52:44 -08:00
Tamas Nemeth
f10d622e47
fix(ingest/bigquery): Improve memory usage of lineage extraction ( #7326 )
2023-02-13 19:59:11 +01:00
Tamas Nemeth
b34e4fe1f1
fix(ingest/bigquery): Fix for table cache was not cleared ( #7323 )
2023-02-13 19:04:19 +01:00
Harshal Sheth
55442042ff
feat(cli): improve startup time ( #7292 )
2023-02-10 21:36:01 +05:30
Aseem Bansal
ca65f9db18
refactor(github): change github reference to git references ( #7308 )
2023-02-10 21:07:44 +05:30
Tamas Nemeth
1402071e48
fix(ingest/bigquery) - Fix for Bigquery parser quoted semicolon in the FROM table name as well ( #7277 )
2023-02-08 10:18:55 +01:00
