Tamas Nemeth
1b53c03794
fix(ingest/snowflake): fixing Snowflake state issue ( #7443 )
2023-02-27 13:59:30 +01:00
Tamas Nemeth
14a660428e
fix(ingest/bigquery): Querying table metadata details in batch properly ( #7429 )
2023-02-27 11:10:24 +01:00
Harshal Sheth
d02701d91c
docs(ingest): add ingestion configs guide ( #7438 )
2023-02-26 16:04:23 -08:00
Shirshanka Das
221b1ae801
fix(ingest): lookml - add support for includes, extends, view_name i… ( #7428 )
2023-02-24 12:05:21 -08:00
Tamas Nemeth
3a4c9a69f6
fix(ingest/bigquery): Fixing double quoting in profiling approx count query ( #7416 )
2023-02-24 09:39:52 +01:00
Shirshanka Das
95750317e1
refactor(ingest): lookml - fix up golden files in normalized form ( #7423 )
2023-02-24 00:10:18 -08:00
Andrew Sikowitz
0532cc9056
fix(ingest/bigquery) Filter upstream lineage by list of existing tables ( #7415 )
...
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
- Creates global stores table_refs and view_upstream_tables when extracting lineage
- Moves lineage processing to the end, after schema processing
- Adds `project_ids` config option to specify multiple projects to ingest; adds corresponding tests
- Changes `created` timestamps to `auditStamp` on `UpstreamClass`; uses VIEW type for lineage identified through view ddl parsing
2023-02-23 19:40:00 -05:00
Tamas Nemeth
4c1bf18f9a
feat(ingest/bigquery) - Emit cross-project usage from gcp logs ( #7364 )
2023-02-22 18:53:35 -05:00
Andrew Sikowitz
e82e284982
fix(ingest/kafka): Remove topic from kafka browse path ( #7398 )
2023-02-22 18:38:08 -05:00
Mayuri Nehate
d436ab9f9b
feat(ingest/kafka-connect): add config to lowercase urns, do not emit… ( #7393 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-02-22 11:42:44 -08:00
Mayuri Nehate
5db133619f
fix(ingest/bigquery): Prefer parsed lineage for view over lineage from audit logs ( #7408 )
2023-02-22 11:51:04 -05:00
Andrew Sikowitz
c5c2bdb983
fix(ingest/bigquery): Correctly upsert lineage_map when parsing view ddl ( #7403 )
2023-02-22 11:57:01 +01:00
Andrew Sikowitz
2764c44977
fix(ingest): Do not require platform_instance for stateful ingestion ( #7397 )
2023-02-21 21:27:44 -05:00
Chris Collins
2de779adbf
fix(docs): Update transformers docs to note not minting urns ( #7399 )
2023-02-21 13:29:36 -08:00
서재권(Data Platform)
3068e7f0b1
fix(ingest/oracle) add database name to oracle urn name ( #7016 )
2023-02-21 13:50:24 -05:00
Andrew Sikowitz
1402e88e3a
build(idea): mark metadata-ingestion sources and tests ( #7394 )
2023-02-21 09:50:03 -05:00
Aseem Bansal
f8a73005d4
chore(ci): relax bigquery dependency ( #7309 )
2023-02-21 08:33:00 +01:00
Tamas Nemeth
097d4e6bbd
fix(dep/json-schema): Fixing json-schema dependencies ( #7383 )
2023-02-20 14:02:08 -08:00
John Joyce
08a215951c
feat(queries): Overhaul Queries Tab ( #7366 )
2023-02-20 11:10:18 -08:00
Andrew Sikowitz
8fd2cc5f20
fix(ingest/snowflake): Improve memory usage of metadata extraction ( #7349 )
2023-02-20 14:46:10 +01:00
Aseem Bansal
986086ae00
test(cli): add check for missing init files ( #7378 )
2023-02-20 18:41:12 +05:30
Shirshanka Das
07e4d0696f
feat(ingest): json-schema - add json schema support for files and kaf… ( #7361 )
2023-02-19 08:43:13 -08:00
Andrew Sikowitz
a605f0752f
fix(deps): pin snowflake-connector-python ( #7365 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-02-18 10:44:55 +01:00
Andrew Sikowitz
632f730803
fix(ingest/looker): do not instantiate LookerDashboardSource on test_connection ( #7369 )
2023-02-18 09:32:28 +01:00
Tim
ad742f14c8
refactor(ingest,athena): update athena sample recipe ( #7368 )
2023-02-17 16:24:24 -08:00
Teppo Naakka
702221089d
feat(powerbi): add chart entities to similar browsepath as dashboards ( #7293 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-02-17 13:38:48 -08:00
mohdsiddique
79f576e2e1
fix(ingestion): powerbi # continue ingestion if m-query parsing fail ( #7360 )
...
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2023-02-17 13:24:10 -08:00
Tamas Nemeth
32ab7949de
fix(ingestion/snowflake): Fixing stateful ingestion commit at Snowflake source ( #7363 )
2023-02-17 13:23:16 -08:00
Pedro Silva
50f7935d5b
fix(cli): Corrects search filter for delete ( #7367 )
2023-02-17 13:23:01 -08:00
Aseem Bansal
5690560b6e
feat(cli): make deprecations, renames easier to notice ( #7310 )
2023-02-17 22:50:41 +05:30
Tamas Nemeth
aa388f04c2
fix(ingest/bigquery): Increase batch size in metadata extraction if no partitioned table involved ( #7252 )
2023-02-17 11:49:47 +01:00
Aseem Bansal
372f673aef
chore(ci): mark tests correctly ( #7337 )
2023-02-15 16:32:53 +05:30
skrydal
8207e4637a
fix(ingest/tableau): make Tableau ingestor resilient to timeout exceptions ( #7333 )
2023-02-15 11:21:31 +01:00
Shirshanka Das
46810e0df9
logging(cli): dropping neo4j message to debug to avoid confusion ( #7340 )
2023-02-14 11:32:03 -08:00
mohdsiddique
3a095f960f
feat(ingestion): powerbi # Configurable Admin API ( #7055 )
...
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
2023-02-14 09:58:34 -08:00
Mayuri Nehate
2cffec9452
fix(check upgrade): update logic to compare server and client version ( #7238 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-02-13 13:09:38 -08:00
Andrew Sikowitz
8901498582
fix(transformers): pattern add domain transformer - enable replace_existing ( #7317 )
2023-02-13 12:52:44 -08:00
Felix Lüdin
da2b0c9e1b
fix(docs): sort sources by display name in doc's sidebar ( #7322 )
2023-02-13 12:39:54 -08:00
Tamas Nemeth
f10d622e47
fix(ingest/bigquery): Improve memory usage of lineage extraction ( #7326 )
2023-02-13 19:59:11 +01:00
Tamas Nemeth
b34e4fe1f1
fix(ingest/bigquery): Fix for table cache was not cleared ( #7323 )
2023-02-13 19:04:19 +01:00
Harshal Sheth
76846b4175
perf(ingest): speed up MCPW.validate() ( #7319 )
2023-02-11 23:42:28 +01:00
Harshal Sheth
582fdf95cd
chore(ingest): upgrade to mypy 1.0.0 ( #7313 )
2023-02-10 13:24:05 -08:00
Tamas Nemeth
793f303a79
fix(ingest/bigquery): Lowering significantly the memory usage of the BigQuery connector ( #7315 )
2023-02-10 13:12:02 -08:00
Harshal Sheth
55442042ff
feat(cli): improve startup time ( #7292 )
2023-02-10 21:36:01 +05:30
Aseem Bansal
ca65f9db18
refactor(github): change github reference to git references ( #7308 )
2023-02-10 21:07:44 +05:30
Tamas Nemeth
97355bc805
fix(ingest/bigquery): Usage rate limiting and lineage exported log fix ( #7297 )
2023-02-10 20:22:49 +05:30
Shirshanka Das
fc12fad74c
fix(docs): kafka - update docs to indicate protobuf support ( #7280 )
...
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
2023-02-09 14:40:00 -08:00
Tamas Nemeth
bf7c00d88d
feat(ingest/bigquery) - Reporting current state of BigQuery ingestion ( #7282 )
2023-02-08 20:22:49 +01:00
Tamas Nemeth
1402071e48
fix(ingest/bigquery) - Fix for Bigquery parser quoted semicolon in the FROM table name as well ( #7277 )
2023-02-08 10:18:55 +01:00
Shirshanka Das
e0f9ba1d1f
feat(cli): add 0.10 awareness to upgrade prompt ( #7273 )
2023-02-07 16:50:40 -08:00
