Mayuri Nehate
301c8616ed
refactor(ingest/bigquery): add inline comments + refactor in table name parsing ( #7609 )
2023-03-24 14:44:30 -04:00
Andrew Sikowitz
95f99198af
fix(ingest/bigquery): Pass whether view is materialized; pass last_altered correctly ( #7660 )
2023-03-22 13:40:57 -04:00
Tamas Nemeth
14a660428e
fix(ingest/bigquery): Querying table metadata details in batch properly ( #7429 )
2023-02-27 11:10:24 +01:00
Andrew Sikowitz
0532cc9056
fix(ingest/bigquery) Filter upstream lineage by list of existing tables ( #7415 )
...
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
- Creates global stores table_refs and view_upstream_tables when extracting lineage
- Moves lineage processing to the end, after schema processing
- Adds `project_ids` config option to specify multiple projects to ingest; adds corresponding tests
- Changes `created` timestamps to `auditStamp` on `UpstreamClass`; uses VIEW type for lineage identified through view ddl parsing
2023-02-23 19:40:00 -05:00
Tamas Nemeth
4c1bf18f9a
feat(ingest/bigquery) - Emit cross-project usage from gcp logs ( #7364 )
2023-02-22 18:53:35 -05:00
Tamas Nemeth
f10d622e47
fix(ingest/bigquery): Improve memory usage of lineage extraction ( #7326 )
2023-02-13 19:59:11 +01:00
Tamas Nemeth
b34e4fe1f1
fix(ingest/bigquery): Fix for table cache was not cleared ( #7323 )
2023-02-13 19:04:19 +01:00
Tamas Nemeth
ead0074169
deprecate(ingest): bigquery - Removing bigquery-legacy source ( #6851 )
...
Co-authored-by: John Joyce <john@acryl.io>
2022-12-29 13:19:05 -08:00
Tamas Nemeth
e41b455e14
fix(ingest): bigquery - sharded table support improvements ( #6789 )
2022-12-19 18:57:37 +01:00
Mayuri Nehate
7b88de89d5
fix(ingest): snowflake - allow profiling to work with geography type ( #6162 )
2022-10-10 08:05:09 -07:00
Mugdha Hardikar
a6dc669891
docs(bigquery): add changelog and unittest for profiling limits ( #5407 )
2022-07-19 09:39:09 +05:30
Aseem Bansal
336a628c5b
fix(bigquery): fix lineage bug, improve docs, add dataset filter config ( #4607 )
...
* fix(bigquery): fix metadata from exported logs, doc missing permission, improve logging, add tests
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-04-07 13:10:21 -07:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage ( #4111 )
2022-03-08 12:29:10 -08:00
Tamas Nemeth
3d02b5bec8
feat(ingest): bigquery - ignore temporary tables from lineage and connect edges directly ( #4160 )
2022-02-20 14:23:23 -08:00
Tamas Nemeth
5df5150e51
feat(ingest): Add config option to set Bigquery credential in source config ( #3786 )
2021-12-27 05:48:45 -08:00
