Shubham Jagtap
9f2c5d36f3
feat(ingestion/bigquery): BigQuery Owner Label to Datahub Ownership ( #10047 )
2024-03-28 15:50:25 -07:00
Shubham Jagtap
f83a2fab44
fix(ingestion/bigquery): Table-view-snapshot Lineage Bug fix ( #9579 )
...
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-01-25 18:48:41 +05:30
Andrew Sikowitz
806f09ae23
test(ingest/unity): Unity catalog data generation ( #8949 )
2023-12-05 12:33:00 -05:00
Andrew Sikowitz
6c932e8afe
feat(ingest/bigquery): Attempt to support raw dataset pattern ( #9109 )
2023-10-25 13:17:09 -07:00
Tamas Nemeth
dd418de76d
fix(ingest/bigquery): Fix shard regexp to match without underscore as well ( #8934 )
2023-10-12 13:10:59 +02:00
Upendra Rao Vedullapalli
13508a9d88
feat(bigquery): excluding projects without any datasets from ingestion ( #8535 )
...
Co-authored-by: Upendra Vedullapalli <upendra.rao.vedullapalli@entur.org>
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-10-04 09:20:51 -04:00
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call ( #8626 )
2023-09-15 11:55:39 -07:00
Mayuri Nehate
e680a97046
fix(ingest/bigquery): fix partition and median queries for profiling ( #8778 )
2023-09-06 12:48:11 -07:00
Andrew Sikowitz
22c35f1a23
fix(ingest/bigquery): Add config option to create DataPlatformInstance, default off ( #8659 )
2023-08-24 14:46:06 +05:30
Harshal Sheth
d4135d57b7
feat(ingest/bigquery): support column-level lineage ( #8382 )
2023-07-11 11:12:51 -07:00
Harshal Sheth
3e47b3d228
feat(ingest): schema-aware SQL parsing for column-level lineage ( #8334 )
2023-07-07 16:24:35 -07:00
Andrew Sikowitz
c5cc53b99a
feat(ingest/bigquery_v2): enable platform instance using project id ( #8216 )
...
Co-authored-by: Adrián Pertíñez <khurzak92@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-06-14 09:50:21 -07:00
Andrew Sikowitz
369a04ae30
revert(ingest/bigquery): Do not emit DataPlatformInstance; remove references to platform_instance ( #8196 )
2023-06-09 13:44:24 +05:30
Adrián Pertíñez
743439c11d
feat(ingest/bigquery_v2): enable platform instance using project id ( #8142 )
...
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-06-05 15:17:40 -07:00
Andrew Sikowitz
8019d17aa6
fix(ingest/bigquery): Filter projects for lineage and usage ( #7954 )
2023-05-04 18:14:48 +02:00
Mayuri Nehate
031aee4298
fix(ingest/bigquery): fix handling of time decorator offset queries ( #7843 )
...
Co-authored-by: Andrew Sikowitz <andrew.sikowitz@acryl.io>
2023-04-25 13:51:20 -07:00
Andrew Sikowitz
44663fa035
fix(ingest/bigquery): Raise report_failure threshold; add robustness around table parsing ( #7772 )
...
- Converted getting views and tables to iterators
- Catches exception around table expiration time being impossible to represent in python because it's too far in the future
2023-04-06 13:24:22 -07:00
Tamas Nemeth
96bacfc5d7
fix(ingest/redshift): Fixing adding back db name in redshift urn ( #7765 )
2023-04-06 11:45:10 +02:00
Tamas Nemeth
29d2492667
fix(ingest/bigquery): Lineage edges use datetime with timezone; correctly parse last_altered ( #7762 )
2023-04-06 02:46:50 +00:00
Andrew Sikowitz
419bee8614
fix(ingest/bigquery): Fix BigQueryTableType enum accesses ( #7685 )
2023-03-25 00:08:11 +00:00
Mayuri Nehate
301c8616ed
refactor(ingest/bigquery): add inline comments + refactor in table name parsing ( #7609 )
2023-03-24 14:44:30 -04:00
Andrew Sikowitz
95f99198af
fix(ingest/bigquery): Pass whether view is materialized; pass last_altered correctly ( #7660 )
2023-03-22 13:40:57 -04:00
Tamas Nemeth
14a660428e
fix(ingest/bigquery): Querying table metadata details in batch properly ( #7429 )
2023-02-27 11:10:24 +01:00
Andrew Sikowitz
0532cc9056
fix(ingest/bigquery) Filter upstream lineage by list of existing tables ( #7415 )
...
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
- Creates global stores table_refs and view_upstream_tables when extracting lineage
- Moves lineage processing to the end, after schema processing
- Adds `project_ids` config option to specify multiple projects to ingest; adds corresponding tests
- Changes `created` timestamps to `auditStamp` on `UpstreamClass`; uses VIEW type for lineage identified through view ddl parsing
2023-02-23 19:40:00 -05:00
Tamas Nemeth
4c1bf18f9a
feat(ingest/bigquery) - Emit cross-project usage from gcp logs ( #7364 )
2023-02-22 18:53:35 -05:00
Tamas Nemeth
f10d622e47
fix(ingest/bigquery): Improve memory usage of lineage extraction ( #7326 )
2023-02-13 19:59:11 +01:00
Tamas Nemeth
b34e4fe1f1
fix(ingest/bigquery): Fix for table cache was not cleared ( #7323 )
2023-02-13 19:04:19 +01:00
Tamas Nemeth
ead0074169
deprecate(ingest): bigquery - Removing bigquery-legacy source ( #6851 )
...
Co-authored-by: John Joyce <john@acryl.io>
2022-12-29 13:19:05 -08:00
Tamas Nemeth
e41b455e14
fix(ingest): bigquery - sharded table support improvements ( #6789 )
2022-12-19 18:57:37 +01:00
Mayuri Nehate
7b88de89d5
fix(ingest): snowflake - allow profiling to work with geography type ( #6162 )
2022-10-10 08:05:09 -07:00
Mugdha Hardikar
a6dc669891
docs(bigquery): add changelog and unittest for profiling limits ( #5407 )
2022-07-19 09:39:09 +05:30
Aseem Bansal
336a628c5b
fix(bigquery): fix lineage bug, improve docs, add dataset filter config ( #4607 )
...
* fix(bigquery): fix metadata from exported logs, doc missing permission, improve logging, add tests
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-04-07 13:10:21 -07:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage ( #4111 )
2022-03-08 12:29:10 -08:00
Tamas Nemeth
3d02b5bec8
feat(ingest): bigquery - ignore temporary tables from lineage and connect edges directly ( #4160 )
2022-02-20 14:23:23 -08:00
Tamas Nemeth
5df5150e51
feat(ingest): Add config option to set Bigquery credential in source config ( #3786 )
2021-12-27 05:48:45 -08:00
