Mayuri Nehate
a7fc7f519a
feat(ingest/bq): integrate bigquery-queries into main source ( #11247 )
2024-08-30 18:16:45 -07:00
Mayuri Nehate
223650dd7a
feat(ingest): add bigquery-queries source ( #10994 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-08-25 22:51:00 -07:00
Tamas Nemeth
5c06f7a245
fix(ingest/bigquery): Supporting lineage extraction in case the select query result's target table is set on job ( #10191 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2024-04-02 22:13:05 +02:00
Harshal Sheth
0d780e5f8f
feat(ingest): sql parsing aggregator ( #9786 )
2024-02-09 16:27:45 -05:00
Mayuri Nehate
cdb9f5ba62
feat(bigquery): add better timers around every API call ( #8626 )
2023-09-15 11:55:39 -07:00
Andrew Sikowitz
a4e726872b
fix(ingest/bigquery): Filter out fine grained lineage with no upstreams ( #8758 )
2023-08-31 12:44:24 -04:00
Harshal Sheth
d4135d57b7
feat(ingest/bigquery): support column-level lineage ( #8382 )
2023-07-11 11:12:51 -07:00
Tamas Nemeth
29d2492667
fix(ingest/bigquery): Lineage edges use datetime with timezone; correctly parse last_altered ( #7762 )
2023-04-06 02:46:50 +00:00
Andrew Sikowitz
0532cc9056
fix(ingest/bigquery) Filter upstream lineage by list of existing tables ( #7415 )
...
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
- Creates global stores table_refs and view_upstream_tables when extracting lineage
- Moves lineage processing to the end, after schema processing
- Adds `project_ids` config option to specify multiple projects to ingest; adds corresponding tests
- Changes `created` timestamps to `auditStamp` on `UpstreamClass`; uses VIEW type for lineage identified through view ddl parsing
2023-02-23 19:40:00 -05:00
Tamas Nemeth
9a1f78fc60
fix(ingest): profiling - Changing profiling defaults on Bigquery and Snowflake ( #6640 )
2022-12-07 10:33:10 +01:00
Tamas Nemeth
6e34cd6001
feat(ingest): bigquery-beta - Parsing view ddl definition for lineage ( #6187 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-10-12 18:24:04 -07:00
