RyanHolstien
0d5873db2a
feat(patch): patch support for flow info and job info and refactor patchbuilders for java sdk ( #7495 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: David Leifker <david.leifker@acryl.io>
2023-04-13 15:46:35 -05:00
Harshal Sheth
4f59169566
feat(ingest/lookml): correctly handle include directives from imported projects ( #7798 )
2023-04-13 13:28:58 -07:00
Harshal Sheth
204727a6ee
feat(ingest/unity): support extracting ownership ( #7801 )
2023-04-12 19:45:41 -07:00
Harshal Sheth
3079f0a7e1
feat(sdk): support executing graphql via DataHubGraph ( #7753 )
...
Co-authored-by: Hyejin Yoon <0327jane@gmail.com>
2023-04-12 11:30:05 -07:00
Andrew Sikowitz
73016ebff9
test(ingest/bigquery): Add sql parser xfail test to fix later ( #7792 )
2023-04-12 10:51:29 -07:00
Tamas Nemeth
0cc12bcce7
feat(ingest): redshift - Redshift rework ( #6906 )
2023-04-12 19:15:43 +02:00
Mayuri Nehate
ec1228f67d
fix(dep): add sqllineage dependency for tableau ( #7803 )
2023-04-12 15:33:31 +02:00
Andrew Sikowitz
b7feb2a671
config(ingest/bigquery): Default lineage_use_sql_parser to true; update description ( #7797 )
2023-04-11 23:00:41 -07:00
Andrew Sikowitz
156d9df6b5
fix(ingest/bigquery): Fix lineage / usage table ref checks ( #7800 )
2023-04-11 23:00:27 -07:00
Andrew Sikowitz
54f047e1a8
test(ingest/snowflake): fix tests around host_port ( #7791 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-11 16:06:35 -07:00
David Sanchez
a50c71264d
feat(ingest/tableau): extract lineage from csql queries ( #7561 )
...
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-11 11:12:15 -07:00
Mayuri Nehate
cf365c32fd
fix(ingest/snowflake): fix to not emit upstream external lineage for non-existent, non-allowed entities ( #7778 )
2023-04-11 09:08:56 -07:00
Harshal Sheth
905d7fed66
chore(ingest): cleanup unused fields in bigquery/snowflake ( #7787 )
2023-04-11 09:49:13 +02:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting ( #7763 )
2023-04-10 14:14:42 -07:00
Mayuri Nehate
f563695bc6
fix(ingest/snowflake): fix tags without lineage query, remove comma ( #7779 )
2023-04-10 13:18:45 -07:00
Hyejin Yoon
294c5ff507
feat(docs): refactor guide on graphql ( #7745 )
...
Co-authored-by: Hyejin Yoon <yoonhyejin@Hyejins-MacBook-Pro.local>
Co-authored-by: Hyejin Yoon <hyejin.yoon@acryl.io>
2023-04-08 08:26:58 +09:00
mohdsiddique
5e145cbb2d
feat(ingestion/okta): okta stateful ingestion ( #7736 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
2023-04-07 13:44:32 -07:00
Mayuri Nehate
5fd7981532
fix(ingest/snowflake): fix incorrect tag urn case, improve tag display name ( #7758 )
2023-04-07 13:07:08 -07:00
Andrew Sikowitz
087855f374
fix(ingest/bigquery): Support cross project usage using FileBackedDict ( #7663 )
...
Includes major refactor of bigquery usage ingestion, minor refactor of the source as a whole, and reporting cleanup.
Includes bigquery performance testing changes.
2023-04-07 12:18:26 -07:00
Mayuri Nehate
1fda92441f
feat(snowflake): improve snowflake lineage perf and memory, push down to snowflake ( #7710 )
2023-04-07 11:06:06 -07:00
Andrew Sikowitz
44663fa035
fix(ingest/bigquery): Raise report_failure threshold; add robustness around table parsing ( #7772 )
...
- Converted getting views and tables to iterators
- Catches exception around table expiration time being impossible to represent in python because it's too far in the future
2023-04-06 13:24:22 -07:00
Tamas Nemeth
37b350c8e3
fix(ingest/redshift): fixing sql which extracts lineage from insert queries ( #7770 )
2023-04-06 16:34:34 +02:00
Tamas Nemeth
96bacfc5d7
fix(ingest/redshift): Fixing adding back db name in redshift urn ( #7765 )
2023-04-06 11:45:10 +02:00
Tamas Nemeth
29d2492667
fix(ingest/bigquery): Lineage edges use datetime with timezone; correctly parse last_altered ( #7762 )
2023-04-06 02:46:50 +00:00
Harshal Sheth
2840cba68b
docs(ingest/lookml): update error message for Looker connection fetch ( #7756 )
2023-04-05 13:41:17 -07:00
Harshal Sheth
5bb0e60bd3
fix(ingest/dbt-cloud): use correct dbt cloud IDE urls ( #7755 )
2023-04-05 13:40:56 -07:00
Harshal Sheth
e06117af66
fix(ingest/demo-data): fix bug in path type ( #7749 )
2023-04-04 23:16:15 -07:00
Mayuri Nehate
20504aae70
fix(ingest/bigquery): fix and refractor exported audit logs query ( #7699 )
2023-04-05 11:17:25 +05:30
Harshal Sheth
e71c0d3490
feat(sdk): fix ownership emission for groups ( #7751 )
2023-04-05 11:15:06 +05:30
Aseem Bansal
a11a7fa9d0
feat(snowflake): better error message on key pair authentication ( #7734 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-05 00:46:07 +00:00
Harshal Sheth
8d99babf75
feat(ingest/dbt): update subtypes for dbt ( #7750 )
2023-04-04 17:11:23 -07:00
Andrew Sikowitz
06bc1c32e0
refactor(ingest/bigquery): Standardize audit log parsing and make TopKDict a DefaultDict ( #7738 )
...
- Moves get_sanitized_table_ref calls to ReadEvent / QueryEvent creation
- Standardizes how the audit log is read and parsed, unifying code when reading from gcp logging vs audit metadata (exported logs)
- Adds error handling around the parsing of each event, to catch errors from the new get_sanitized_table_ref calls
- Makes TopKDict inherit from DefaultDict and cleans up calls around that.
2023-04-04 11:58:48 -07:00
Andrew Sikowitz
ce1ac7fa12
refactor(ingest): Use sqlite.Row row_factory for FileBackedCollections ( #7739 )
2023-04-04 11:53:56 -07:00
Tim
23e57fffa2
fix(sdk): remove rest emitter to graph cache in CorpGroup ( #7743 )
2023-04-04 10:32:15 -07:00
Harshal Sheth
f860ce95c0
feat(ingest): emit state payloads as soft-deleted ( #7714 )
2023-04-04 17:06:21 +00:00
Harshal Sheth
82dc2b6393
feat(docs): clear up source configs ( #7720 )
2023-04-04 18:40:19 +05:30
Harshal Sheth
8394dcb538
chore(ingest): change kafka connect mapped ports ( #7728 )
2023-04-04 18:38:30 +05:30
Harshal Sheth
1634edaf25
feat(ingest/dbt): include dbt unique_id in properties ( #7737 )
2023-04-04 13:37:13 +05:30
Harshal Sheth
f780da4c0a
feat(ingest/lookml): support views with derived_table.explore_source ( #7704 )
2023-04-03 16:18:39 -07:00
Andrew Sikowitz
de587b2bfe
refactor(ingest): Minor cleanup of File, CsvEnricher, BusinessGlossary, and FileLineage sources ( #7718 )
...
- Adds auto_workunit_reporter to each source
- Standardizes comments around remote paths
- Adds back AuditStamp to FileLineage source
- Some generic refactoring
2023-03-31 15:49:24 -07:00
Andrew Sikowitz
a2f8c76388
feat(ingest/bigquery): Capture all operation types when ingesting operational stats ( #7723 )
2023-03-31 16:01:28 +05:30
Harshal Sheth
f6d7e1a325
feat(ingest/snowflake): hide host_port from snowflake docs ( #7717 )
2023-03-31 15:58:52 +05:30
Aseem Bansal
f0a675f9aa
docs(okta): add how to use email in urns ( #7708 )
2023-03-31 15:55:22 +05:30
xiphl
7d240c600a
feat(ingestion) Allow for ingestion to read files remotely ( #7552 )
...
Co-authored-by: xiphl <xiphlerl9@gmail.com>
Allows the CsvEnricher, BusinessGlossary, File, and LineageFile sources to read from URLs.
2023-03-29 18:10:46 -07:00
Harshal Sheth
575909e41c
feat(docs): support inlining code snippets from files ( #7712 )
2023-03-30 00:02:21 +00:00
Sergio Gómez Villamor
25808478cb
fix(ingestion): fix AssertionError in base_transformer ( #7702 )
...
Co-authored-by: Sergio Gomez Villamor <sergio.gomez.villamor@adevinta.com>
2023-03-29 16:15:57 -07:00
Andrew Sikowitz
54a372795b
test(ingest/bigquery): Add performance testing framework for bigquery usage ( #7690 )
...
- Creates metadata-ingestion/tests/performance directory
- Excludes metadata-ingestion/tests from docs generation
- Updates bigquery reporting around project state
2023-03-29 14:13:43 -07:00
Harshal Sheth
94fa62d431
chore(ingest): formatting + cleanup MCPW usages ( #7706 )
2023-03-29 11:43:25 -07:00
mohdsiddique
c0f7ba2f85
feat(ingestion): azure-ad stateful ingestion ( #7701 )
2023-03-29 21:50:31 +05:30
Tamas Nemeth
f348113b38
fix(ingest/redshift): Lineage query fix to work with the latest redshift ( #7698 )
2023-03-29 09:32:09 +02:00
