Hyejin Yoon
2bc0a781a6
fix: refactor toc ( #7862 )
2023-04-21 18:36:10 -07:00
Andrew Sikowitz
e9c2f9afcc
feat(ingest/unity): Ingest ownership for containers; lookup service principal display names ( #7869 )
2023-04-21 11:02:39 -07:00
mohdsiddique
f21eeed6e7
feat(ingestion): lookml refinement support ( #7781 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-21 10:55:31 -07:00
Yusuf Mahtab
fa10256c47
feat(glue): allow resource links to be ignored ( #7639 )
...
Co-authored-by: Justas Cernas <justas.cernas@fundingcircle.com>
2023-04-21 10:42:32 -07:00
Aezo
1a5c716b87
feat(ingest/powerbi): support modified_since, extract_dataset_schema and many more ( #7519 )
...
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-20 22:58:45 -07:00
eeepmb
2616a16ec8
docs(ingest/powerbi): update workspace concept mapping ( #7835 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-04-20 22:03:36 -07:00
Harshal Sheth
f37ca4e49c
docs(ingest): fix CorpGroup example ( #7816 )
2023-04-20 21:09:12 -07:00
Harshal Sheth
66f44945e3
docs(ingest): update dbt and aws docs ( #7870 )
2023-04-20 21:08:22 -07:00
Andrew Sikowitz
1ff6949e36
refactor(ingest): Add helper DataHubGraph methods ( #7851 )
...
Adds:
- get_urns_by_filter(), using scroll by entities
- get_latest_pipeline_checkpoint()
- soft_delete_urn()
2023-04-20 10:16:33 -07:00
Aseem Bansal
535e1abe44
chore(ci): fix CI failing due to lint ( #7863 )
2023-04-20 16:53:36 +05:30
Harshal Sheth
6802142f6e
fix(ingest/salesforce): use report timestamp for operations ( #7838 )
...
Co-authored-by: John Joyce <john@acryl.io>
2023-04-19 20:39:07 -07:00
Hyejin Yoon
e5d06733f2
feat(docs): consolidate api guides ( #7857 )
...
Co-authored-by: socar-dini <dini@socar.kr>
2023-04-20 12:17:11 +09:00
Hyejin Yoon
ea4036c1c8
feat: enriching guide on creating dataset ( #7777 )
...
Co-authored-by: Hyejin Yoon <hyejin.yoon@acryl.io>
Co-authored-by: socar-dini <dini@socar.kr>
2023-04-19 12:58:03 +09:00
Harshal Sheth
f0ea79060b
chore(ingest): bug fix in sqlparse pin ( #7848 )
2023-04-18 16:05:23 -07:00
Harshal Sheth
cf7eb570a0
fix(ingest): pin sqlparse version ( #7847 )
2023-04-18 14:25:42 -07:00
John Joyce
b46822399c
feat(timeseries): Support sorting timeseries aspects by non-timestampMillis field + fix operations resolver ( #7840 )
2023-04-18 09:10:04 -07:00
Harshal Sheth
399e3333ad
feat(cli): improve quickstart stability ( #7839 )
2023-04-17 21:19:19 -07:00
Harshal Sheth
e461d03d94
feat(ingest/unity): capture create/lastModified timestamps ( #7819 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
2023-04-17 12:18:21 -07:00
Mayuri Nehate
a8681dae75
fix(ingest/snowflake): fix column name in snowflake optimised lineage ( #7834 )
2023-04-17 11:44:53 -07:00
Harshal Sheth
af566e1184
feat(model): fully populate the entity registry ( #7818 )
2023-04-15 13:33:05 -07:00
Harshal Sheth
342830c68c
fix(cli): use correct ingestion image in script ( #7826 )
2023-04-14 23:47:08 -07:00
Andrew Sikowitz
1ac1ccf26e
perf(ingest/bigquery): Improve bigquery usage disk usage and speed ( #7825 )
2023-04-14 18:09:43 -07:00
Andrew Sikowitz
e839ac4c40
fix(ingest/bigquery): Handle null values from usage aggregation ( #7827 )
2023-04-14 16:54:22 -07:00
Mayuri Nehate
8ec74ce41c
fix(ingest/bigquery): update usage query, remove erroneous init ( #7811 )
2023-04-14 13:38:50 -07:00
Andrew Sikowitz
37e7485184
fix(ingest/bigquery): Do not query columns when not ingesting tables or views ( #7823 )
2023-04-14 09:08:22 -07:00
Andrew Sikowitz
408cd7db2a
fix(ingest/bigquery): Enable lineage and usage ingestion without tables ( #7820 )
2023-04-14 01:41:00 -07:00
Andrew Sikowitz
d8d8176b1a
fix(ingest/bigquery): Add to lineage, not overwrite, when using sql parser ( #7814 )
2023-04-14 08:46:10 +02:00
Tamas Nemeth
4ec280ee20
fix(ingest/redshift): Remove pg_user table from metadata queries ( #7815 )
2023-04-13 15:35:26 -07:00
Andrew Sikowitz
ce795406b9
feat(ingest): Track disk usage in report ( #7812 )
2023-04-13 14:43:25 -07:00
RyanHolstien
0d5873db2a
feat(patch): patch support for flow info and job info and refactor patchbuilders for java sdk ( #7495 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: david-leifker <114954101+david-leifker@users.noreply.github.com>
Co-authored-by: David Leifker <david.leifker@acryl.io>
2023-04-13 15:46:35 -05:00
Harshal Sheth
4f59169566
feat(ingest/lookml): correctly handle include directives from imported projects ( #7798 )
2023-04-13 13:28:58 -07:00
Harshal Sheth
204727a6ee
feat(ingest/unity): support extracting ownership ( #7801 )
2023-04-12 19:45:41 -07:00
Harshal Sheth
3079f0a7e1
feat(sdk): support executing graphql via DataHubGraph ( #7753 )
...
Co-authored-by: Hyejin Yoon <0327jane@gmail.com>
2023-04-12 11:30:05 -07:00
Andrew Sikowitz
73016ebff9
test(ingest/bigquery): Add sql parser xfail test to fix later ( #7792 )
2023-04-12 10:51:29 -07:00
Tamas Nemeth
0cc12bcce7
feat(ingest): redshift - Redshift rework ( #6906 )
2023-04-12 19:15:43 +02:00
Mayuri Nehate
ec1228f67d
fix(dep): add sqllineage dependency for tableau ( #7803 )
2023-04-12 15:33:31 +02:00
Andrew Sikowitz
b7feb2a671
config(ingest/bigquery): Default lineage_use_sql_parser to true; update description ( #7797 )
2023-04-11 23:00:41 -07:00
Andrew Sikowitz
156d9df6b5
fix(ingest/bigquery): Fix lineage / usage table ref checks ( #7800 )
2023-04-11 23:00:27 -07:00
Andrew Sikowitz
54f047e1a8
test(ingest/snowflake): fix tests around host_port ( #7791 )
...
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-11 16:06:35 -07:00
David Sanchez
a50c71264d
feat(ingest/tableau): extract lineage from csql queries ( #7561 )
...
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
2023-04-11 11:12:15 -07:00
Mayuri Nehate
cf365c32fd
fix(ingest/snowflake): fix to not emit upstream external lineage for non-existent, non-allowed entities ( #7778 )
2023-04-11 09:08:56 -07:00
Harshal Sheth
905d7fed66
chore(ingest): cleanup unused fields in bigquery/snowflake ( #7787 )
2023-04-11 09:49:13 +02:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting ( #7763 )
2023-04-10 14:14:42 -07:00
Mayuri Nehate
f563695bc6
fix(ingest/snowflake): fix tags without lineage query, remove comma ( #7779 )
2023-04-10 13:18:45 -07:00
Hyejin Yoon
294c5ff507
feat(docs): refactor guide on graphql ( #7745 )
...
Co-authored-by: Hyejin Yoon <yoonhyejin@Hyejins-MacBook-Pro.local>
Co-authored-by: Hyejin Yoon <hyejin.yoon@acryl.io>
2023-04-08 08:26:58 +09:00
mohdsiddique
5e145cbb2d
feat(ingestion/okta): okta stateful ingestion ( #7736 )
...
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
2023-04-07 13:44:32 -07:00
Mayuri Nehate
5fd7981532
fix(ingest/snowflake): fix incorrect tag urn case, improve tag display name ( #7758 )
2023-04-07 13:07:08 -07:00
Andrew Sikowitz
087855f374
fix(ingest/bigquery): Support cross project usage using FileBackedDict ( #7663 )
...
Includes major refactor of bigquery usage ingestion, minor refactor of the source as a whole, and reporting cleanup.
Includes bigquery performance testing changes.
2023-04-07 12:18:26 -07:00
Mayuri Nehate
1fda92441f
feat(snowflake): improve snowflake lineage perf and memory, push down to snowflake ( #7710 )
2023-04-07 11:06:06 -07:00
Andrew Sikowitz
44663fa035
fix(ingest/bigquery): Raise report_failure threshold; add robustness around table parsing ( #7772 )
...
- Converted getting views and tables to iterators
- Catches exception around table expiration time being impossible to represent in python because it's too far in the future
2023-04-06 13:24:22 -07:00
