yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-17 06:12:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,004 Commits 196 Branches 118 Tags
Commit Graph

945 Commits

Author SHA1 Message Date
Mayuri Nehate
28986d8081
fix(ingestion/tableau): backward compatibility with version 2021.1 and above (#7864) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-24 11:08:56 -07:00
Mayuri Nehate
3212e74969
feat(ingest/snowflake): optionally emit all upstreams irrespective of recipe pattern (#7842) 2023-04-24 11:01:15 -07:00
Andrew Sikowitz
e9c2f9afcc
feat(ingest/unity): Ingest ownership for containers; lookup service principal display names (#7869) 2023-04-21 11:02:39 -07:00
mohdsiddique
f21eeed6e7
feat(ingestion): lookml refinement support (#7781) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-21 10:55:31 -07:00
Aezo
1a5c716b87
feat(ingest/powerbi): support modified_since, extract_dataset_schema and many more (#7519) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-20 22:58:45 -07:00
Harshal Sheth
6802142f6e
fix(ingest/salesforce): use report timestamp for operations (#7838) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-04-19 20:39:07 -07:00
Harshal Sheth
e461d03d94
feat(ingest/unity): capture create/lastModified timestamps (#7819) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-04-17 12:18:21 -07:00
Harshal Sheth
204727a6ee
feat(ingest/unity): support extracting ownership (#7801) 2023-04-12 19:45:41 -07:00
Tamas Nemeth
0cc12bcce7
feat(ingest): redshift - Redshift rework (#6906) 2023-04-12 19:15:43 +02:00
David Sanchez
a50c71264d
feat(ingest/tableau): extract lineage from csql queries (#7561) 
Co-authored-by: Mayuri Nehate <33225191+mayurinehate@users.noreply.github.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-04-11 11:12:15 -07:00
Harshal Sheth
e99875cac6
chore(ingest): enable flake8 bugbear linting (#7763) 2023-04-10 14:14:42 -07:00
mohdsiddique
5e145cbb2d
feat(ingestion/okta): okta stateful ingestion (#7736) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-04-07 13:44:32 -07:00
Mayuri Nehate
5fd7981532
fix(ingest/snowflake): fix incorrect tag urn case, improve tag display name (#7758) 2023-04-07 13:07:08 -07:00
Mayuri Nehate
1fda92441f
feat(snowflake): improve snowflake lineage perf and memory, push down to snowflake (#7710) 2023-04-07 11:06:06 -07:00
Harshal Sheth
8d99babf75
feat(ingest/dbt): update subtypes for dbt (#7750) 2023-04-04 17:11:23 -07:00
Harshal Sheth
8394dcb538
chore(ingest): change kafka connect mapped ports (#7728) 2023-04-04 18:38:30 +05:30
Harshal Sheth
1634edaf25
feat(ingest/dbt): include dbt unique_id in properties (#7737) 2023-04-04 13:37:13 +05:30
Harshal Sheth
f780da4c0a
feat(ingest/lookml): support views with derived_table.explore_source (#7704) 2023-04-03 16:18:39 -07:00
Andrew Sikowitz
de587b2bfe
refactor(ingest): Minor cleanup of File, CsvEnricher, BusinessGlossary, and FileLineage sources (#7718) 
- Adds auto_workunit_reporter to each source
- Standardizes comments around remote paths
- Adds back AuditStamp to FileLineage source
- Some generic refactoring
 2023-03-31 15:49:24 -07:00
xiphl
7d240c600a
feat(ingestion) Allow for ingestion to read files remotely (#7552) 
Co-authored-by: xiphl <xiphlerl9@gmail.com>
Allows the CsvEnricher, BusinessGlossary, File, and LineageFile sources to read from URLs.
 2023-03-29 18:10:46 -07:00
Harshal Sheth
94fa62d431
chore(ingest): formatting + cleanup MCPW usages (#7706) 2023-03-29 11:43:25 -07:00
mohdsiddique
c0f7ba2f85
feat(ingestion): azure-ad stateful ingestion (#7701) 2023-03-29 21:50:31 +05:30
Mayuri Nehate
fc238c2513
feat(ingest/postgres): support extracting metadata from all databases in single recipe (#7581) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-03-28 14:16:12 -07:00
Harshal Sheth
b2689b7514
test(ingest/dbt): add test for column meta match (#7673) 2023-03-28 21:51:31 +05:30
Harshal Sheth
d1bab5616c
feat(ingest/looker): enable looker usage ingestion by default (#7684) 2023-03-27 00:02:25 +00:00
Harshal Sheth
c8abf9a1d4
fix(ingest/dbt): enable incremental lineage by default (#7674) 2023-03-24 18:14:19 -04:00
mohdsiddique
6d6d59141e
feat(ingestion): powerbi # uniquly identify the multiple instance of same platform (#7632) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-21 09:27:29 -07:00
mohdsiddique
7efac2215d
feat(ingestion): powerbi # support platform instance (#7583) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-21 09:07:31 -07:00
Harshal Sheth
cbd8e14b78
feat(ingest): add auto_materialize_referenced_tags helper (#7626) 2023-03-20 16:34:22 -07:00
mohdsiddique
fc8757d25e
feat(ingestion): powerbi # Amazon Redshift lineage support (#7562) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-20 10:24:34 -07:00
mohdsiddique
0a36826407
fix(ingest/powerbi): support each expression in m-query function invocation (#7541) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-15 09:06:29 -07:00
Harsha Mandadi
bf36c935fa
feat(ingest/s3): support path_specs of different S3 buckets in the same recipe (#7514) 2023-03-14 21:55:57 -07:00
Mayuri Nehate
70c9b20c6d
feat(ingest/kafka-connect): add stateful ingestion and platform instance support (#7526) 2023-03-14 13:59:54 -07:00
John Joyce
18f387c6ea
fix(cli): Adding exit code to correctly return failure or success (#7520) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
 2023-03-13 13:32:40 -07:00
Teppo Naakka
23830423f8
feat(ingest): powerbi # store powerbi entity descriptions (#7154) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-13 13:31:21 -07:00
Shirshanka Das
26cf0a71ab
fix(test): suppress s3 golden file test for specific paths (#7551) 2023-03-12 10:43:02 -07:00
Harshal Sheth
91a8e5ea16
feat(ingest/powerbi): support PowerBI parameter references (#7523) 2023-03-09 11:14:55 -08:00
J Feldman
aa4228734c
feat(ingest/looker): upgrade to Looker API from 3.1 to 4.0 (#7411) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2023-03-07 16:49:37 -08:00
Arun Vasudevan
e93a17e917
feat(ingest/postgres): support estimated row counts in profiling (#7476) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-07 11:08:09 -08:00
mohdsiddique
de719663ff
feat(ingestion): powerbi # support Google BigQuery table lineage (#7502) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-07 09:07:00 -08:00
Mayuri Nehate
406b11a9ed
feat(ingest/GX): add urn lowercasing option for GX assertions (#7472) 2023-03-06 20:42:23 -08:00
mohdsiddique
29d171106b
feat(ingest/tableau): project path and container support (#7426) 
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-02 16:53:19 -08:00
Aseem Bansal
1adbc2cab0
chore(ci): upgrade GE version (#7290) 2023-03-02 10:47:38 -08:00
Harshal Sheth
49029943f9
fix(ingest): remove extraneous platform configs (#7454) 2023-03-02 01:10:35 -08:00
Shirshanka Das
17e85979dd
refactor(ingest): subtypes - standardize (#7437) 2023-02-28 13:11:07 -08:00
nachiket-juneja
e07cd2090b
Feat/s3 ingestion enhancement to update schema from latest partition (#7410) 
Co-authored-by: Prashant Singh Thakur <prashant.thakur@nucleusteq.com>
 2023-02-28 08:58:28 +01:00
Harshal Sheth
3b8b5e8aa4
chore(ingest): cleanup unused files/vars in tests (#7450) 2023-02-28 08:07:34 +01:00
Tamas Nemeth
62e33e03a3
fix(ingest/unity): Use assigned metastore if not metastore listed in unity catalog (#7446) 2023-02-28 08:06:28 +01:00
Shirshanka Das
221b1ae801
fix(ingest): lookml - add support for includes, extends, view_name i… (#7428) 2023-02-24 12:05:21 -08:00
Shirshanka Das
95750317e1
refactor(ingest): lookml - fix up golden files in normalized form (#7423) 2023-02-24 00:10:18 -08:00