yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-19 07:30:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
11,753 Commits 200 Branches 121 Tags
Commit Graph

927 Commits

Author SHA1 Message Date
Andrew Sikowitz
de587b2bfe
refactor(ingest): Minor cleanup of File, CsvEnricher, BusinessGlossary, and FileLineage sources (#7718) 
- Adds auto_workunit_reporter to each source
- Standardizes comments around remote paths
- Adds back AuditStamp to FileLineage source
- Some generic refactoring
 2023-03-31 15:49:24 -07:00
xiphl
7d240c600a
feat(ingestion) Allow for ingestion to read files remotely (#7552) 
Co-authored-by: xiphl <xiphlerl9@gmail.com>
Allows the CsvEnricher, BusinessGlossary, File, and LineageFile sources to read from URLs.
 2023-03-29 18:10:46 -07:00
Harshal Sheth
94fa62d431
chore(ingest): formatting + cleanup MCPW usages (#7706) 2023-03-29 11:43:25 -07:00
mohdsiddique
c0f7ba2f85
feat(ingestion): azure-ad stateful ingestion (#7701) 2023-03-29 21:50:31 +05:30
Mayuri Nehate
fc238c2513
feat(ingest/postgres): support extracting metadata from all databases in single recipe (#7581) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-03-28 14:16:12 -07:00
Harshal Sheth
b2689b7514
test(ingest/dbt): add test for column meta match (#7673) 2023-03-28 21:51:31 +05:30
Harshal Sheth
d1bab5616c
feat(ingest/looker): enable looker usage ingestion by default (#7684) 2023-03-27 00:02:25 +00:00
Harshal Sheth
c8abf9a1d4
fix(ingest/dbt): enable incremental lineage by default (#7674) 2023-03-24 18:14:19 -04:00
mohdsiddique
6d6d59141e
feat(ingestion): powerbi # uniquly identify the multiple instance of same platform (#7632) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-21 09:27:29 -07:00
mohdsiddique
7efac2215d
feat(ingestion): powerbi # support platform instance (#7583) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-21 09:07:31 -07:00
Harshal Sheth
cbd8e14b78
feat(ingest): add auto_materialize_referenced_tags helper (#7626) 2023-03-20 16:34:22 -07:00
mohdsiddique
fc8757d25e
feat(ingestion): powerbi # Amazon Redshift lineage support (#7562) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-20 10:24:34 -07:00
mohdsiddique
0a36826407
fix(ingest/powerbi): support each expression in m-query function invocation (#7541) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-15 09:06:29 -07:00
Harsha Mandadi
bf36c935fa
feat(ingest/s3): support path_specs of different S3 buckets in the same recipe (#7514) 2023-03-14 21:55:57 -07:00
Mayuri Nehate
70c9b20c6d
feat(ingest/kafka-connect): add stateful ingestion and platform instance support (#7526) 2023-03-14 13:59:54 -07:00
John Joyce
18f387c6ea
fix(cli): Adding exit code to correctly return failure or success (#7520) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
Co-authored-by: Aseem Bansal <asmbansal2@gmail.com>
 2023-03-13 13:32:40 -07:00
Teppo Naakka
23830423f8
feat(ingest): powerbi # store powerbi entity descriptions (#7154) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-13 13:31:21 -07:00
Shirshanka Das
26cf0a71ab
fix(test): suppress s3 golden file test for specific paths (#7551) 2023-03-12 10:43:02 -07:00
Harshal Sheth
91a8e5ea16
feat(ingest/powerbi): support PowerBI parameter references (#7523) 2023-03-09 11:14:55 -08:00
J Feldman
aa4228734c
feat(ingest/looker): upgrade to Looker API from 3.1 to 4.0 (#7411) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2023-03-07 16:49:37 -08:00
Arun Vasudevan
e93a17e917
feat(ingest/postgres): support estimated row counts in profiling (#7476) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-07 11:08:09 -08:00
mohdsiddique
de719663ff
feat(ingestion): powerbi # support Google BigQuery table lineage (#7502) 
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-03-07 09:07:00 -08:00
Mayuri Nehate
406b11a9ed
feat(ingest/GX): add urn lowercasing option for GX assertions (#7472) 2023-03-06 20:42:23 -08:00
mohdsiddique
29d171106b
feat(ingest/tableau): project path and container support (#7426) 
Co-authored-by: mayurinehate <mayuri.nehate@gslab.com>
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: John Joyce <john@acryl.io>
 2023-03-02 16:53:19 -08:00
Aseem Bansal
1adbc2cab0
chore(ci): upgrade GE version (#7290) 2023-03-02 10:47:38 -08:00
Harshal Sheth
49029943f9
fix(ingest): remove extraneous platform configs (#7454) 2023-03-02 01:10:35 -08:00
Shirshanka Das
17e85979dd
refactor(ingest): subtypes - standardize (#7437) 2023-02-28 13:11:07 -08:00
nachiket-juneja
e07cd2090b
Feat/s3 ingestion enhancement to update schema from latest partition (#7410) 
Co-authored-by: Prashant Singh Thakur <prashant.thakur@nucleusteq.com>
 2023-02-28 08:58:28 +01:00
Harshal Sheth
3b8b5e8aa4
chore(ingest): cleanup unused files/vars in tests (#7450) 2023-02-28 08:07:34 +01:00
Tamas Nemeth
62e33e03a3
fix(ingest/unity): Use assigned metastore if not metastore listed in unity catalog (#7446) 2023-02-28 08:06:28 +01:00
Shirshanka Das
221b1ae801
fix(ingest): lookml - add support for includes, extends, view_name i… (#7428) 2023-02-24 12:05:21 -08:00
Shirshanka Das
95750317e1
refactor(ingest): lookml - fix up golden files in normalized form (#7423) 2023-02-24 00:10:18 -08:00
Andrew Sikowitz
e82e284982
fix(ingest/kafka): Remove topic from kafka browse path (#7398) 2023-02-22 18:38:08 -05:00
Mayuri Nehate
d436ab9f9b
feat(ingest/kafka-connect): add config to lowercase urns, do not emit… (#7393) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-02-22 11:42:44 -08:00
서재권(Data Platform)
3068e7f0b1
fix(ingest/oracle) add database name to oracle urn name (#7016) 2023-02-21 13:50:24 -05:00
Andrew Sikowitz
8fd2cc5f20
fix(ingest/snowflake): Improve memory usage of metadata extraction (#7349) 2023-02-20 14:46:10 +01:00
Teppo Naakka
702221089d
feat(powerbi): add chart entities to similar browsepath as dashboards (#7293) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-02-17 13:38:48 -08:00
Aseem Bansal
372f673aef
chore(ci): mark tests correctly (#7337) 2023-02-15 16:32:53 +05:30
skrydal
8207e4637a
fix(ingest/tableau): make Tableau ingestor resilient to timeout exceptions (#7333) 2023-02-15 11:21:31 +01:00
mohdsiddique
3a095f960f
feat(ingestion): powerbi # Configurable Admin API (#7055) 
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
 2023-02-14 09:58:34 -08:00
Aseem Bansal
ca65f9db18
refactor(github): change github reference to git references (#7308) 2023-02-10 21:07:44 +05:30
Aditya Radhakrishnan
bd17dde7b0
fix(ingest-idp): emit empty GroupMembership when there are no groups (#7196) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-02-02 11:56:40 -08:00
Dmitry Bryazgin
1a1f2906db
feat(ingest): Add custom properties to the ldap ingestion (#7125) 
Co-authored-by: Harshal Sheth <hsheth2@gmail.com>
 2023-02-01 16:37:14 -08:00
Harshal Sheth
670342ab64
feat(ingest): switch dbt to use auto_stale_entity_removal (#7160) 2023-01-30 19:59:20 -08:00
mohdsiddique
54562cd2d5
fix(ingestion/powerbi): Not all arguments converted to string (#7157) 
Co-authored-by: MohdSiddique Bagwan <mohdsiddique.bagwan@gslab.com>
 2023-01-27 18:49:27 +01:00
Teppo Naakka
2b43bce3d6
feat(ingest): powerbi # add powerbi workspaces to containers (#6532) 
Co-authored-by: John Joyce <john@acryl.io>
 2023-01-24 15:06:50 -08:00
Tamas Nemeth
0cdb5e4b4b
refactor(ingest/containers): Refactoring container creation to common place (#6877) 2023-01-21 00:14:31 +01:00
Harshal Sheth
d7aa61285b
fix(ingest): support git clone of non-github repos (#7065) 2023-01-18 13:30:24 -08:00
Harshal Sheth
fc41f455a0
feat(ingest): support snapshots in dbt and dbt-cloud (#7062) 
Co-authored-by: Tamas Nemeth <treff7es@gmail.com>
 2023-01-18 08:35:03 -08:00
Mayuri Nehate
4e7faa5503
fix(ingest/tableau): fix node limit exceeded error for workbooks query (#7068) 2023-01-18 14:34:40 +01:00