yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-09 02:02:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,004 Commits 189 Branches 118 Tags
Commit Graph

1646 Commits

Author SHA1 Message Date
Dyana Rose
5b22d96e04
fix(ingestion): looker - extract explore views from join name (#4627) 
Co-authored-by: Dyana Rose <dyanarose@gmail.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-04-12 08:20:10 -07:00
Xu Wang
7b1487135a
feat(ingest): add Urn python library for DataJob, DataFlow, Domain and Tag (#4618) 
* feat(ingest): add python library for DataJobUrn

* add DataFlowUrn lib and fix DataJobUrn

* fix create_from_str method

* fix lint error and unit test

* add DomainUrn and TagUrn

Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
 2022-04-12 09:02:28 +02:00
Marcin Szymański
e7c5eb357c
feat(ingest): add trino platform for great expectations (#4594) 2022-04-11 19:48:15 -07:00
jchen0824
524d183d93
feat: add presto-on-hive metadata ingestion source (#4625) 
* feat(metadata ingestion source): add presto-on-hive metadata ingestion source

Co-authored-by: Houren Chen <houren.chen@grabtaxi.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-11 17:46:44 -07:00
Aseem Bansal
61a95f41ae
chore: fix lint and remove incorrect integration mark from unit tests (#4621) 
* chore: fix lint and remove incorrect integration mark from unit tests

* add to test requirements

* revert athena source tests
 2022-04-08 17:18:48 +02:00
Aseem Bansal
336a628c5b
fix(bigquery): fix lineage bug, improve docs, add dataset filter config (#4607) 
* fix(bigquery): fix metadata from exported logs, doc missing permission, improve logging, add tests

Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-04-07 13:10:21 -07:00
David Haglund
0785ed6143
fix: urlencode slash in urns too (#4527) 
* fix: urlencode slash in urns too + tests

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-07 13:04:57 -07:00
Gabe Lyons
112589db32
feat(tableau): add some logic to normalize table names in tableau (#4609) 
* add some logic to normalize table names in tableau
 2022-04-07 12:15:41 -07:00
Ravindra Lanka
5e25cd1e22
feat(ingestion): Redshift Usage Source - simplify OperationalStats workunit generation. (#4585) 
* feat(ingestion): Redshift Usage Source - simplify OperationalStats workunit generation.
 2022-04-07 11:24:26 -07:00
mayurinehate
0a97fa22f9
fix(tableau): fix for incorrect schema returned by tableau api for snowflake connectionType (#4577) 2022-04-05 14:56:35 -07:00
Ravindra Lanka
fe5f24c2b3
fix(ingestion): Refactor redshift_usage source: simplify, annotate & fix bugs. (#4572) 2022-04-05 09:21:27 -07:00
David Haglund
df9e07fda2
fix: replace direct and indirect references to linkedin with datahub-project (#4557) 
* Update links for github-related links to use datahub-project:
  - https://github.com
  - https://img.shields.io/github/...
  - https://raw.githubusercontent.com/...
* Also replace references for github repo linkedin/datahub with
  datahub-project/datahub.
 2022-04-04 14:39:30 -05:00
Abhiram98
26742728a6
feat(ingestion): schema, table filtering for redshift-usage (#4396) 
* Filter based on table/schema pattern + documentation

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-01 20:48:23 -07:00
darapuk
a05d798939
(fix): Update path generated when creating LookML URL (#4554) 
* (fix): Update path generated when creating LookML URL
 2022-04-01 11:54:36 -07:00
Corentin
2fc3a48bc5
feat(ingest): indent sql queries for usage sources (#3782) 
* feat(ingest): indent sql queries for usage connectors.

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-140.eu-west-1.compute.internal>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-31 15:15:09 -07:00
mayurinehate
c09834d52b
fix(kafka-connect): add platform for default case in jdbc connector, update tests for platform instance map (#4545) 2022-03-31 08:13:09 -07:00
mayurinehate
467ea7917c
fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d… (#4375) 
* fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy dialects in jdbc source connnector

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 20:59:02 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability (#4130) 2022-03-30 18:50:26 -07:00
mohdsiddique
57002c766d
feat(stateful dbt): add stateful ingestion capability in dbt source (#4456) 
* feat(stateful dbt): add stateful ingestion capability in dbt source

Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 18:09:02 -07:00
mayurinehate
9ba36100ab
feat(tableau): emit lineage edge from embedded datasource to upstream… (#4470) 
* feat(tableau): emit lineage edge from embedded datasource to upstream published datasource

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 15:32:15 -07:00
Sunil Patil
36e9552d61
feat(ingestion): Support pluggable Schema Registry for Kafka Source (#4535) 
* Support for pluggable schema registry for the Kafka source.
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 13:20:23 -07:00
Kevin Hu
1bad3c7bc9
fix(ingest): mssql - support database_alias (#4523) 2022-03-29 20:47:43 -07:00
Tamas Nemeth
4358d8fb01
feat(ingest): athena - set Athena location as upstream (#4503) 2022-03-29 07:06:48 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source (#4490) 
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
 2022-03-29 11:52:57 +02:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently (#4496) 2022-03-27 18:49:26 -05:00
Xu Wang
d04092e634
feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn (#4469) 
* feat: add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn

Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-03-23 16:07:57 -07:00
Aseem Bansal
c5f1d2c9bd
feat(ingestion): snowflake, bigquery - enhancements to log and bugfix (#4442) 
feat(ingestion): add logging for snowflake, bigquery
 2022-03-21 09:50:36 -07:00
cuong-pham
12bb2e1231
getting database directly from upstream tables incase there are multiple databases in upstreamDatabases (#4447) 2022-03-18 14:11:07 -07:00
Ravindra Lanka
60925e3e8c
Fix bug in the SchemaField type computation for AVRO logical types. (#4433) 2022-03-18 12:06:54 +01:00
mayurinehate
2f078c981c
feat(ingestion): tableau - support for tableau version 2021.1 and older (#4437) 
fixes #4119
 2022-03-17 14:07:36 -07:00
Tamas Nemeth
f557b2c1b3
fix(ingestion) containers: Adding platform instance to container keys (#4279) 2022-03-16 14:57:50 -07:00
Jorgen Evens
af5c4ee4d0
fix(ingest): handle endpoints without 200 response in openapi (#4332) 2022-03-14 17:52:08 -07:00
cccs-eric
cb9b99f0ba
fix(ingest) Azure AD: support nested groups (#4367) (#4368) 
LGTM - Thanks!
 2022-03-14 08:59:04 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift (#4344) 2022-03-14 08:50:29 -07:00
mayurinehate
3ea72869f3
feat(GE): add option to disable sql parsing, use default parser (#4377) 2022-03-10 17:36:59 -08:00
Vishal Shah
733413f58e
feat(ingest): mysql - add database_alias functionality (#4319) 
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-03-09 09:29:58 -08:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage (#4111) 2022-03-08 12:29:10 -08:00
MugdhaHardikar-GSLab
f198a92def
fix(config-parsing): add support for variable expansion for in variables in between string (#4350) 2022-03-08 12:24:08 -08:00
Aseem Bansal
7eec30b2ec
fix(hive): clean protocol for hive source (#4330) 2022-03-08 11:57:26 -08:00
Swaroop Jagadish
35b187a8d4
feat(ingest): transformers - add support for processing MCP-s (#4337) 2022-03-07 13:14:29 -08:00
mayurinehate
92b0e1c7c7
feat(tableau): emit workbook as container entity in tableau source, some minor fixes in tableau source (#4261) 2022-03-04 11:52:04 -08:00
John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA (#4305) 2022-03-04 11:51:31 -08:00
Aseem Bansal
9a9a5c3084
fix: revert changes to OpenApi casing (#4291) 2022-03-04 10:35:44 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source (#4201) 2022-02-28 17:37:22 -08:00
Vincenzo Lavorini
a113e4357e
fix(ingest): openapi - add support for user, password auth (#4086) 2022-02-24 23:29:01 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling (#4131) 2022-02-24 23:26:06 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file (#4116) 2022-02-24 17:02:38 -08:00
David Sanchez
8f75c278a1
fix(ingest): tableau - fix sheets external url ingestion (#4231) 2022-02-24 15:44:31 -08:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 (#4205) 2022-02-22 22:13:50 -08:00
Xu Wang
aa3363bcc2
feat(ingest): lib - add better support for working with urns (#4172) 
Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
 2022-02-22 19:39:24 -08:00