yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-07-16 05:34:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,004 Commits 200 Branches 118 Tags
Commit Graph

935 Commits

Author SHA1 Message Date
cccs-eric
f25a5f043d
fix(iceberg): Change how MapType are mapped to Avro to support complex Map key type. (#5060) 2022-06-09 08:13:10 +05:30
John Joyce
5c501fba2c
fix(cli): Bumping avro-gen3 to 0.7.4 (#5098) 2022-06-07 12:56:06 -07:00
Lal Rishav
4f82e29697
feat(ingest): snowflake using oauth for Azure (#4647) 2022-06-06 17:49:49 +05:30
Ravindra Lanka
96f923e007
fix(ingest): common - fix nullability determination for the AVRO fixed type. (#5023) 2022-05-29 09:06:08 -07:00
Aseem Bansal
d926fa26f5
fix(parsing): incorrect parsing for commas (#5027) 2022-05-27 22:22:13 +05:30
Aseem Bansal
64bb918939
fix(parsing): improve sql parsing, some debugging redash (#5025) 2022-05-27 19:28:45 +05:30
Ravindra Lanka
26c0acf786
fix(build): fix for hana build failure for aarch64. (#5019) 2022-05-26 15:29:21 -07:00
Harshal Sheth
480fb2e770
test(ingestion): change class names to avoid unittest warnings (#5005) 2022-05-26 22:09:40 +05:30
Ravindra Lanka
108b492ed1
feat(ingestion): Add Iceberg source (#5010) 
Co-authored-by: cccs-eric <eric.ladouceur@cyber.gc.ca>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
 2022-05-26 08:05:57 -07:00
buggythepirate
92338c7912
feat(ingest): Added new ingestion source SAP HANA (#4376) 
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-05-26 03:42:50 -07:00
Ebu (えぶ)
2911e1ed1b
feat(ingest): Add Source from Vertica (#4555) 
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-05-26 03:26:28 -07:00
Claudio Benfatto
bbd0ab823d
feat(ingestion): optionally disable some kafka schema warnings (#4169) 
Co-authored-by: Claudio Benfatto <claudio.benfatto@adevinta.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-05-24 14:27:02 -07:00
Gabe Lyons
a8079ca163
feat(transformers): add transformers to provide tags & terms to schema fields based on regex patterns (#4936) 
* add tag & term transformers for schemas

* added documentation

* lint fixes

* add clarification that only first set of matching terms is applied
 2022-05-18 16:03:34 -07:00
BZ
367fac6066
feat(ingestion): For all usage connectors, allow exclusion of top_n_queries from ingestion via a config param. (#4839) 
* feat(redshift-usage): allow users to not ingest top_n_queries

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-05-12 14:26:03 -07:00
Sebo Kim
f3df15d6dc
fix(ingestion): ElasticSearch when no properties from elastic_mappings, gracefully continue (#4853) 
* when no properties from elastic_mappings, gracefully continue

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-05-11 09:07:18 -07:00
Ravindra Lanka
842fb391eb
feat(ingestion): kafka - add protobuf schema support (#4819) 
Co-authored-by: Luis Angel Vicente Sanchez <luis.vicentesanchez@aaqua.live>
 2022-05-04 17:07:01 -07:00
Aseem Bansal
3ff53b417b
fix(snowflake): passing connect args should not cause failures (#4764) 
* fix(snowflake): passing connect args should not cause failures

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-05-03 05:20:11 -07:00
Vladislavs Gaidass
8a24408cbf
fix(bigquery): improve handling of extracted audit log sql queries (#4735) 2022-05-03 14:43:23 +05:30
Ravindra Lanka
2b62ed5260
fix(ingest): avro - fix schema field type for avro logical types (#4801) 2022-05-02 17:43:42 -07:00
Shirshanka Das
a9ad138172
feat(ingest): docs - overhaul source connector docs to make it code driven (#4798) 
Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
 2022-05-02 00:18:15 -07:00
vanmeete
74d6d35881
feat(ingestion): add Pulsar source (#4721) 2022-04-29 15:57:02 +05:30
Jordan Wolinsky
bbac4a7a11
feat(ingestion): glue/s3 - Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job (#4689) 2022-04-29 10:09:06 +02:00
Shirshanka Das
d0eb772301
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … (#4786) 2022-04-28 22:31:19 -07:00
Sebo Kim
958b52f2f4
fix(ingest): bigquery - Fix BigQuery Datetime/Timestamp type column partition table profile bug (#4658) 
* fix BigQuery Datetime type column partition table profile bug

* inplace datetime replace

* extract out 'if' blocks and write a unit-test

* parse logic inside get_partition_range func
 2022-04-26 16:54:19 +02:00
Tamas Nemeth
474b0ba61e
feat(ingest): dbt - add query tag mapping and match template (#4744) 2022-04-25 10:56:45 -07:00
Shirshanka Das
a518e3d13e
feat(cli): improve error reporting, make sink config optional (#4718) 2022-04-24 17:12:21 -07:00
Aseem Bansal
c66ef7c1fe
fix(snowflake): deprecate config, update examples (#4644) 
* fix(snowflake): deprecate config, update examples

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-20 15:21:09 -07:00
Aseem Bansal
4b7f407e26
fix(bigquery): error due to not handling date properly (#4702) 2022-04-20 18:14:33 +02:00
Aseem Bansal
73d69510f8
fix(sqlparser): fix sqlparser breaking due to # sign (#4662) 
* fix(sqlparser): fix sqlparser breaking due to # sign

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-13 17:15:38 -07:00
Tamas Nemeth
f99d27fd8c
feat(ingest): airflow - add support to capture airflow executions, add high level dataflow jobs api to python sdk (#4615) 
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
 2022-04-12 23:19:39 -07:00
Xu Wang
7b1487135a
feat(ingest): add Urn python library for DataJob, DataFlow, Domain and Tag (#4618) 
* feat(ingest): add python library for DataJobUrn

* add DataFlowUrn lib and fix DataJobUrn

* fix create_from_str method

* fix lint error and unit test

* add DomainUrn and TagUrn

Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
 2022-04-12 09:02:28 +02:00
Marcin Szymański
e7c5eb357c
feat(ingest): add trino platform for great expectations (#4594) 2022-04-11 19:48:15 -07:00
Aseem Bansal
61a95f41ae
chore: fix lint and remove incorrect integration mark from unit tests (#4621) 
* chore: fix lint and remove incorrect integration mark from unit tests

* add to test requirements

* revert athena source tests
 2022-04-08 17:18:48 +02:00
Aseem Bansal
336a628c5b
fix(bigquery): fix lineage bug, improve docs, add dataset filter config (#4607) 
* fix(bigquery): fix metadata from exported logs, doc missing permission, improve logging, add tests

Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-04-07 13:10:21 -07:00
David Haglund
0785ed6143
fix: urlencode slash in urns too (#4527) 
* fix: urlencode slash in urns too + tests

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-04-07 13:04:57 -07:00
Corentin
2fc3a48bc5
feat(ingest): indent sql queries for usage sources (#3782) 
* feat(ingest): indent sql queries for usage connectors.

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-140.eu-west-1.compute.internal>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-31 15:15:09 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability (#4130) 2022-03-30 18:50:26 -07:00
Sunil Patil
36e9552d61
feat(ingestion): Support pluggable Schema Registry for Kafka Source (#4535) 
* Support for pluggable schema registry for the Kafka source.
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
 2022-03-30 13:20:23 -07:00
Tamas Nemeth
4358d8fb01
feat(ingest): athena - set Athena location as upstream (#4503) 2022-03-29 07:06:48 -07:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently (#4496) 2022-03-27 18:49:26 -05:00
Xu Wang
d04092e634
feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn (#4469) 
* feat: add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn

Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
 2022-03-23 16:07:57 -07:00
Aseem Bansal
c5f1d2c9bd
feat(ingestion): snowflake, bigquery - enhancements to log and bugfix (#4442) 
feat(ingestion): add logging for snowflake, bigquery
 2022-03-21 09:50:36 -07:00
Ravindra Lanka
60925e3e8c
Fix bug in the SchemaField type computation for AVRO logical types. (#4433) 2022-03-18 12:06:54 +01:00
Tamas Nemeth
f557b2c1b3
fix(ingestion) containers: Adding platform instance to container keys (#4279) 2022-03-16 14:57:50 -07:00
Jorgen Evens
af5c4ee4d0
fix(ingest): handle endpoints without 200 response in openapi (#4332) 2022-03-14 17:52:08 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift (#4344) 2022-03-14 08:50:29 -07:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage (#4111) 2022-03-08 12:29:10 -08:00
MugdhaHardikar-GSLab
f198a92def
fix(config-parsing): add support for variable expansion for in variables in between string (#4350) 2022-03-08 12:24:08 -08:00
Aseem Bansal
7eec30b2ec
fix(hive): clean protocol for hive source (#4330) 2022-03-08 11:57:26 -08:00
Swaroop Jagadish
35b187a8d4
feat(ingest): transformers - add support for processing MCP-s (#4337) 2022-03-07 13:14:29 -08:00