170 Commits

Author SHA1 Message Date
Harshal Sheth
ea2b092fe8
chore(ingest): remove unused groupby_unsorted utility (#4011) 2022-02-10 21:03:33 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy (#4092) 2022-02-08 23:21:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] (#4063) 2022-02-08 14:26:44 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion (#4051) 2022-02-07 09:51:49 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting (#3807) 2022-02-02 13:19:15 -08:00
Ravindra Lanka
f4209504f1
feat(ingest): support Kafka confluent external schema resolution by name or subject (#4035) 2022-02-02 07:44:56 -08:00
mayurinehate
1afe8876b7
feat(ingest): nifi - handle provenance api variation for older versions (#4022) 2022-02-01 10:03:05 -08:00
Tamas Nemeth
771c8567da
fix(ingest): snowflake - Run authentication validation if default value used (#4024) 2022-02-01 10:01:29 -08:00
Tamas Nemeth
68711222d4
feat(ingest): usage-stats - add ability to ignore users from top users calculation (#3735) 2022-02-01 00:11:23 -08:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms (#4009) 2022-01-30 11:47:53 -08:00
Aseem Bansal
400e0fe838
feat(ingest): kafka - support schema references (#3862) 2022-01-17 14:29:54 -08:00
Ravindra Lanka
1efe04f88a
feat(ingest): glue - support for nested structs (#3895) 2022-01-17 14:21:53 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) (#3900) 2022-01-16 12:03:38 -08:00
Ravindra Lanka
a44b48a6b8
feat(ingest): elasticsearch - add Elasticsearch Source (#3893)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 13:10:12 -08:00
Tamas Nemeth
e95446be1c
fix(ingest): sqlparser - Not lowercasing looker source's special table name (#3891)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 12:22:17 -08:00
Rickard Cardell
2f7e49b399
feat(ingest): datahub-rest - Make datahub-rest client more robust by configurable retries. (#3826) (#3860) 2022-01-11 19:35:06 -08:00
Swaroop Jagadish
480ad24282
feat(ingest): dbt - add support for dbt meta mapping (#3832) 2022-01-05 16:32:05 -08:00
Tamas Nemeth
e5d8313788
feat(ingest): sql parsing - Replacing sqlmetadata sql parser lib with sqlineage parser lib (#3806) 2022-01-05 12:22:59 -08:00
Tamas Nemeth
5df5150e51
feat(ingest): Add config option to set Bigquery credential in source config (#3786) 2021-12-27 05:48:45 -08:00
Sergio Gómez Villamor
3fa22e1138
feat(transform): adds simple add dataset properties transform (#3778) 2021-12-22 08:51:38 -08:00
Tamas Nemeth
2ee1a78f4e
fix(ingestion): Fix for same schema foreign key reference (#3769) 2021-12-20 07:11:55 -08:00
Harshal Sheth
22cef5f897
refactor(test): replace CliRunner with run_datahub_cmd method (#3746) 2021-12-16 20:07:38 -08:00
Harshal Sheth
adf9d2ead7
test(ingest): fix pytest warning for class starting with Test (#3745) 2021-12-14 22:44:42 -08:00
Tamas Nemeth
b9f67c5b65
feat(ingest): trim long sql queries in usage connector (#3725) 2021-12-13 09:16:24 -08:00
mayurinehate
bd4ecbc7b9
fix(nifi): add env in nifi config, add unit tests, fix nifi doc (#3703) 2021-12-09 13:34:13 -08:00
ecooklin
1a5121a5ae
feat(ingest): adds glossary terms transformer (#3657) 2021-12-07 21:54:15 -08:00
Adrian Garcia Badaracco
bd53f377ec
fix(tests): fix typo in test name (#3582)
* fix(tests): fix typo in test name

* find & replace function name
2021-11-28 21:09:42 -08:00
Swaroop Jagadish
6a00cfb4a1
feat(ingest): adding snowflake app name to connection string (#3574) 2021-11-15 13:08:07 -08:00
Vincenzo Lavorini
6cd31b5bdc
feat(ingest): adding openapi ingestion (#2706) 2021-11-09 21:22:15 -08:00
Noé López
cde1ce043d
feat(ingest): add transformer to add properties (#3480) 2021-11-09 20:03:21 -08:00
Ravindra Lanka
dcb8789f17
feat(ingestion): Support for converting AVRO schemas with logical types to MCE fields. (#3546) 2021-11-09 19:59:01 -08:00
Ravindra Lanka
c3f1bf0534
fix(ingestion): Make AVRO schema parsing robust to exceptions. (#3541) 2021-11-09 18:01:18 -08:00
Swaroop Jagadish
ec406c7928
feat(ingest): read-modify-write capabilities during ingestion, adding ownership patch transformer (#3506) 2021-11-03 21:39:52 -07:00
Kevin Hu
de41134a33
fix(ingestion): fix incorrect glue job names (#3503) 2021-11-02 22:54:47 -07:00
Harshal Sheth
dccef30daf
perf(ingest): lazy load ingestion plugins (#3430) 2021-10-25 14:10:24 -07:00
aseembansal-gogo
9dff1b7a1c
fix(ingest): allow database alias, remove extra removal from connect_… (#3352) 2021-10-19 22:29:38 -07:00
mayurinehate
f5d37c0621
feat(ingest): add complex types support in hive and trino source (#3375) 2021-10-18 22:53:51 -07:00
Harshal Sheth
6e5d1fe42f
fix(ingest): switch to avro from deprecated avro-python3 (#3412) 2021-10-18 15:35:27 -07:00
aseembansal-gogo
ebe0b3f7c1
feat(ingest): add mariadb as possible source (#3245) 2021-10-12 23:27:47 -07:00
Harshal Sheth
b43945ce55
test(airflow): remove unneeded execution_date parameter from test (#3368) 2021-10-12 23:25:27 -07:00
Swaroop Jagadish
7dab06519d
fix(ci): fix task instance constructor in airflow tests (#3366) 2021-10-11 20:58:37 -07:00
taufiqibrahim
ed38168f54
feat(ingest): add support for parsing lineage for redash connector (#3267)
Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
2021-10-05 23:08:02 -07:00
Sergio Gómez Villamor
696ad2ccc4
feat(ingest): adds replace_existing config to browse path transformer (#3313) 2021-10-01 12:35:16 -07:00
adriaanslechten
185f7e2f5f
feat(ingest): optional custom headers REST emitter (#3290)
Co-authored-by: Adriaan Slechten <adriaan.slechten@klarna.com>
2021-09-23 12:53:39 -07:00
Gabe Lyons
ff527f4bed
feat(foreign keys): add foreign key models (#3275) 2021-09-22 10:29:27 -07:00
aseembansal-gogo
97bed71e1c
feat(ingest): support for env variable in cli (#3215) 2021-09-16 23:18:42 -07:00
aseembansal-gogo
903348e1db
fix(ingest): add missing partition keys in schema for glue sources (#3238) 2021-09-15 21:39:49 -07:00
Sergio Gómez Villamor
9e28423c6d
feat(ingest): add optional config for ownership type in ownership transformers (#3159)
Co-authored-by: Sergio Gómez <sergio.gomez.villamor@adevinta.com>
2021-09-01 22:44:03 -07:00
Fredrik Sannholm
f73725fdf6
feat(ingestion): bring your own SQL parser (#3110) 2021-08-23 20:21:48 -07:00
taufiqibrahim
dd1b81b727
feat(ingest): adding redash source (#2918)
Co-authored-by: tibrahim <taufiq.ibrahim@bizzy.co.id>
2021-08-18 12:03:03 -07:00