Harshal Sheth
355c129c7c
chore(ingest): drop python 3.6 support ( #5521 )
2022-08-10 15:00:31 -07:00
liyuhui666
0481075705
fix(ingest): Fix ingest Clickhouse without password ( #5511 )
...
* fix(ingest): Fix ingest Clickhouse without password
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-08-09 10:30:56 -07:00
Jordan Wolinsky
33339e2c89
Expose catalog_name in athena.py ( #5548 )
...
* expose catalog_name to the sql alchemy uri that is passed into pyathena
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-08 11:40:05 -07:00
Harshal Sheth
9790f3cefa
feat(ingest): infer aspectName from aspect type in MCP ( #5566 )
2022-08-07 07:52:58 -07:00
Piotr Sierkin
828a711684
feat(ingest): dbt - control over emitting test_results, test_definitions, etc. ( #5328 )
...
Co-authored-by: Piotr Sierkin <piotr.sierkin@getindata.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-08-06 21:42:53 -07:00
Harshal Sheth
64e7da8a68
fix(ingest): use temp dir for file generated during test ( #5505 )
2022-07-27 14:29:11 -07:00
Mayuri Nehate
04de6c27b7
feat(ingest): snowflake - test_connection add support for capability report ( #5472 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-07-26 09:29:57 -07:00
Shirshanka Das
7ed9cd2838
feat(ingest): snowflake - basic test connection capability ( #5464 )
2022-07-22 09:14:37 +02:00
Shirshanka Das
14d764a26f
fix(ingest): fix serialization of report to handle nesting ( #5455 )
2022-07-20 18:25:07 -07:00
Aseem Bansal
acb9879eb4
feat(cli,build): remove deprecated variables GMS_HOST/_PORT ( #5451 )
2022-07-20 20:54:43 +05:30
Mugdha Hardikar
ced6c38239
fix(ingest): bigquery-usage - fix dataset name for sharded table ( #5412 )
2022-07-19 20:59:02 -07:00
Mugdha Hardikar
a6dc669891
docs(bigquery): add changelog and unittest for profiling limits ( #5407 )
2022-07-19 09:39:09 +05:30
Pedro Silva
b2edd44b6a
Adds support for Domains in CSV source ( #5372 )
2022-07-15 14:20:41 +05:30
Felix Lüdin
a0303448ba
feat(dashboards): add datasets field to DashboardInfo aspect ( #5188 )
...
Co-authored-by: John Joyce <john@acryl.io>
2022-07-14 09:54:02 -07:00
Mugdha Hardikar
94dd3ad5a1
fix(ingest): bigquery-usage - dataset name for sharded tables ( #5347 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-07-07 00:37:13 -07:00
Aditya Radhakrishnan
fc8e59387d
feat(ingest): update CSV source to support description and ownership type ( #5346 )
2022-07-06 21:29:29 +05:30
Shirshanka Das
4b3135a0f8
feat(ingest): dbt - improving dbt_meta mapping ( #5237 )
2022-06-24 13:43:12 +02:00
Aditya Radhakrishnan
82ca92f8f9
feat(ingest): adds csv enricher ingestion source ( #5221 )
2022-06-22 12:25:39 +05:30
Tamas Nemeth
393c07ee52
refactor(ingest): bigquery-usage - Adding tests for bigquery usage filters ( #5195 )
2022-06-20 18:28:27 -07:00
Aseem Bansal
d518b5a085
fix(cli): correct handling of env variables ( #5203 )
2022-06-20 20:53:47 +05:30
cccs-eric
f25a5f043d
fix(iceberg): Change how MapType are mapped to Avro to support complex Map key type. ( #5060 )
2022-06-09 08:13:10 +05:30
John Joyce
5c501fba2c
fix(cli): Bumping avro-gen3 to 0.7.4 ( #5098 )
2022-06-07 12:56:06 -07:00
Lal Rishav
4f82e29697
feat(ingest): snowflake using oauth for Azure ( #4647 )
2022-06-06 17:49:49 +05:30
Ravindra Lanka
96f923e007
fix(ingest): common - fix nullability determination for the AVRO fixed type. ( #5023 )
2022-05-29 09:06:08 -07:00
Aseem Bansal
d926fa26f5
fix(parsing): incorrect parsing for commas ( #5027 )
2022-05-27 22:22:13 +05:30
Aseem Bansal
64bb918939
fix(parsing): improve sql parsing, some debugging redash ( #5025 )
2022-05-27 19:28:45 +05:30
Ravindra Lanka
26c0acf786
fix(build): fix for hana build failure for aarch64. ( #5019 )
2022-05-26 15:29:21 -07:00
Harshal Sheth
480fb2e770
test(ingestion): change class names to avoid unittest warnings ( #5005 )
2022-05-26 22:09:40 +05:30
Ravindra Lanka
108b492ed1
feat(ingestion): Add Iceberg source ( #5010 )
...
Co-authored-by: cccs-eric <eric.ladouceur@cyber.gc.ca>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-05-26 08:05:57 -07:00
buggythepirate
92338c7912
feat(ingest): Added new ingestion source SAP HANA ( #4376 )
...
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-26 03:42:50 -07:00
Ebu (えぶ)
2911e1ed1b
feat(ingest): Add Source from Vertica ( #4555 )
...
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-26 03:26:28 -07:00
Claudio Benfatto
bbd0ab823d
feat(ingestion): optionally disable some kafka schema warnings ( #4169 )
...
Co-authored-by: Claudio Benfatto <claudio.benfatto@adevinta.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-24 14:27:02 -07:00
Gabe Lyons
a8079ca163
feat(transformers): add transformers to provide tags & terms to schema fields based on regex patterns ( #4936 )
...
* add tag & term transformers for schemas
* added documentation
* lint fixes
* add clarification that only first set of matching terms is applied
2022-05-18 16:03:34 -07:00
BZ
367fac6066
feat(ingestion): For all usage connectors, allow exclusion of top_n_queries from ingestion via a config param. ( #4839 )
...
* feat(redshift-usage): allow users to not ingest top_n_queries
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-12 14:26:03 -07:00
Sebo Kim
f3df15d6dc
fix(ingestion): ElasticSearch when no properties from elastic_mappings, gracefully continue ( #4853 )
...
* when no properties from elastic_mappings, gracefully continue
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-11 09:07:18 -07:00
Ravindra Lanka
842fb391eb
feat(ingestion): kafka - add protobuf schema support ( #4819 )
...
Co-authored-by: Luis Angel Vicente Sanchez <luis.vicentesanchez@aaqua.live>
2022-05-04 17:07:01 -07:00
Aseem Bansal
3ff53b417b
fix(snowflake): passing connect args should not cause failures ( #4764 )
...
* fix(snowflake): passing connect args should not cause failures
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-03 05:20:11 -07:00
Vladislavs Gaidass
8a24408cbf
fix(bigquery): improve handling of extracted audit log sql queries ( #4735 )
2022-05-03 14:43:23 +05:30
Ravindra Lanka
2b62ed5260
fix(ingest): avro - fix schema field type for avro logical types ( #4801 )
2022-05-02 17:43:42 -07:00
Shirshanka Das
a9ad138172
feat(ingest): docs - overhaul source connector docs to make it code driven ( #4798 )
...
Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
2022-05-02 00:18:15 -07:00
vanmeete
74d6d35881
feat(ingestion): add Pulsar source ( #4721 )
2022-04-29 15:57:02 +05:30
Jordan Wolinsky
bbac4a7a11
feat(ingestion): glue/s3 - Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job ( #4689 )
2022-04-29 10:09:06 +02:00
Shirshanka Das
d0eb772301
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … ( #4786 )
2022-04-28 22:31:19 -07:00
Sebo Kim
958b52f2f4
fix(ingest): bigquery - Fix BigQuery Datetime/Timestamp type column partition table profile bug ( #4658 )
...
* fix BigQuery Datetime type column partition table profile bug
* inplace datetime replace
* extract out 'if' blocks and write a unit-test
* parse logic inside get_partition_range func
2022-04-26 16:54:19 +02:00
Tamas Nemeth
474b0ba61e
feat(ingest): dbt - add query tag mapping and match template ( #4744 )
2022-04-25 10:56:45 -07:00
Shirshanka Das
a518e3d13e
feat(cli): improve error reporting, make sink config optional ( #4718 )
2022-04-24 17:12:21 -07:00
Aseem Bansal
c66ef7c1fe
fix(snowflake): deprecate config, update examples ( #4644 )
...
* fix(snowflake): deprecate config, update examples
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-20 15:21:09 -07:00
Aseem Bansal
4b7f407e26
fix(bigquery): error due to not handling date properly ( #4702 )
2022-04-20 18:14:33 +02:00
Aseem Bansal
73d69510f8
fix(sqlparser): fix sqlparser breaking due to # sign ( #4662 )
...
* fix(sqlparser): fix sqlparser breaking due to # sign
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-13 17:15:38 -07:00
Tamas Nemeth
f99d27fd8c
feat(ingest): airflow - add support to capture airflow executions, add high level dataflow jobs api to python sdk ( #4615 )
...
Co-authored-by: Gabe Lyons <itsgabelyons@gmail.com>
2022-04-12 23:19:39 -07:00