3975 Commits

Author SHA1 Message Date
Aseem Bansal
87ac8a32aa
doc(ingest): mysql - describe required grants (#4958) 2022-05-20 15:52:44 -07:00
Aseem Bansal
2260118232
feat(bigquery): reduce logging (#4961)
* feat(bigquery): reduce logging

* doc: add entry for behaviour change
2022-05-20 09:42:55 -07:00
Mert Tunç
b38cab0f7c
docs(datahub-kafka-sink): add topic_routes config to doc of datahub-kafka-sink (#4965) 2022-05-19 20:55:10 -07:00
mayurinehate
ec356d3aab
feat(cli): raise error if get entity api fails (#4922) 2022-05-19 20:50:59 -07:00
Aseem Bansal
2bb2c5243c
fix(bigquery): add dataset_id for bigquery (#4932) 2022-05-19 10:43:06 +05:30
Gabe Lyons
a8079ca163
feat(transformers): add transformers to provide tags & terms to schema fields based on regex patterns (#4936)
* add tag & term transformers for schemas

* added documentation

* lint fixes

* add clarification that only first set of matching terms is applied
2022-05-18 16:03:34 -07:00
Gabe Lyons
7b1cf6f8b2
feat(dbt): enable data platform instance on dbt (#4926) 2022-05-17 16:53:16 -07:00
Aseem Bansal
9a97af1350
fix(lint): lint failure due to mypy upgrade (#4933)
* fix(lint): lint failure due to mypy upgrade
2022-05-17 21:49:41 +02:00
Aseem Bansal
1a8e3f5ae7
docs(ingest): remove incorrectly annotated lineage capability (#4914) 2022-05-13 09:11:30 -07:00
mayurinehate
d316fc47ff
docs(transformer): update custom transform example to add missing super init (#4912) 2022-05-13 12:12:32 +05:30
Aseem Bansal
15438f62f1
fix(doc): update doc url to generated docs (#4860) 2022-05-13 10:19:46 +05:30
Pedro Silva
78c3ca039e
feat(gms): Add support for deleting reference pointers when deleting by urn (#4791) 2022-05-12 19:02:13 -07:00
BZ
367fac6066
feat(ingestion): For all usage connectors, allow exclusion of top_n_queries from ingestion via a config param. (#4839)
* feat(redshift-usage): allow users to not ingest top_n_queries

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-12 14:26:03 -07:00
Aseem Bansal
39fcdd2855
doc(biqquery): add caveat for materialized view (#4859) 2022-05-12 12:48:59 -07:00
Aseem Bansal
5ac6ba58f8
fix(build): Fix breaking changes from GE 0.15.3 (#4905) 2022-05-12 08:46:23 -07:00
Aseem Bansal
b9f78026c8
revert(bigquery-usage): dataset allow filter impl (#4901)
* Revert "fix(ingestion): bigquery-usage: Fix biquery usage table deny pattern template (#4898)"
2022-05-11 20:03:03 +02:00
Sagar Tiwari
830f2b0d6f
fix(docs): Update developing.md to mention directory context (#4899) 2022-05-11 10:00:37 -07:00
Sebo Kim
f3df15d6dc
fix(ingestion): ElasticSearch when no properties from elastic_mappings, gracefully continue (#4853)
* when no properties from elastic_mappings, gracefully continue

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-11 09:07:18 -07:00
Tamas Nemeth
6627e6b70f
fix(ingestion): bigquery-usage: Fix biquery usage table deny pattern template (#4898) 2022-05-11 13:59:54 +02:00
Shirshanka Das
8d281fc013
fix(ingest): lookml - add view definitions for all views (#4875) 2022-05-10 10:48:36 -07:00
Aseem Bansal
d0cdadbb3f
fix(sql-parsing): improve error handling (#4862) 2022-05-10 12:18:54 +02:00
Shirshanka Das
f8171702f6
feat(ci): docker actions simplify, add vulnerability scanner, simplify smoke-tests (#4881)
co-authored-by: Dexter Lee <dexter@acryl.io>
2022-05-09 19:49:57 -07:00
Shirshanka Das
8b235b1706
fix(docs): ingest - sort modules, fix small typos (#4880) 2022-05-09 13:01:44 -07:00
Tamas Nemeth
84f71dadf0
fix(ingestion): dependencies - Downgrading typing-extension dependency to work with Airflow 2.0.2 (#4855)
* Downgrading typing-extension dependency to work with Airflow 2.0.2 restricting typing-extension on python 3.7
2022-05-09 09:43:49 -07:00
Zach Bluhm
6ced69cf31
fix(bigquery-usage): dataset allow filter impl (#4776)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-06 13:45:37 -07:00
Cristian Calugaru
2a19a85f97
ssl configuration support for elasticsearch source (#4843) 2022-05-06 11:22:33 -07:00
Ravindra Lanka
5c64e9d541
fix(ingestion): Allow profiling of only those tables that are allowed by the table_pattern. (#4842) 2022-05-06 11:07:31 +02:00
Tamas Nemeth
56ee4d9651
feat(ingest): s3 - add support for multiple pathspecs in one recipe (#4777) 2022-05-05 10:09:47 -07:00
mayurinehate
b2c82dcaa3
feat(ingest): great-expectations - add more logs (#4832) 2022-05-05 09:25:58 -07:00
Aseem Bansal
bddfd89002
fix(ingestion): bigquery - extract temp table prefix as config, fix reporting, logging (#4766) 2022-05-05 10:02:10 +02:00
mayurinehate
d3fb6ce026
fix(ingest): great-expectations - fix failure to serialize type Decimal (#4763) 2022-05-04 22:56:08 -07:00
Ravindra Lanka
842fb391eb
feat(ingestion): kafka - add protobuf schema support (#4819)
Co-authored-by: Luis Angel Vicente Sanchez <luis.vicentesanchez@aaqua.live>
2022-05-04 17:07:01 -07:00
Tamas Nemeth
075d19ef16
fix(doc): improving docs across multiple sources (#4815) 2022-05-04 12:57:26 -07:00
Shirshanka Das
d152ca367c
fix(docgen): fix failure count incrementing during doc generation (#4806) 2022-05-03 22:56:35 +02:00
Aseem Bansal
3ff53b417b
fix(snowflake): passing connect args should not cause failures (#4764)
* fix(snowflake): passing connect args should not cause failures

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-05-03 05:20:11 -07:00
Aseem Bansal
6828dc3d4c
fix(snowflake): get external tables when there is default namespace (#4803) 2022-05-03 15:35:04 +05:30
Vladislavs Gaidass
8a24408cbf
fix(bigquery): improve handling of extracted audit log sql queries (#4735) 2022-05-03 14:43:23 +05:30
Ravindra Lanka
df75eafcfc
fix(ci): fix presto_on_hive tests. (#4802) 2022-05-02 21:09:33 -07:00
mitchelllovessoftware123
27ed4508ef
Create sample_pii_glossary.yml (#4795) 2022-05-02 21:09:01 -07:00
Ravindra Lanka
2b62ed5260
fix(ingest): avro - fix schema field type for avro logical types (#4801) 2022-05-02 17:43:42 -07:00
Aditya Radhakrishnan
c20a47f34c
feat(operation): display the reported time for last updated in the UI (#4800) 2022-05-02 16:00:29 -07:00
Shirshanka Das
a9ad138172
feat(ingest): docs - overhaul source connector docs to make it code driven (#4798)
Co-authored-by: MugdhaHardikar-GSLab <mugdha.hardikar@gslab.com>
2022-05-02 00:18:15 -07:00
Shirshanka Das
c15b44a1de
fix(ingest): databricks - hive ingestion should not fail on table comment (#4787) 2022-04-29 13:56:42 -07:00
mayurinehate
c34a1ba735
fix(s3): improved handling for corner cases (#4774) 2022-04-29 12:25:41 -07:00
vanmeete
74d6d35881
feat(ingestion): add Pulsar source (#4721) 2022-04-29 15:57:02 +05:30
Aseem Bansal
f099aeb550
fix(m1): tweak m1 preflight (#4771) 2022-04-29 10:29:47 +02:00
Aseem Bansal
ee6413f0ab
feat(snowflake): add option to disable checking role grants (#4760)
* feat(ingest): add option to disable checking role grants
2022-04-29 10:14:05 +02:00
Aseem Bansal
37c1fce964
fix(snowflake): improve debug log for external tables (#4772) 2022-04-29 10:10:46 +02:00
Jordan Wolinsky
bbac4a7a11
feat(ingestion): glue/s3 - Ingest Tags from s3 bucket on an AWS Glue job and S3 Data Lake Ingest Job (#4689) 2022-04-29 10:09:06 +02:00
Shirshanka Das
d0eb772301
fix(ingest): fwk - datahub_api should be initialized by datahub-rest … (#4786) 2022-04-28 22:31:19 -07:00