Aseem Bansal
beb51ebf59
fix(ingestion): add logging, make job more resilient to errors ( #4331 )
2022-03-07 14:32:44 -08:00
BZ
e2d05cd8eb
docs: postgres - update support for platform instance ( #4292 )
2022-03-07 13:16:39 -08:00
Salih Can
915798a5ad
fix(ingest): elasticsearch - connector should work with defaults for auth ( #4329 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-07 13:16:05 -08:00
mayurinehate
92b0e1c7c7
feat(tableau): emit workbook as container entity in tableau source, some minor fixes in tableau source ( #4261 )
2022-03-04 11:52:04 -08:00
Kevin Hu
b2b8826118
fix(ingest): clarify s3/s3a requirements and platform defaults ( #4263 )
2022-03-02 08:58:27 -08:00
Tamas Nemeth
2a5cf3dd07
feat(ingest): bigquery - ability to disable partition profiling ( #4228 )
2022-03-01 22:29:48 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source ( #4201 )
2022-02-28 17:37:22 -08:00
Vishal Shah
93ff09517b
feat(ingest): add lineage_client_project_id field to the BigQuery config ( #4138 )
...
* feat(ingest): add lineage_client_project_id field to the bigquery config
* fix linting issues
* add type annotation for arguments
2022-02-28 11:19:23 -08:00
Vincenzo Lavorini
a113e4357e
fix(ingest): openapi - add support for user, password auth ( #4086 )
2022-02-24 23:29:01 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling ( #4131 )
2022-02-24 23:26:06 -08:00
Sunil Patil
16f3d4683a
feat(ingest): elasticsearch - add support for url_prefix in configuration ( #4214 )
...
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-24 17:06:38 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file ( #4116 )
2022-02-24 17:02:38 -08:00
Ravindra Lanka
84005d3848
feat(ingest): kafka - add support for non-default schema registry subject name strategies ( #4215 )
2022-02-22 16:05:46 -08:00
Jie Qiu
c372b93804
Fix config typo in stateful ingestion README ( #4202 )
2022-02-22 15:20:53 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support ( #4057 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
cuong-pham
ede6d91534
Update the doc to including options to include Views ( #4164 )
2022-02-16 19:42:09 -08:00
Ravindra Lanka
51d72c6a29
feat(ingestion): Add support for snowflake view lineage. ( #4163 )
...
* Add support for snowflake view lineage.
* Add a config flag. Sepearate upstream & downstream view lineage computation. Update documentation.
2022-02-16 18:27:40 -08:00
Tamas Nemeth
b2664916e3
feat(ingest): Glue - Support for domains and containers ( #4110 )
...
* Add container and domain support for Glue.
Adding option to set aws profile for Glue.
* Adding domain doc for Glue
* Making get_workunits less complex
* Updating golden file
* Addressing pr review comments
* Remove unneded empty line
2022-02-16 08:29:14 -08:00
Aseem Bansal
d33a868f19
fix(docs): fix example of delta lake ( #4149 )
2022-02-15 14:44:37 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets ( #4123 )
...
* Getting table properties for Athena datasets
* Isorting
* Fixing mypy error
* Addressing pr review comments
Adding tests
* Adding missing import
* black
* Fixing test run
* fixing flake8
* Adding athen to tox tests as well
* Not running athena tests on python < 3.7
* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Kevin Hu
9bdc9af7b9
fix(ingest): postgres - ignore information_schema tables by default ( #4069 )
2022-02-09 23:20:25 -08:00
Aseem Bansal
dbcfe9e50e
docs(kafka): add example for using domains, change for clarity ( #4100 )
2022-02-09 08:56:27 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] ( #4063 )
2022-02-08 14:26:44 -08:00
Tamas Nemeth
2de29dc623
feat(ingest) - bigquery: More verbose and faster lineage generation and option to set partition datetime for profiling ( #4079 )
2022-02-08 10:09:30 -08:00
Gabe Lyons
ec062b6787
feat(glue): make ownership configurable in glue source ( #4078 )
2022-02-07 14:29:51 -08:00
iasoon
782e66f5cf
docs(ingestion) glue: document required IAM permissions ( #3929 )
2022-02-07 10:10:35 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion ( #4051 )
2022-02-07 09:51:49 -08:00
Pedro Silva
ef617784b1
Mark data lake metadata source as Beta ( #4061 )
2022-02-04 16:27:44 -08:00
Ravindra Lanka
f20382f956
feat(ingest): framework - client side changes for monitoring and reporting ( #3807 )
2022-02-02 13:19:15 -08:00
Tamas Nemeth
928ab74f33
feat(ingest): bigquery - profile only the latest partition/shard ( #3930 )
2022-02-01 10:05:35 -08:00
Tamas Nemeth
68711222d4
feat(ingest): usage-stats - add ability to ignore users from top users calculation ( #3735 )
2022-02-01 00:11:23 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 ( #3888 )
2022-01-31 09:22:55 -08:00
Aseem Bansal
f32643148e
fix(docs): fix business glossary docs ( #3916 )
2022-01-30 17:25:11 -08:00
Michael A. Schlosser
c36662f837
feat(ingest): snowflake - support for additional auth mechanisms ( #4009 )
2022-01-30 11:47:53 -08:00
Swaroop Jagadish
9a9f448ebf
feat(ingest): platform-instance - changes to support platform instances in sources ( #3996 )
2022-01-27 15:31:25 -08:00
John Joyce
cf38b61a7f
feat(ingest): looker - optional external_base_url config for chart and dashboard urls ( #3985 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-26 15:06:34 -08:00
Kevin Hu
62f7d05557
feat(ingest): data-lake - support for Avro files ( #3913 )
2022-01-26 10:29:00 -08:00
iasoon
5dac30c93e
fix(ingestion): match default username for Azure OIDC and Azure ingestion source ( #3926 )
2022-01-25 18:57:18 -08:00
Ravindra Lanka
715a2e5f8e
feat(ingest): snowflake, glue - add support for lineage to s3 ( #3946 )
2022-01-21 17:19:21 -08:00
Aseem Bansal
6c3441ccdc
fix(docs): fix openapi docs ( #3940 )
2022-01-21 07:55:50 -08:00
Kevin Hu
d4075b972c
docs(ingest): data lake - describe path-spec configuration ( #3905 )
2022-01-17 15:41:02 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets ( #3656 )
2022-01-17 15:01:23 -08:00
Jiwei
f804184884
docs(elasticsearch): fix typo in config example ( #3901 )
2022-01-17 14:56:17 -08:00
Ravindra Lanka
a44b48a6b8
feat(ingest): elasticsearch - add Elasticsearch Source ( #3893 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-01-14 13:10:12 -08:00
iasoon
285aa1b9ac
docs(ingest): azure-ad - specify required permission type ( #3886 )
2022-01-13 12:30:16 -08:00
Aseem Bansal
7498f4bf54
doc(kafka): add permissions required for confluent cloud ( #3850 )
2022-01-12 14:50:58 -08:00
iasoon
988baeb467
feat(ingest): metabase - allow configuring how database engines get mapped to platforms ( #3869 )
2022-01-12 00:52:55 -08:00
iasoon
130d5ff635
docs(ingest): azure-ad - correct default value for username attr ( #3861 )
2022-01-11 19:37:28 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources ( #3845 )
2022-01-07 17:27:09 -08:00