Aditya Radhakrishnan
aeafa7e63f
feat(okta) - add support for filtering/searching when ingesting Okta groups and users ( #4586 )
2022-04-05 16:15:34 -07:00
mayurinehate
0a97fa22f9
fix(tableau): fix for incorrect schema returned by tableau api for snowflake connectionType ( #4577 )
2022-04-05 14:56:35 -07:00
Aseem Bansal
809d1beae9
feat(snowflake): reduce permissions provisioned by default ( #4543 )
...
* feat(snowflake): reduce permissions provisioned by default
Co-authored-by: John Joyce <john@acryl.io>
2022-04-05 09:03:00 -07:00
Kevin Hu
030d25f0a1
feat(ingest): add option for external Spark cluster ( #4571 )
...
* Add option for configuring spark cluster manager
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-04-04 15:56:50 -07:00
David Haglund
df9e07fda2
fix: replace direct and indirect references to linkedin with datahub-project ( #4557 )
...
* Update links for github-related links to use datahub-project:
- https://github.com
- https://img.shields.io/github/ ...
- https://raw.githubusercontent.com/ ...
* Also replace references for github repo linkedin/datahub with
datahub-project/datahub.
2022-04-04 14:39:30 -05:00
mayurinehate
58e4364354
fix(tableau): gracefully stop ingestion if tableau sign in not successful ( #4548 )
...
* fix(tableau): gracefully stop ingestion if tableau sign in not successful
* Update tableau.md
* Update tableau.md
* docs(tableau): update doc, add caveats, use env variables in credentials
Co-authored-by: John Joyce <john@acryl.io>
2022-04-04 13:15:08 +02:00
Abhiram98
26742728a6
feat(ingestion): schema, table filtering for redshift-usage ( #4396 )
...
* Filter based on table/schema pattern + documentation
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-01 20:48:23 -07:00
Corentin
2fc3a48bc5
feat(ingest): indent sql queries for usage sources ( #3782 )
...
* feat(ingest): indent sql queries for usage connectors.
Co-authored-by: EC2 Default User <ec2-user@ip-172-31-22-140.eu-west-1.compute.internal>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-31 15:15:09 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability ( #4130 )
2022-03-30 18:50:26 -07:00
mayurinehate
9ba36100ab
feat(tableau): emit lineage edge from embedded datasource to upstream… ( #4470 )
...
* feat(tableau): emit lineage edge from embedded datasource to upstream published datasource
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 15:32:15 -07:00
Sunil Patil
36e9552d61
feat(ingestion): Support pluggable Schema Registry for Kafka Source ( #4535 )
...
* Support for pluggable schema registry for the Kafka source.
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 13:20:23 -07:00
Aseem Bansal
5d9c146bf7
doc: add example of profiling in default example ( #4532 )
2022-03-30 10:07:32 -07:00
Arun Vasudevan
c79c778270
feat(ingest): kafka-connect - support mapping for multiple DB instances ( #4501 )
2022-03-29 20:46:07 -07:00
Andres Lowrie
8564d45404
docs(metadata-ingestion): callout props in para ( #4485 )
2022-03-29 17:43:12 -07:00
Aseem Bansal
d30b6e1465
feat(ingest): Add config to improve user exp for initial ingestion and fix docs ( #4510 )
...
* feat(snowflake): change defaults to improve user experience for initial ingestion + documentation
2022-03-29 07:15:36 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source ( #4490 )
...
* feat(data-lake): add containers and folder level dataset support
* docs(data-lake): Update readme for data lake
* doc(data-lake): fix examples, update doc
* lint fix
* feat(s3): add s3 source, restore old data-lake source
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00
Aseem Bansal
6b04dff913
docs: add example of database and schema allow/deny patterns ( #4505 )
2022-03-28 13:06:40 +02:00
Aseem Bansal
9596e73706
doc: add caveats to snowflake doc ( #4467 )
2022-03-24 16:24:38 +01:00
Sergio Gómez Villamor
9fbb521bfe
chore: acryl-data 0.6.12 ( #4474 )
2022-03-23 10:24:48 -07:00
mayurinehate
885cf26828
docs(hive): update recipe with example to specify kerberos auth ( #4457 )
2022-03-22 13:38:21 +01:00
Kevin Neville
d8e6f890a9
fix: Replace old repository link with new link ( #4446 )
2022-03-18 14:12:19 -07:00
Gabe Lyons
431ba4b2a9
fix(ingestion): looker - various fixes ( #4394 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-15 15:48:34 -07:00
cccs-eric
cb9b99f0ba
fix(ingest) Azure AD: support nested groups ( #4367 ) ( #4368 )
...
LGTM - Thanks!
2022-03-14 08:59:04 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift ( #4344 )
2022-03-14 08:50:29 -07:00
Vishal Shah
733413f58e
feat(ingest): mysql - add database_alias functionality ( #4319 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-09 09:29:58 -08:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage ( #4111 )
2022-03-08 12:29:10 -08:00
Aseem Bansal
05f2507e16
fix(doc): remove duplicate entry for permission ( #4341 )
2022-03-08 09:32:32 -08:00
Aseem Bansal
beb51ebf59
fix(ingestion): add logging, make job more resilient to errors ( #4331 )
2022-03-07 14:32:44 -08:00
BZ
e2d05cd8eb
docs: postgres - update support for platform instance ( #4292 )
2022-03-07 13:16:39 -08:00
Salih Can
915798a5ad
fix(ingest): elasticsearch - connector should work with defaults for auth ( #4329 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-07 13:16:05 -08:00
mayurinehate
92b0e1c7c7
feat(tableau): emit workbook as container entity in tableau source, some minor fixes in tableau source ( #4261 )
2022-03-04 11:52:04 -08:00
Kevin Hu
b2b8826118
fix(ingest): clarify s3/s3a requirements and platform defaults ( #4263 )
2022-03-02 08:58:27 -08:00
Tamas Nemeth
2a5cf3dd07
feat(ingest): bigquery - ability to disable partition profiling ( #4228 )
2022-03-01 22:29:48 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source ( #4201 )
2022-02-28 17:37:22 -08:00
Vishal Shah
93ff09517b
feat(ingest): add lineage_client_project_id field to the BigQuery config ( #4138 )
...
* feat(ingest): add lineage_client_project_id field to the bigquery config
* fix linting issues
* add type annotation for arguments
2022-02-28 11:19:23 -08:00
Vincenzo Lavorini
a113e4357e
fix(ingest): openapi - add support for user, password auth ( #4086 )
2022-02-24 23:29:01 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling ( #4131 )
2022-02-24 23:26:06 -08:00
Sunil Patil
16f3d4683a
feat(ingest): elasticsearch - add support for url_prefix in configuration ( #4214 )
...
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-24 17:06:38 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file ( #4116 )
2022-02-24 17:02:38 -08:00
Ravindra Lanka
84005d3848
feat(ingest): kafka - add support for non-default schema registry subject name strategies ( #4215 )
2022-02-22 16:05:46 -08:00
Jie Qiu
c372b93804
Fix config typo in stateful ingestion README ( #4202 )
2022-02-22 15:20:53 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support ( #4057 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
cuong-pham
ede6d91534
Update the doc to including options to include Views ( #4164 )
2022-02-16 19:42:09 -08:00
Ravindra Lanka
51d72c6a29
feat(ingestion): Add support for snowflake view lineage. ( #4163 )
...
* Add support for snowflake view lineage.
* Add a config flag. Sepearate upstream & downstream view lineage computation. Update documentation.
2022-02-16 18:27:40 -08:00
Tamas Nemeth
b2664916e3
feat(ingest): Glue - Support for domains and containers ( #4110 )
...
* Add container and domain support for Glue.
Adding option to set aws profile for Glue.
* Adding domain doc for Glue
* Making get_workunits less complex
* Updating golden file
* Addressing pr review comments
* Remove unneded empty line
2022-02-16 08:29:14 -08:00
Aseem Bansal
d33a868f19
fix(docs): fix example of delta lake ( #4149 )
2022-02-15 14:44:37 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets ( #4123 )
...
* Getting table properties for Athena datasets
* Isorting
* Fixing mypy error
* Addressing pr review comments
Adding tests
* Adding missing import
* black
* Fixing test run
* fixing flake8
* Adding athen to tox tests as well
* Not running athena tests on python < 3.7
* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Kevin Hu
9bdc9af7b9
fix(ingest): postgres - ignore information_schema tables by default ( #4069 )
2022-02-09 23:20:25 -08:00
Aseem Bansal
dbcfe9e50e
docs(kafka): add example for using domains, change for clarity ( #4100 )
2022-02-09 08:56:27 -08:00