mayurinehate
467ea7917c
fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d… ( #4375 )
...
* fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy dialects in jdbc source connnector
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 20:59:02 -07:00
Sergio Gómez Villamor
bdf17f551e
feat(ingest): glue - adds platform instance capability ( #4130 )
2022-03-30 18:50:26 -07:00
mohdsiddique
57002c766d
feat(stateful dbt): add stateful ingestion capability in dbt source ( #4456 )
...
* feat(stateful dbt): add stateful ingestion capability in dbt source
Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 18:09:02 -07:00
mayurinehate
9ba36100ab
feat(tableau): emit lineage edge from embedded datasource to upstream… ( #4470 )
...
* feat(tableau): emit lineage edge from embedded datasource to upstream published datasource
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 15:32:15 -07:00
Sunil Patil
36e9552d61
feat(ingestion): Support pluggable Schema Registry for Kafka Source ( #4535 )
...
* Support for pluggable schema registry for the Kafka source.
Co-authored-by: Sunil Patil <spatil@twilio.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 13:20:23 -07:00
Kevin Hu
1bad3c7bc9
fix(ingest): mssql - support database_alias ( #4523 )
2022-03-29 20:47:43 -07:00
Tamas Nemeth
4358d8fb01
feat(ingest): athena - set Athena location as upstream ( #4503 )
2022-03-29 07:06:48 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source ( #4490 )
...
* feat(data-lake): add containers and folder level dataset support
* docs(data-lake): Update readme for data lake
* doc(data-lake): fix examples, update doc
* lint fix
* feat(s3): add s3 source, restore old data-lake source
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently ( #4496 )
2022-03-27 18:49:26 -05:00
Xu Wang
d04092e634
feat(ingest): add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn ( #4469 )
...
* feat: add python utility classes for NotebookUrn, CorpuserUrn and CorpGroupUrn
Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
Co-authored-by: Ravindra Lanka <rslanka@gmail.com>
2022-03-23 16:07:57 -07:00
Aseem Bansal
c5f1d2c9bd
feat(ingestion): snowflake, bigquery - enhancements to log and bugfix ( #4442 )
...
feat(ingestion): add logging for snowflake, bigquery
2022-03-21 09:50:36 -07:00
cuong-pham
12bb2e1231
getting database directly from upstream tables incase there are multiple databases in upstreamDatabases ( #4447 )
2022-03-18 14:11:07 -07:00
Ravindra Lanka
60925e3e8c
Fix bug in the SchemaField type computation for AVRO logical types. ( #4433 )
2022-03-18 12:06:54 +01:00
mayurinehate
2f078c981c
feat(ingestion): tableau - support for tableau version 2021.1 and older ( #4437 )
...
fixes #4119
2022-03-17 14:07:36 -07:00
Tamas Nemeth
f557b2c1b3
fix(ingestion) containers: Adding platform instance to container keys ( #4279 )
2022-03-16 14:57:50 -07:00
Jorgen Evens
af5c4ee4d0
fix(ingest): handle endpoints without 200 response in openapi ( #4332 )
2022-03-14 17:52:08 -07:00
cccs-eric
cb9b99f0ba
fix(ingest) Azure AD: support nested groups ( #4367 ) ( #4368 )
...
LGTM - Thanks!
2022-03-14 08:59:04 -07:00
Aseem Bansal
4bcc2b3d12
feat(ingestion): improve logging, docs for bigquery, snowflake, redshift ( #4344 )
2022-03-14 08:50:29 -07:00
mayurinehate
3ea72869f3
feat(GE): add option to disable sql parsing, use default parser ( #4377 )
2022-03-10 17:36:59 -08:00
Vishal Shah
733413f58e
feat(ingest): mysql - add database_alias functionality ( #4319 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-09 09:29:58 -08:00
Tamas Nemeth
48380ada4c
fix(ingest) bigquery-usage: Adding credential support for bigquery usage ( #4111 )
2022-03-08 12:29:10 -08:00
MugdhaHardikar-GSLab
f198a92def
fix(config-parsing): add support for variable expansion for in variables in between string ( #4350 )
2022-03-08 12:24:08 -08:00
Aseem Bansal
7eec30b2ec
fix(hive): clean protocol for hive source ( #4330 )
2022-03-08 11:57:26 -08:00
Swaroop Jagadish
35b187a8d4
feat(ingest): transformers - add support for processing MCP-s ( #4337 )
2022-03-07 13:14:29 -08:00
mayurinehate
92b0e1c7c7
feat(tableau): emit workbook as container entity in tableau source, some minor fixes in tableau source ( #4261 )
2022-03-04 11:52:04 -08:00
John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA ( #4305 )
2022-03-04 11:51:31 -08:00
Aseem Bansal
9a9a5c3084
fix: revert changes to OpenApi casing ( #4291 )
2022-03-04 10:35:44 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source ( #4201 )
2022-02-28 17:37:22 -08:00
Vincenzo Lavorini
a113e4357e
fix(ingest): openapi - add support for user, password auth ( #4086 )
2022-02-24 23:29:01 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling ( #4131 )
2022-02-24 23:26:06 -08:00
Edward Vaisman
6ff551cbcd
feat(ingest): lineage-file - add ability to provide lineage manually through a file ( #4116 )
2022-02-24 17:02:38 -08:00
David Sanchez
8f75c278a1
fix(ingest): tableau - fix sheets external url ingestion ( #4231 )
2022-02-24 15:44:31 -08:00
Harshal Sheth
49a8ece02a
fix(ingestion): enable compat with avro 1.11 ( #4205 )
2022-02-22 22:13:50 -08:00
Xu Wang
aa3363bcc2
feat(ingest): lib - add better support for working with urns ( #4172 )
...
Co-authored-by: Xu Wang <xu.wang@grandrounds.com>
2022-02-22 19:39:24 -08:00
Ravindra Lanka
7f4cb87c57
Revert "fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )" ( #4218 )
...
This reverts commit 6c75185445bbb23974932ff64cb142ee6bf5b51b.
2022-02-22 16:21:40 -08:00
Ravindra Lanka
84005d3848
feat(ingest): kafka - add support for non-default schema registry subject name strategies ( #4215 )
2022-02-22 16:05:46 -08:00
Gabe Lyons
a9c1b7e6a3
fix(ingest): dbt - dont produce key aspects if the entity has no other aspects ( #4217 )
2022-02-22 16:02:58 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support ( #4057 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Tamas Nemeth
3d02b5bec8
feat(ingest): bigquery - ignore temporary tables from lineage and connect edges directly ( #4160 )
2022-02-20 14:23:23 -08:00
Harshal Sheth
1b60fae014
test(airflow): fix airflow version parsing ( #4142 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 18:13:01 -08:00
abiwill
8bbc66b3e6
fix(ingest): elasticsearch - http/https host config support ( #4191 )
...
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-19 11:42:01 -08:00
Harshal Sheth
75d2ec2a39
ci(ingestion): fix airflow 1 deps for tox ( #4083 )
2022-02-17 00:33:28 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. ( #4140 )
2022-02-16 19:45:07 -08:00
Tamas Nemeth
b2664916e3
feat(ingest): Glue - Support for domains and containers ( #4110 )
...
* Add container and domain support for Glue.
Adding option to set aws profile for Glue.
* Adding domain doc for Glue
* Making get_workunits less complex
* Updating golden file
* Addressing pr review comments
* Remove unneded empty line
2022-02-16 08:29:14 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion ( #4028 )
...
* test: test stateful ingestion for kafka
test: some more advancement
test: some improvements
refactoring
* refactor: remove some linter modifications
* tests: add unit tests for kafka state
* refactor: minor changes
* tests: improve test coverage
* fix: fix naming
* style: fix format with black
* fix: fix broken test
* revert: revert smoke tests to master
* feat: add reporting to kafka source
* tests: add smoke tests for kafka reporting
* revert: revert changes to the smoke tests
* test: add kafka integration test for stateful ingestion
* docs: update documentation on kafka source
* fix: return empty string when no platform instance
* revert: remove unwanted file
* fix: solve problem with platform instance
* chore: use console sink instead of file
* fix: disable complexity check for _extract_record
* fix: remove if condition in get_platform_instance_id
* chore: remove unneeded integration test
* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Tamas Nemeth
bfaec300b6
feat(ingest) Athena: Getting table properties for Athena datasets ( #4123 )
...
* Getting table properties for Athena datasets
* Isorting
* Fixing mypy error
* Addressing pr review comments
Adding tests
* Adding missing import
* black
* Fixing test run
* fixing flake8
* Adding athen to tox tests as well
* Not running athena tests on python < 3.7
* Adressing more pr comments
2022-02-14 13:51:45 -08:00
Harshal Sheth
ea2b092fe8
chore(ingest): remove unused groupby_unsorted utility ( #4011 )
2022-02-10 21:03:33 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging ( #4077 )
2022-02-10 20:02:23 -08:00
Claudio Benfatto
f944a9ba05
fix(ingest): enforce correct behaviour for commit policy ( #4092 )
2022-02-08 23:21:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] ( #4063 )
2022-02-08 14:26:44 -08:00
