927 Commits

Author SHA1 Message Date
mayurinehate
0a97fa22f9
fix(tableau): fix for incorrect schema returned by tableau api for snowflake connectionType (#4577) 2022-04-05 14:56:35 -07:00
Ravindra Lanka
fe5f24c2b3
fix(ingestion): Refactor redshift_usage source: simplify, annotate & fix bugs. (#4572) 2022-04-05 09:21:27 -07:00
David Haglund
df9e07fda2
fix: replace direct and indirect references to linkedin with datahub-project (#4557)
* Update links for github-related links to use datahub-project:
  - https://github.com
  - https://img.shields.io/github/...
  - https://raw.githubusercontent.com/...
* Also replace references for github repo linkedin/datahub with
  datahub-project/datahub.
2022-04-04 14:39:30 -05:00
Abhiram98
26742728a6
feat(ingestion): schema, table filtering for redshift-usage (#4396)
* Filter based on table/schema pattern + documentation

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-04-01 20:48:23 -07:00
darapuk
a05d798939
(fix): Update path generated when creating LookML URL (#4554)
* (fix): Update path generated when creating LookML URL
2022-04-01 11:54:36 -07:00
mayurinehate
c09834d52b
fix(kafka-connect): add platform for default case in jdbc connector, update tests for platform instance map (#4545) 2022-03-31 08:13:09 -07:00
mayurinehate
467ea7917c
fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy d… (#4375)
* fix(kafka-connect): fix lineage for postgres-like 3-level hierarchy dialects in jdbc source connnector

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 20:59:02 -07:00
mohdsiddique
57002c766d
feat(stateful dbt): add stateful ingestion capability in dbt source (#4456)
* feat(stateful dbt): add stateful ingestion capability in dbt source

Co-authored-by: MohdSiddiqueBagwan <mohdsiddique.bagwan@gslab.com>
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 18:09:02 -07:00
mayurinehate
9ba36100ab
feat(tableau): emit lineage edge from embedded datasource to upstream… (#4470)
* feat(tableau): emit lineage edge from embedded datasource to upstream published datasource

Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
2022-03-30 15:32:15 -07:00
Kevin Hu
1bad3c7bc9
fix(ingest): mssql - support database_alias (#4523) 2022-03-29 20:47:43 -07:00
MugdhaHardikar-GSLab
37aedfc87c
feat(s3): add s3 source (#4490)
* feat(data-lake): add containers and folder level dataset support

* docs(data-lake): Update readme for data lake

* doc(data-lake): fix examples, update doc

* lint fix

* feat(s3): add s3 source, restore old data-lake source

Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-03-29 11:52:57 +02:00
Shirshanka Das
a69eac8247
feat(ingest): dbt,looker,sql_common,kafka - moving sources to produce display names and subtypes more consistently (#4496) 2022-03-27 18:49:26 -05:00
Aseem Bansal
c5f1d2c9bd
feat(ingestion): snowflake, bigquery - enhancements to log and bugfix (#4442)
feat(ingestion): add logging for snowflake, bigquery
2022-03-21 09:50:36 -07:00
cuong-pham
12bb2e1231
getting database directly from upstream tables incase there are multiple databases in upstreamDatabases (#4447) 2022-03-18 14:11:07 -07:00
mayurinehate
2f078c981c
feat(ingestion): tableau - support for tableau version 2021.1 and older (#4437)
fixes #4119
2022-03-17 14:07:36 -07:00
Tamas Nemeth
f557b2c1b3
fix(ingestion) containers: Adding platform instance to container keys (#4279) 2022-03-16 14:57:50 -07:00
cccs-eric
cb9b99f0ba
fix(ingest) Azure AD: support nested groups (#4367) (#4368)
LGTM - Thanks!
2022-03-14 08:59:04 -07:00
mayurinehate
3ea72869f3
feat(GE): add option to disable sql parsing, use default parser (#4377) 2022-03-10 17:36:59 -08:00
Vishal Shah
733413f58e
feat(ingest): mysql - add database_alias functionality (#4319)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-03-09 09:29:58 -08:00
mayurinehate
92b0e1c7c7
feat(tableau): emit workbook as container entity in tableau source, some minor fixes in tableau source (#4261) 2022-03-04 11:52:04 -08:00
John Joyce
9f1c5a8f75
feat(assertions): Adding Assertions Entity & Great Expectations BETA (#4305) 2022-03-04 11:51:31 -08:00
Aseem Bansal
9a9a5c3084
fix: revert changes to OpenApi casing (#4291) 2022-03-04 10:35:44 -08:00
mohdsiddique
e2f8db7926
feat: powerbi - add new source (#4201) 2022-02-28 17:37:22 -08:00
Vincenzo Lavorini
a113e4357e
fix(ingest): openapi - add support for user, password auth (#4086) 2022-02-24 23:29:01 -08:00
Kevin Hu
02fe05eb8f
feat(ingest): data-lake - remove spark requirement if not profiling (#4131) 2022-02-24 23:26:06 -08:00
David Sanchez
8f75c278a1
fix(ingest): tableau - fix sheets external url ingestion (#4231) 2022-02-24 15:44:31 -08:00
Ravindra Lanka
7f4cb87c57
Revert "fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. (#4140)" (#4218)
This reverts commit 6c75185445bbb23974932ff64cb142ee6bf5b51b.
2022-02-22 16:21:40 -08:00
Gabe Lyons
a9c1b7e6a3
fix(ingest): dbt - dont produce key aspects if the entity has no other aspects (#4217) 2022-02-22 16:02:58 -08:00
Alexander Chashnikov
c2065bd7fe
feat(ingest): clickhouse - add initial support (#4057)
Co-authored-by: Shirshanka Das <shirshanka@apache.org>
2022-02-21 07:36:08 -08:00
Ravindra Lanka
6c75185445
fix(ingest): Use lower-case dataset names in the dataset urns for all SQL-styled datasets. (#4140) 2022-02-16 19:45:07 -08:00
Claudio Benfatto
aeefde4fa1
feat(ingestion): Kafka stateful ingestion (#4028)
* test: test stateful ingestion for kafka

test: some more advancement

test: some improvements

refactoring

* refactor: remove some linter modifications

* tests: add unit tests for kafka state

* refactor: minor changes

* tests: improve test coverage

* fix: fix naming

* style: fix format with black

* fix: fix broken test

* revert: revert smoke tests to master

* feat: add reporting to kafka source

* tests: add smoke tests for kafka reporting

* revert: revert changes to the smoke tests

* test: add kafka integration test for stateful ingestion

* docs: update documentation on kafka source

* fix: return empty string when no platform instance

* revert: remove unwanted file

* fix: solve problem with platform instance

* chore: use console sink instead of file

* fix: disable complexity check for _extract_record

* fix: remove if condition in get_platform_instance_id

* chore: remove unneeded integration test

* test: test platform instance in kafka source unit tests
2022-02-15 07:18:36 -08:00
Aditya Radhakrishnan
b331106dc7
fix(ingest): okta - better use of asyncio and additional debug logging (#4077) 2022-02-10 20:02:23 -08:00
John Joyce
2a9a076fc1
feat(ingest): Adding Tableau Source Connector [BETA] (#4063) 2022-02-08 14:26:44 -08:00
Tamas Nemeth
63bc830cfe
Data domain containers ingestion (#4051) 2022-02-07 09:51:49 -08:00
ksrinath
4c24f386a6
feat(model): data quality model (#3787)
Co-authored-by: Ravindra Lanka <rlanka@acryl.io>
Co-authored-by: Mayuri N <mayuri.nehate@gslab.com>
2022-02-02 22:56:08 -08:00
Swaroop Jagadish
ded16809da
feat(ingest): add tests for platform instance (#4047) 2022-02-02 22:52:50 -08:00
mayurinehate
1afe8876b7
feat(ingest): nifi - handle provenance api variation for older versions (#4022) 2022-02-01 10:03:05 -08:00
mayurinehate
faac911df8
feat(ingest): use trino python client with sqlalchemy, supports python3.6 (#3888) 2022-01-31 09:22:55 -08:00
Tamas Nemeth
646dd6d5f0
fix(ingest): profiler - enable profiling for low cardinality number columns (#3990) 2022-01-30 17:36:18 -08:00
Swaroop Jagadish
9a9f448ebf
feat(ingest): platform-instance - changes to support platform instances in sources (#3996) 2022-01-27 15:31:25 -08:00
Kevin Hu
62f7d05557
feat(ingest): data-lake - support for Avro files (#3913) 2022-01-26 10:29:00 -08:00
iasoon
5dac30c93e
fix(ingestion): match default username for Azure OIDC and Azure ingestion source (#3926) 2022-01-25 18:57:18 -08:00
Kevin Hu
9e30b42638
feat(ingest): data lake - support for profiling data lake assets (#3656) 2022-01-17 15:01:23 -08:00
Swaroop Jagadish
7d986ec880
fix(ingest): populate system metadata for all metadata events (mcp, mcpw) (#3900) 2022-01-16 12:03:38 -08:00
Aditya Radhakrishnan
c3e98645a2
feat(ingest): operational stats - show last updated for sql usage sources (#3845) 2022-01-07 17:27:09 -08:00
Paolo Genissel
744dee5206
fix(ingest): bigquery-usage - fix remove_extras to remove all partitions (#3842) 2022-01-07 12:30:45 -08:00
Ravindra Lanka
9f80e5487d
fix(ingest): metabase - fix the datetime parsing issue (#3831)
fixes #3803
2022-01-05 16:34:05 -08:00
Swaroop Jagadish
480ad24282
feat(ingest): dbt - add support for dbt meta mapping (#3832) 2022-01-05 16:32:05 -08:00
Hyun Min Choi
e76d40b57a
fix(lkml): bump lkml version up to 1.1.2 to support sql_preamble expression (#3757) 2021-12-21 12:13:13 -08:00
Tamas Nemeth
599edd22ae
fix(ingest): profiling - disable expensive profilers by default (#3759) 2021-12-17 17:17:25 -08:00