1947 Commits

Author SHA1 Message Date
Mohit Yadav
466e325e0a
Added Server migration table to utility (#12263)
* Added Server migration table to utility

* updated to be in not to migrate

* py checkstyle failure
2023-07-03 19:45:50 +02:00
Onkar Ravgan
48b98d0be8
default gen time for comp time (#12256) 2023-07-03 12:30:02 +05:30
Pere Miquel Brull
49caca5309
Sample Data - Add users with dot in the name (#12253) 2023-07-03 07:25:23 +02:00
Pere Miquel Brull
b59ff24531
Fix looker view parser (#12254) 2023-07-03 07:25:08 +02:00
Teddy
93d132de5c
fix: removed SampleDataConnection class from package (#12229) 2023-07-01 11:21:21 -07:00
Onkar Ravgan
2a5a355c5f
fixed dbt table level test (#12247) 2023-07-01 19:00:41 +02:00
Pere Miquel Brull
1ecf5607c7
Looker - Fix file extension and blob import (#12232)
* Fix file extension and blob import

* Fix file extension and blob import
2023-06-29 16:14:17 +02:00
Onkar Ravgan
c1a379e202
made tableau name fields optional (#12223) 2023-06-29 14:04:07 +05:30
Onkar Ravgan
acf25f4555
Added pagination Tableau data sources graphql (#12187)
* Added pagination tableau graphql

* changed downstream workbook
2023-06-28 14:57:09 +02:00
Mayur Singal
d91a01a666
Fix vertica e2e test (#12195) 2023-06-28 14:04:27 +05:30
Ayush Shah
b764641a97
Connection Doc modifications - mysql, postgres & Mysql E2E fix (#12193)
* Fix Mysql E2E

* Fix Docs

* modify Security.md
2023-06-28 13:48:10 +05:30
Teddy
caa78a2310
fix: updated MySQL and Postres conn in backup/restore (#12191)
* fix: updated MySQL and Postres conn in backup/restore

* fix: ran linting
2023-06-28 13:46:29 +05:30
Ayush Shah
9e0a84e904
Fix Profiler Workflow infinite loop (#12185)
* Fix Profiler Workflow infinite loop
2023-06-27 20:19:02 +05:30
Ayush Shah
81a152140c
fix connection kwargs error in profiler (#12181) 2023-06-27 19:44:53 +05:30
Pere Miquel Brull
c22c5871f8
Update logging messages for JWT tokens (#12169) 2023-06-27 12:32:08 +02:00
Teddy
45a5ff49c8
fix: check on undefined bool value of clause (#12162) 2023-06-26 20:31:37 +02:00
Onkar Ravgan
9cbaf98435
remvoed ibm db arch (#12158) 2023-06-26 15:34:55 +02:00
Mayur Singal
b44d4f1e5e
Fix SQLLineage Test (#12152) 2023-06-26 17:09:49 +05:30
Pere Miquel Brull
a9135e8db5
Mask queries, Table sample data and Topic sample data (#12139)
* Filter out PII queries

* Mask topic PII

* Mask topic PII

* Update sample data test

* Format

* Moved logic to repository or PIIMasker

* Fix test

* chore(ui): update sample data api endpoint for topic

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-26 11:47:39 +02:00
Ayush Shah
29d3de2520
Move Top Level Imports (#12145) 2023-06-26 11:54:25 +05:30
Mayur Singal
a3fd6e9522
Fix #11659: Add support for filter patterns in dbt workflow (#12063) 2023-06-26 11:30:35 +05:30
Mayur Singal
18c8eb318e
Databricks Support Table Constraints (#12138)
* Databricks Support Table Constraints

* pylint fix

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-06-26 07:01:53 +02:00
Mayur Singal
7d1b123efe
Fix atlas lineage custom db name issue (#12108)
* Fix atlas lineage custom db name issue

* Fix test
2023-06-25 19:16:00 -07:00
Onkar Ravgan
3a9d8a9a45
powerbi loop bug fix (#12120) 2023-06-25 18:20:59 +02:00
Pere Miquel Brull
d0bc44127e
Set logger level for ingestion-base (#12137)
* Set logger level for ingestion-base

* Set logger level for ingestion-base
2023-06-25 08:20:46 -07:00
Pere Miquel Brull
97e08ee25c
Fix #12106 - Fix looker chart sourceUrl & Array datatype handling (#12113)
* Fix looker chart url

* Handle array datatype

* Handle array datatype
2023-06-25 08:18:36 -07:00
Aniket Katkar
67ffa1cce3
fix(UI): reverted the ES config step for 'Data Insights Pipeline' (#12121)
* Reverted the ES config step for Data Insights pipeline

* es sink fixes for ingestion

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-06-24 18:15:51 +05:30
Onkar Ravgan
d9d3f6895b
Fix 6874: Added Support for lineage from dbt ephemeral nodes (#12101)
* fixed dbt ephemeral nodes lin

* fixed dbt tests
2023-06-23 10:01:22 +02:00
Mayur Singal
258a2e33e2
Fix #12078: Add clientSessionKeepAlive Field for snowflake (#12107) 2023-06-23 10:00:53 +02:00
Pere Miquel Brull
b5c4d5a9d3
Fix #10175 - Add Oracle lineage and usage (#12090)
* Add Oracle lineage and usage

* Add docs

* Add examples
2023-06-23 06:52:34 +02:00
Teddy
6f33465b7c
fix: move service retrival to workflow (#12109) 2023-06-23 06:40:32 +02:00
07Himank
9355c0677d
Open search elastic search client task (#11879)
* Added diffrent clients for elasticSearch and openSearch and did the interface implementation

* removed some comments and some changes

* Add Elastic Search Test Container

* upgarded es version

* added test cases for elastic search

* added test case for, when tag is deleted cehck its relationships also get deleted

* changed package of open,elastic to openSearch and elasticSearch

* added support for skip search test case if index is not present of that entity

* Refactor bunch of files

* unnecessary semicolon

* refactor test case indexing

* fix checkstyle

* checkstyle

* added hardcoded configs and changed package name

* fixed no response from hits

* stopped elasticsearch container and some fixes

* closed elasticsearch container

* revert

* closed client after used in test

* style

* edit es config in junit

* checking if using already es version release memory for maven ci

* revert docker file changes

* remove es container for testing the ci

* commented out all test cases for elasticsearch

* fix

* creating elastic search with reusable enabled

* trying to fix ci error

* start container only if it is not running

* revert docker changes and done changes for elastic search container

* changed var name of sort_order for es

* lower the version of es client

* fixes

* all latest version of es including docker

* Use Single sort order

* Fix Keyword issue

* Immediate refresh policy for entity created

* increase sleep time

* disable test

* added reindexing logic in run_local_docker.sh

* enabled test case

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-06-22 16:55:05 +05:30
Onkar Ravgan
23b153991b
fixed mysqlclient dep (#12087) 2023-06-22 15:18:39 +05:30
Ayush Shah
cb6e42941a
Fix 12025: Clickhouse NaN issue (#12079) 2023-06-22 12:51:56 +05:30
Onkar Ravgan
5197682921
Fixed dagster bugs and Added Pydantic Models (#12048) 2023-06-22 10:59:09 +05:30
Onkar Ravgan
78d7faf04a
Added Azure Blob Storage to dbt config (#11839) 2023-06-22 10:58:38 +05:30
Pere Miquel Brull
3b8e5e69fd
Fix #12056 - Clean verbose debug logs (#12060)
* Clean verbose debug logs

* Format

* Remove unnecessary test

* remove verbosity
2023-06-22 06:54:09 +02:00
Teddy
1e86b6533c
Fixes #11743 - Remove SQLParse dependency for System Metrics (#12072)
* fix: removed sqlparse dependency for system metrics

* fix: update sample query

* fix: move system test os retrieval to `.get()`

* fix: move os.environ to `get`
2023-06-22 06:51:24 +02:00
Onkar Ravgan
8fac9d8976
Added Powerbi Lineage across workspaces and datamodel projects (#12052)
* powerbi lin multiple workspace

* Addressed review comments
2023-06-21 17:16:09 +02:00
Onkar Ravgan
0cfd8d29c5
Added dbt test_metadata check (#12067) 2023-06-21 12:17:26 +02:00
Ayush Shah
9fa7b78d87
Issue 11686: Add IncludeOwner + scheduleInterval in Airflow (#12005) 2023-06-21 14:18:45 +05:30
Onkar Ravgan
47514f6c5f
Fixed dbt test col name (#12065) 2023-06-21 10:47:54 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Pere Miquel Brull
8751427679
Add project to Data Models JSON Schema & Implement in Looker (#12030)
* Add project to dashboards and models

* Add project to dashboards and models

* Update openmetadata-spec/src/main/resources/json/schema/entity/data/dashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Update openmetadata-spec/src/main/resources/json/schema/api/data/createDashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Fix tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-06-20 11:45:29 +02:00
Mayur Singal
2738310887
Fix #12012: Fix clickhouse usage schema_name error (#12032) 2023-06-20 11:41:28 +05:30
Ayush Shah
83e9b6c310
Fixes 10395: Validation of yaml workflow configs (#11985) 2023-06-20 11:20:59 +05:30
Onkar Ravgan
f07c421264
Removed Empty Description Assignment to entities and added database name logic to tableau (#12031) 2023-06-19 19:19:42 +05:30
Teddy
76f5d3d571
Fixes #11994 - Update dbt and GE integration with new DQ flow (#12018)
* feat: updated GE integration to match new test workflow

* feat: updated unit tests to match new signature

* feat: added GE integration tests

* feat: ran python linting

* feat: updated dbt ingestion to match new TestSuite workflow

* feat: ran python linting

* feat: remove testSuite from Elasticsearch event test case update

* feat: ran java linting
2023-06-19 15:05:51 +02:00
Sriharsha Chintalapani
9dbaabad44
Adding MD5 hash to fullyQualifiedName and names to simplify DB indexes for lookups and increasing the size of FQN (#11960)
* Fix fqn hash

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* merge commits

* merge commits

* merge commits

* Fix glossary import/export

* Fix BotResource Tests

* Fix BotResource Tests

* Fix Glossary Term tests

* Fix Glossary Term tests

* Fix Glossary Import/Export tests

* All backend tests are fixed.

* merge with main

* Fix tests and revert local changes

* Fix tests and revert local changes

* Fix NullPointerException for Glossary and Query

* Fix Tests

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-19 15:43:05 +05:30
Pere Miquel Brull
4a8554c313
Fix Redshift performance and memory issues for ingestion-base (#12002)
* base op

* Update redshift deps and ingestion-base image

* Remove vim

* Remove vim
2023-06-19 12:09:09 +02:00