797 Commits

Author SHA1 Message Date
Mayur Singal
0c6435ae9b
Fix #2957: Add support for qlik sense enterprise connector (#12450) 2023-07-19 21:08:05 +05:30
Suresh Srinivas
35154615d4
Code cleanup.20230718 (#12493)
* Move setting common patch and updated fields to EntityRepository

* Cleanup entityReferenceList.json and references
2023-07-18 15:10:22 -07:00
07Himank
b6d5a438ec
Support '%' in entity Name (#12382)
* added support for the % in entity name

* added support for the % in glossary and glossary term  name

* Added U exclude

* Fixed pytest

* fix: support % in glossary name

* added suppport for % in entityName

* fix % issue in the glossary or term

* fix: glossary with % encoding

* fix: tags redirect issue for glossaries with %

* fix: glossary import path

* fixed CI

* fixed Py Test

* fixed PyTest

* fixed glossary resource test

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: karanh37 <33024356+karanh37@users.noreply.github.com>
2023-07-18 17:58:16 +05:30
Sachin Chaurasiya
8734e77acc
fix: json schema ref and id path (#12471) 2023-07-18 07:52:23 +00:00
Mohit Yadav
ac9ccc3c2b
Add missing Expert Enum name (#12454) 2023-07-17 11:49:41 +05:30
Suresh Srinivas
0d119de96e
Fixes #12447 Use JSON schema constrain validation in CSV imports (#12448) 2023-07-15 15:02:17 -07:00
Pere Miquel Brull
3433e2b5dd
Ingestion Pipeline - Bulk deploy API & Status response updated (#12365)
* Redeploy in tables init

* Use Status class

* Update response Status

* Format

* Rename pipelineServiceClientResponse
2023-07-14 07:38:29 +02:00
vanshika18
bbe58b1b5f
Handle Table type for Snowflake Transient Tables (#12274)
* Handle Table type for Snowflake Transient Tables

* Clean the code by removing comments

* Optimize the code

* Optimize the code

* Clean the code

* Rename includeTempTables with includeTransientTables

* Rename includeTempTables to includeTransientTables

* Update Documentation and resolved migartion query error

* Resolved migartion query for postgres connection

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-13 19:25:46 +02:00
Pere Miquel Brull
f0f74dbe59
Fix #12368 - Make sourcePythonClass not required to allow generic use of custom connectors (#12408) 2023-07-13 18:03:14 +02:00
Onkar Ravgan
e200a42926
Removed datamodels references from dashboard and datamodels (#12371) 2023-07-13 14:26:28 +05:30
Pere Miquel Brull
1b7c5e3233
Fix #12321 - Deprecate impala scheme from Hive (#12327)
* Deprecate impala scheme from Hive

* Add migrations
2023-07-11 17:38:32 +02:00
Sriharsha Chintalapani
155cf354e3
Fix OpenMetadata schema to TypeScript generation (#12335)
* initial fix to add relative path to json schemas

* initial fix to add relative path to json schemas

* fix schema generation

* fix github workflow

* add license

* add jq to developer prerequisites

* add jq to developer prerequisites
2023-07-09 21:12:46 -07:00
Megumi AIKAWA
9aede5e821
Fixes 11924 : Add pgspider support (#12159)
* Add PGSpider service connector

* add sqlalchemy-pgspider to setup.py

* Delete unnecessary code and move files.

* Change to work as a Postgres connector sub-module

* Rebuild PGSpiderLineage class into a set of funcs

---------

Co-authored-by: Pham Ngoc Son <son1.phamngoc@toshiba.co.jp>
2023-07-05 09:18:59 +02:00
Suresh Srinivas
4f2831f3bb
Upgrade dependencies for 1.2.0 release (#12280) 2023-07-04 20:03:40 -07:00
Suresh Srinivas
97140e13fd
12142.task2 - Add domain and data product properties to entities (#12255) 2023-07-03 15:42:35 -07:00
Suresh Srinivas
4d4993effc
12142.task1 - Add support for Domains (#12143) 2023-07-02 11:32:52 -07:00
07Himank
7244c3f530
changed enum from Elastic/Open Search to elastic/open search (#12237) 2023-06-29 20:13:49 -07:00
Onkar Ravgan
acf25f4555
Added pagination Tableau data sources graphql (#12187)
* Added pagination tableau graphql

* changed downstream workbook
2023-06-28 14:57:09 +02:00
Mayur Singal
a3fd6e9522
Fix #11659: Add support for filter patterns in dbt workflow (#12063) 2023-06-26 11:30:35 +05:30
Mayur Singal
258a2e33e2
Fix #12078: Add clientSessionKeepAlive Field for snowflake (#12107) 2023-06-23 10:00:53 +02:00
Pere Miquel Brull
b5c4d5a9d3
Fix #10175 - Add Oracle lineage and usage (#12090)
* Add Oracle lineage and usage

* Add docs

* Add examples
2023-06-23 06:52:34 +02:00
Teddy
712dcf8587
fix: added field to testCase entity to list all Test Suites (#12093)
* fix: added  field to testCase entity to list all the linked test suites (executable and logical)

* fix: resolved conflicts from main

* fix: removed unecessary comment
2023-06-22 18:00:19 +02:00
07Himank
9355c0677d
Open search elastic search client task (#11879)
* Added diffrent clients for elasticSearch and openSearch and did the interface implementation

* removed some comments and some changes

* Add Elastic Search Test Container

* upgarded es version

* added test cases for elastic search

* added test case for, when tag is deleted cehck its relationships also get deleted

* changed package of open,elastic to openSearch and elasticSearch

* added support for skip search test case if index is not present of that entity

* Refactor bunch of files

* unnecessary semicolon

* refactor test case indexing

* fix checkstyle

* checkstyle

* added hardcoded configs and changed package name

* fixed no response from hits

* stopped elasticsearch container and some fixes

* closed elasticsearch container

* revert

* closed client after used in test

* style

* edit es config in junit

* checking if using already es version release memory for maven ci

* revert docker file changes

* remove es container for testing the ci

* commented out all test cases for elasticsearch

* fix

* creating elastic search with reusable enabled

* trying to fix ci error

* start container only if it is not running

* revert docker changes and done changes for elastic search container

* changed var name of sort_order for es

* lower the version of es client

* fixes

* all latest version of es including docker

* Use Single sort order

* Fix Keyword issue

* Immediate refresh policy for entity created

* increase sleep time

* disable test

* added reindexing logic in run_local_docker.sh

* enabled test case

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-06-22 16:55:05 +05:30
Onkar Ravgan
78d7faf04a
Added Azure Blob Storage to dbt config (#11839) 2023-06-22 10:58:38 +05:30
Pere Miquel Brull
520d5e2657
Add Pipeline Service Status Job to publish to Prometheus (#12046)
* Add Pipeline Service Status Job to publish to Prometheus

* Add Pipeline Service Status Job to publish to Prometheus

* Add config for health check interval
2023-06-22 06:53:56 +02:00
Ayush Shah
9fa7b78d87
Issue 11686: Add IncludeOwner + scheduleInterval in Airflow (#12005) 2023-06-21 14:18:45 +05:30
Pere Miquel Brull
bfa37df298
Add platform in Pipeline Service status msg (#12041)
* Add platform in Pipeline Service status msg

* chore(ui): show message based on platform

* update the styling

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-21 07:21:19 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Pere Miquel Brull
8751427679
Add project to Data Models JSON Schema & Implement in Looker (#12030)
* Add project to dashboards and models

* Add project to dashboards and models

* Update openmetadata-spec/src/main/resources/json/schema/entity/data/dashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Update openmetadata-spec/src/main/resources/json/schema/api/data/createDashboardDataModel.json

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

* Fix tests

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-06-20 11:45:29 +02:00
Sriharsha Chintalapani
9dbaabad44
Adding MD5 hash to fullyQualifiedName and names to simplify DB indexes for lookups and increasing the size of FQN (#11960)
* Fix fqn hash

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* merge commits

* merge commits

* merge commits

* Fix glossary import/export

* Fix BotResource Tests

* Fix BotResource Tests

* Fix Glossary Term tests

* Fix Glossary Term tests

* Fix Glossary Import/Export tests

* All backend tests are fixed.

* merge with main

* Fix tests and revert local changes

* Fix tests and revert local changes

* Fix NullPointerException for Glossary and Query

* Fix Tests

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-19 15:43:05 +05:30
Mayur Singal
8ecd88fd1b
Fix #11312: Deprecate ES Reindex on Ingestion and use reindex APIs instead (#11988) 2023-06-19 12:45:18 +05:30
Mohit Yadav
41c49b9de8
Data Insight Report Job Bug Fixes (#11975)
* Data Insight Report Job Bug Fixes

* ui: add enabled field

* test case fix

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-16 15:40:44 +05:30
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Teddy
3f01ee938f
Fixes #11895 - Add Indexing and Search logic for TestCases (#11989)
* feat: added logic to delete logical test suite + added check to prevent adding existing testCases to executable test suite

* feat: added elasticsearch index logic for testCases

* feat: added deletion logic from index logic when deleting test suites

* feat: added test case index search to  endpoint

* feat: add executable/logical filter in list testSuite + filterOut tables without tests in Table resource

* feat: added summary field to testSuite

* feat: added executionSummary endpoint for test cases

* feat: removed tick marks around timestamp

* feat: addressed test failures

* feat: ran python linting

* feat: add limit to fetch all tables in TableResource testSuite test

* feat: fix conflict

* feat: ran java checkstyle

* feat: fixed mongo linting + disabled mongo failing tests

* feat: removed mongo test skip

* feat: removed unsued pytest import
2023-06-15 21:27:54 +02:00
Onkar Ravgan
d08c928801
Added project property to dashboards (#11986)
* Added projects to dashboards

* Added powerbi proj

* merge conflicts after source url

* fixed mongo pytest
2023-06-15 21:23:43 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities (#11991) 2023-06-15 14:44:48 +05:30
Mayur Singal
7fa963eec3
Fix #1076: Add mongodb support (#11943) 2023-06-15 11:14:22 +05:30
Onkar Ravgan
d409c339e4
Fix 10689: Added Powerbi Datamodels (#11923) 2023-06-14 17:00:08 +05:30
Ayush Shah
2e5560926d
Add Lineage support for mssql (#11971) 2023-06-13 17:10:26 +05:30
Pere Miquel Brull
ba5f929f77
Add fine-grained health validation for Airflow APIs (#11792)
* Add fine-grained health validation for Airflow APIs

* Add ingestion version response

* Improve messaging

* Format

* Format

* Update response

* ui:update the hook to return airflow status based on response status field

* ui: add unit test for useAirflowStatus hook

* chore: only show test connection if airflow is available

* feat: add airflow message banner

* chore: update icon and background color

* chore: update typography to text

* test: add unit test

* address comments

* chore: show banner on service detail page

* fix: update test suite api workflow to reflect new implementation

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-06-12 07:47:45 +02:00
07Himank
62af9bb633
fixed issue for lineage description (#11500)
* fixed issue for lineage description

* fixed issue while ingesting

* fixed issue while ingesting

* added test case for Lingeage with description

* addressing comments .. enhancement

* addressing comments .. enhancement

* modified py test case and removed description from addLineage as we are not using it.

* add support for topic entity and description in lineage details

* fix pylint & test

* pytest fix

* fix column lineage null issue

---------

Co-authored-by: Himank Mehta <himankmehta@Himanks-MacBook-Air.local>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-06-12 11:17:32 +05:30
Mayur Singal
175093d93e
Fix datamodel generate issue for 3.8 (#11934) 2023-06-09 12:33:21 +05:30
Mayur Singal
f43aaf4150
Add database name filed for oracle db (#11906) 2023-06-09 09:31:07 +05:30
Mayur Singal
9e6e00caf9
Fix #9282: Support Databricks Unity Catalog (#11896) 2023-06-07 11:50:31 +05:30
Suresh Srinivas
f23f67dfe9
Fixes #11617 Entity get operations check for ViewAll operation instead of specific view operations based on the field requested (#11904) 2023-06-06 15:18:57 -07:00
Mohit Yadav
57e44c22be
Make Resource Packages Dynamic (#11891) 2023-06-07 01:02:52 +05:30
Teddy
4b9f213dbf
Fixes Issue #11863 - Add Status to DQ (#11893)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting

* feat: fixed  update to
2023-06-06 10:09:16 +00:00
Teddy
721869428e
Revert "Fixe Issue #11863 - Add Status logic for test case results (#11881)" (#11892)
This reverts commit 06735fe8dbaac5b267c9a2cf744ca154f88a9247.
2023-06-06 09:56:12 +02:00
Teddy
06735fe8db
Fixe Issue #11863 - Add Status logic for test case results (#11881)
* feat: added entityReference field in testSuite to link testSuite to an entity when the testSuite is executable.

* feat: added `executableEntityReference` as an entity reference for executable test suite to their entity

* feat: add status object to test case results

* feat: ran python linting
2023-06-06 09:45:49 +02:00
Ayush Shah
65f370e4aa
Rename GCS to GCP (#11812) 2023-06-06 11:57:00 +05:30