2916 Commits

Author SHA1 Message Date
Pere Miquel Brull
dcbb77f46c
Looker - Update GitHub creds for UI form (#11019)
* Update GitHub creds for UI form

* Add filters
2023-04-13 06:05:02 +02:00
Pere Miquel Brull
47cef52fa8
Handle container parents (#11026) 2023-04-12 18:36:04 +02:00
Nahuel
673573a512
Fix#10584: Add filters for data models (#11008)
* Fix#10584: Add filters for data models

* Minor UI error

* fix unit test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-04-12 17:36:01 +05:30
Onkar Ravgan
dbd8da6a29
Added logic for cleaning up schemas (#10998)
* Added logic for deleting schemas

* added UI changes to allow soft deletion and restoration of database schemas

* fixed failing cypress test

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-04-12 16:50:44 +05:30
Onkar Ravgan
1b930fa6f7
Added enum for ssl and auth (#11016) 2023-04-12 10:57:31 +00:00
Mayur Singal
ae6683862f
Fix #10999: Add support for materialized views in oracle (#11005) 2023-04-12 16:21:27 +05:30
Ayush Shah
288358952b
Fixes 8264: add docs around adding taxonomy project id (#11018)
* add docs around taxonomy project id

* Add taxonomyProjectID to bigquery.yaml
2023-04-12 16:00:05 +05:30
Pere Miquel Brull
b5cb1d464a
Deprecate location and old storage service (#11004)
* Deprecate location and old storage service

* Format

* Fix test

* Refactor

* Clean location

* Rename object store to storage

* Rename object store to storage

* Rename object store to storage

* Format

* Format

* Refactor object store for storage

* Refactor object store for storage

* Rename object store to storage

* Fix test

* Fix test

* Format

* chore(ui): change Objectstore to  Storage

* Fixes

* Fix test

* Remove storage service from Glue cypress

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-04-12 11:44:46 +02:00
Schlameel
4c3f142a2c
Fixes #6340: Implement Median support for MySQL (#10962)
* ISSUE #6340: Implement Median support for MySQL
- Added code to existing function that previously returned None
- Important - Tested only external to OM.
- Performance tested - 1/7th the speed of other solutions. Not impacted by index.

* ISSUE #6340 - Implement median support for MySQL
Changed to remove setting user variable in expression per Teddy

* ISSUE #6340 - Implement median support for MySQL
Formatting
2023-04-12 08:07:36 +02:00
Pere Miquel Brull
b988f39152
Fix test usage resources (#11014) 2023-04-12 05:46:29 +00:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning (#10603)
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Deepa Rao
147b7c551e
Fixes #10854 - Create/Update Chart Not working as Expected (#10991) 2023-04-11 06:28:01 -07:00
Mayur Singal
75aa42fe43
Use metabase models in client (#10995) 2023-04-11 18:28:35 +05:30
Pere Miquel Brull
09b283818d
Rel to #10927 - Looker DataModel (#10945)
* Organise calls

* Prepare skeleton

* Add looker model handling

* Parse files as sql

* Handle labels

* Linting

* Format

* Fix version

* Also check the API for explore lineage
2023-04-11 08:44:00 +02:00
Ayush Shah
02a52b3ddf
Fix table name with underscore (#10980) 2023-04-10 11:41:36 +00:00
Onkar Ravgan
3c32658d7a
Fixed dbt logging (#10977)
* Fixed dbt logging

* Fixed pytests

* Fixed pycheckstyle
2023-04-10 10:04:25 +00:00
Mayur Singal
9f2b10f6e2
Fix #10858: Fix dashbaord & pipeline url (#10957) 2023-04-10 09:48:49 +00:00
Mayur Singal
3d345f9b37
Fix #10273: Parse nested json for datalake (#10956) 2023-04-10 14:58:02 +05:30
Ayush Shah
6ff3557917
Issue 10975 - fix workflow break at none (#10976) 2023-04-10 09:05:38 +00:00
Teddy
203f17fa49
fix: add rollback logic to custom sql query to exit out of transaction (#10968) 2023-04-07 14:46:05 +02:00
Keith Sirmons
27984c25f3
Fixes 10959: MSSQL profile has error when averaging large integer column (#10960)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala

* added cast to decimal for column when running average in mean.py

* fixed lint error
2023-04-07 06:48:18 +02:00
Hemal Mamtora
0206af0303
[WIP] Fixes #10721: created client file for metabase and moved get request to client file (#10838)
* ingestion/metadata - created client file and moved get request to client file

* moved code to client.py, created models.py

* added, self to MetabaseClient constructor

* added status_code in metabase client functions and handled Optional datatype in metabase metadata functions

* moved api calls to client

* passing service_connection and metabase_session from constructor to MetabaseClient

* renamed client to metabaseClient to avoid rewriting DashboardServiceSource.client

* moved post request from connection.py to client.py

* run make py_format

* moved client code to init constructor

* added Dashboard Model for Metabase

* updated client type to MetabaseClient and used get_dashboards_list() function in custom_executor

* updated client type to MetabaseClient and used get_dashboards_list() function in custom_executor

* make py_format

* pylint issues resolved

* pylint and run make py_format

* calling init method in Metabase source

* removed init method
2023-04-06 19:26:52 +02:00
Keith Sirmons
42000053aa
Fixed Issue #10943: Impala query engine metadata ingestion and median function profiler (#10944)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala
2023-04-06 18:07:42 +02:00
Nahuel
62b88b0404
Fix#10584: Add lineage for Tableau data models (#10923)
* Refactor Tableau ingestion

* Add lineage from DB -> DataModel -> Dashboard

* Address PR comments

* Address PR comments
2023-04-06 07:28:41 +02:00
Pere Miquel Brull
c53c7b680d
Rel #10828 - Update container index and fix airflow ingestion (#10873)
* Update container index and fix airflow ingestion

* Fix lint

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-04-06 07:09:38 +02:00
Teddy
06b8d8e7ce
fix: explicitly state USE CATALOG for databricks connection (#10940) 2023-04-05 18:47:18 +02:00
Teddy
4683bee91a
Fixes #10911 - Improve Data Insight Logging (#10915) 2023-04-05 18:46:39 +02:00
Onkar Ravgan
c3e28bcf56
Remove the non-required fields from manifest file (#10935)
* remove non required manifest keys

* Added comment

* Addressed review comments
2023-04-05 19:39:53 +05:30
NiharDoshi99
c44b0bbcea
adding timeout option for Dagster and Amundsen changes (#10934)
* adding timeout option for dagster and amundsen changes

* Update ingestion/src/metadata/ingestion/source/pipeline/dagster/connection.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-04-05 10:33:30 +00:00
Mayur Singal
9ed1fc4e17
Test Connection Improvements: Short Circuit Test Connection & Improve Error Messages (#10913) 2023-04-05 13:23:57 +05:30
Mayur Singal
d7e0153000
Fix #10896: Fix snappy coded issue (#10919) 2023-04-05 12:12:47 +05:30
Shailesh Parmar
592ae305c3
upgraded cypress to v12.9.0 and fixed service connection cypress (#10891)
* upgraded cypress to v12.9.0 and fixed service connection cypress

* fixing failing cypress

* updated cy setting

* fixed restore cy test

* added API wait for users spec

* fixed suspected specs

* Fix ES aligning suggests with Java

* separate the tag test for DatabaseSchema

* updated tags spec

* skip advance search

* fixed failing API's

* await on tags api

* debounce search api

* update numKeptInMemory flag to minimum

* fix unit tests and skip failed tests

* fixed failing cy test

* fixed redirection

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-04-05 00:06:11 +05:30
Teddy
9b4e9132ae
fixed #9656 - Add support for date type to column values to be between (#10890)
* fix: renamed  to  submodule

* fix: linting

* fix: columnValuesToBeBetween test for date column type
2023-04-04 17:16:44 +02:00
Mayur Singal
5bd476da23
Test Connection Improvements: Use patch instead of put (#10897)
* Test Connection Improvements: Use patch instead of put

* Update openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/WorkflowRepository.java

Co-authored-by: Nahuel <nahuel@getcollate.io>

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-04-04 14:15:51 +00:00
Onkar Ravgan
608bf51b6a
Added non admin apis support for PowerBI (#10910)
* Added non admin apis support

* addressed review comments
2023-04-04 18:25:48 +05:30
Suresh Srinivas
c8b640674b
10041 part2 - Refactor and cleanup APIs (#10900)
* Use @Tag annotation to group APIs in the swagger documentation.

* Hide internal APIs

* Change API path events/subscription to events/subscriptions

* Change API path from automations/workflow to automations/workflows

* Change API path v1/testCase to v1/testCases

* Change API path v1/testDefinition to v1/testDefinitions

* Change API path v1/testSuite to v1/testSuites

* Rename Kpi and kpi in the documentation to KPI

* Change API path v1/testConnectionDefinition to v1/testConnectionDefinitions

* Update API section in the API documentation

* Fix test failures

* Correctly capitalize Test Cases and Test Suites in API docs
2023-04-03 13:03:48 -07:00
NiharDoshi99
bb3e80c868
Data model entity added for Tableau (#10872)
* data model entity added for tableau

* add column field changes for Data Model

* added column data type enum and changes as per comment
2023-04-03 17:59:54 +02:00
Teddy
2ed2e9f20a
Fix #8726 - Add permission to data insight (#10694)
* feat: added auth logic for reportData endpoint

* feat: added permission logic for dataInsightChart and dataReport

* feat: ran mvn clean for code linting

* feat: changed endpoint name to `/reports`

* fix: renamed analytics API endpoints

* fix: eslint

* fix: updated data insight test collection path

* fix: description and changed dataInsight to dataInsights
2023-04-03 14:16:27 +02:00
Nahuel
6c9ef22168
Update: openmetadata-sqllineage dependency (#10894) 2023-04-03 11:41:13 +00:00
Mayur Singal
69729ac8eb
Fix #10320: Add Athena Usage & Lineage (#10879) 2023-04-03 11:21:55 +00:00
Mayur Singal
752163ac71
Fix #10814: Improve parsing logic for union fields in topic (#10836) 2023-04-01 11:10:05 +05:30
Teddy
ecffd5ffc7
Fixes #10727 (& other minor improvements) (#10856)
* fix: logic for test suite config workflow

* fix: added caching for system metrics (snflk and bq)

* fix: linting

* fix: added tearDown logic for tests suite/case
2023-03-31 16:57:53 +02:00
Schlameel
6d24455738
Fixes 10343: Add methods to update Glossary and GlossaryTerm in Python SDK (#10810)
* ISSUE 10343: Python SDK Glossary and GlossaryTerms
- Added methods to glossary_mixin to PATCH Glossary and GlossaryTerm
- Created in patch_mixin_utils a super class for mixins that PATCH entities
- Moved common Patch enums from patch.py to patch_mixin_utils.py
- Updated imports and super classes for mixins that PATCH entities
- Added tests for Glossary and GlossaryTerm mixins

* ISSUE #10343: Python SDK extensions for Glossary and GlossaryTerms
- Fixed an import
- Fixed two method signatures

* Issue #10343 - Fixed formatting
2023-03-31 16:55:22 +02:00
Nahuel
3b16a73033
Fix#8852: Updating comments on sql lineage flow (#10864)
* Fix#8852: Updating comments on sql lineage flow

* Change docstring
2023-03-31 11:59:36 +00:00
Nahuel
0ca8c12596
Fix#10584: Update dashboard ES indexes for dataModels (#10851)
* Fix#10584: Update dashboard ES indexes for dataModels

* Address pylint error
2023-03-31 11:50:43 +02:00
Pere Miquel Brull
967413e1b1
Update ingestion ES mappings & restructure sink (#10852)
* Fix dashboard chart handling

* Update mappings with latest changes

* Add containers to indexing

* Add queries to indexing

* Clean ES reindex

* Linting
2023-03-31 09:57:30 +02:00
Nahuel
6fa6b95f74
Fix: Remove bot configuration with SSO service account (#10847)
* Add deprecation warning for SSO config for bots

* Fix logging
2023-03-30 17:37:45 +00:00
NiharDoshi99
3406c8c868
removed en_web_md from setup (#10839)
* removed en_web_md from setup

* Use Constant

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-03-30 09:43:41 +00:00
Pere Miquel Brull
2e2c6a0cdf
Operators directory cleanup (#10834)
* Remove old connectors directory

* Add mssql test for url

* Format
2023-03-30 09:51:21 +02:00
Mayur Singal
ec0ca7010e
Fix Snowflake Test Connection when no database passed (#10831)
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-03-29 18:19:22 +00:00