2535 Commits

Author SHA1 Message Date
Nahuel
3f491a8578
Fix#6027: Improve logging in ingestion library (#6931)
* Improve logging in ingestion library

* Some extra fixes in logging
2022-08-29 08:06:29 +02:00
Abhishek Pandey
fde53809eb
added-query-log-parser-in-postgres (#6750)
added-query-log-parser-in-postgres (#6750)
2022-08-29 06:48:29 +02:00
Nahuel
4c6f0c7bda
Fix#6027: Fix logging in metadata package modules except ingestion and orm_profiler (#6970)
* Fix logging in metadata package modules except ingestion and orm_profiler

* Reduce log level to _get_columns_with_constraints method
2022-08-29 06:46:06 +02:00
Sriharsha Chintalapani
f9a3f9988e
Fix #6936: testPlatforms is required field in testSuite and createTes… (#6944)
* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* Fix #6936: testPlatforms is required field in testSuite and createTestSuite schema

* fixed glossary test

* Fixed sample data ingestion

* Add filters for testCaseResultSuccess
testCaseResultFailed, testCaseResultAborted

* Fix #6936: fix testcase listing

* Fix #6936: fix java style

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-08-27 13:01:24 +05:30
Pere Miquel Brull
6cebf37b6c
Fix #6906 - Add Nifi connector (#6956)
* Add Nifi connector

* Mock token

* Format

* pass verify param
2022-08-26 21:50:01 +02:00
Teddy
1234b6f220
Fixes #5313 by adding logic to skip non OM on testSuite workflow (#6969) 2022-08-26 18:53:32 +02:00
Onkar Ravgan
13d433b0d7
Add Owner for DBT (#6872)
* Added owner field in DataModel

* Added user in backend

* review comments addressed

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-26 16:22:27 +05:30
Mayur Singal
6c1f84dcd7
Use Table FQN instead of Name for usage status (#6952)
* Use Table FQN instead of Name for status

* Update log with fqn instead of name
2022-08-26 08:58:30 +02:00
Teddy
075a0196cf
Fixed #6762 DB2 test connection (#6928)
* Fixed DB2 test connection

* Fixed query

* Fixed function
2022-08-26 07:14:17 +02:00
Nahuel
7863f040d1
Fix#6027: Improve logging in ORM profiler (#6919)
* Improve logging in ORM profiler

* Fix failing test

* Updated logging in test_suite module to match company's format

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-25 11:19:31 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Pere Miquel Brull
75dcbe612a
Validate service sink ack (#6895)
* Validate service sink ack

* naming
2022-08-25 08:41:22 +02:00
Mayur Singal
404cc67911
WIP - Fix #6744: allow more than one metadata ingestion workflow (#6831)
* Fix #6744: allow more than one metadata ingestion workflow

* Rename to markDeletedTablesFromFilterOnly

* ui support for new field markDeletedTablesFromFilterOnly

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-08-24 14:21:52 +02:00
Ayush Shah
322f6445e1
Antlr entitylink split fix (#6901)
* Modified Antlr Split Listeners

* Modified EntityLink.g4

* Fix Grammar split
2022-08-24 14:05:21 +02:00
Mayur Singal
17349e4e30
Fix #6856: Added Troubleshooting docs for Okta SSO (#6885)
* Fix #6856: Added Troubleshooting docs for Okta SSO

* rename to service_application_client_id
2022-08-24 09:20:42 +05:30
Teddy
afd12cff04
added testResultValue to testCaseResult (#6881) 2022-08-23 18:32:15 +02:00
Mayur Singal
f32572c949
Fix #6862: Improve Superset dict handling (#6864) 2022-08-23 10:07:09 +02:00
Ayush Shah
3d125d9372
Fixes Okta default scope (#6847) 2022-08-23 13:04:12 +05:30
Onkar Ravgan
8d56cb99b7
Optimize usage queries (#6837)
* Snowflake usage query optimized

* Added docs for snowflake query

* Updated docs

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-23 12:54:34 +05:30
Pere Miquel Brull
102ea93e83
Check if table exists in datamodel rest (#6859) 2022-08-23 07:39:23 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Ayush Shah
383f4497cc
Update Entity Reference parameter fields (#6841) 2022-08-22 19:37:24 +05:30
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832)
* Implementation of temp secret for testing connection

* Fix tests
2022-08-22 08:43:23 +02:00
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805)
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
2022-08-19 16:15:40 +02:00
Mayur Singal
7c004a6785
Fix #6823: Fix hive attribute error (#6824)
* Fix #6823: Fix hive attribute error

* Empty commit to start test
2022-08-19 14:22:54 +02:00
Pere Miquel Brull
ed0a01edea
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
Fix #6787 - GCS credentials marked as required + validate privateKey (#6804)
2022-08-19 11:19:20 +02:00
Pere Miquel Brull
6f3cd4cfed
Fix #6818 - Superset code cleanup and docs (#6821)
* code cleaning

* superset docs

* Fix metabase docs
2022-08-19 11:19:06 +02:00
Teddy
78b5f8c8e2
Part 1 of #5831 -- Profiler workflow implementation (#6809)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells
2022-08-19 10:52:08 +02:00
Onkar Ravgan
1ec0fa3e37
Fixed PowerBi Exception (#6803)
* Fixed exception for chart name

* added id in exception

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-18 19:47:56 +02:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Ayush Shah
399a2a103d
Fix profiler item_type sqlalchemy datatype issue (#6794) 2022-08-18 18:30:00 +05:30
NiharDoshi99
f552960ca7
Clean Atlas, Amundsen and metadata_rest (#6649)
* Clean Atlas, Amundsen and metadata_rest

* FIX: Based on comment

* Added test case for coverage

* Fixed: Py test

* FIX: Added atlas yaml mapping file

* FIX: Py Test

* FIX: Revert Test

* FIX: Removed as per dataset
2022-08-18 16:18:07 +05:30
Sriharsha Chintalapani
0a2e2d8b18
Fix #6571: Add EntityLink for the testCase to ID columns (#6755)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* added entityLink to python tests

* Fix tests

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-18 00:29:59 -07:00
Milan Bariya
1417541a81
Dashboard Service Lineage Issue For Metabase (#6679) 2022-08-18 06:50:48 +02:00
Teddy
abaf8a84e9
Fixes #5661 by removing association between profiler and data quality (#6715)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs
2022-08-17 12:53:16 +02:00
Mayur Singal
66a934c837
Fix #4694: Refactor Source Status (#6480)
* Fix #4694: Refactor Source Status

* Using Calculate Time Decorator

* Fixed Decorator

* Fixed Decorator

* perf_counter

* Move Calculate Sucesss to Soruce

* make py_format

* Fixed Sample Data

* Fixed Pipeline, Messaging, Dashboard & MlModel Service

* Fix ZeroDivisionError
2022-08-17 14:39:50 +05:30
Nahuel
00ce67c01e
Fix#6643: Secure in secrets manager the dbtConfigSource object (#6743)
* Secure in secrets manager the dbtConfigSource object

* Add missing tests

* Fix minor bug

* Minor code refactor
2022-08-17 10:32:43 +02:00
Onkar Ravgan
424239840f
Added result limits to usage queries (#6748)
* Added result limits to queries

* Fixed formatting

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-17 08:07:39 +02:00
Pere Miquel Brull
ad95bdb9c6
Fix #6735 - Improve source validation for inner connections (#6738)
* Add inner connection validations and throw better error

* Use custom classes when retrieving metadata from Airflow to allow more versions
2022-08-16 18:47:50 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added (#6513)
* dagster-connector-added

* code-formatted

* dagster-comments-removed

* dagster-comments-removed

* dagster-unittest-added

* code-formatted-pr-comments-resolved

* improved-code-quality

* increased-py-test-time-out

* improved-dagster-tests

* improved-dagster-tests

* reverted-pytests-timeout

* dagster-pytest-improved

* added-schema-in-omdcore

* updated-schema-in-omd-core

* ui-schema-updated-omd-core

* updated-dagster-schema

* dagter-schema-test-updated
2022-08-13 15:18:14 +05:30
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568)
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-12 13:24:58 +05:30
Onkar Ravgan
af33cc0a31
Fixed Tableau ingestion issue (#6671)
* Fixed Tableau ingestion issue

* revmoved doc string

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-09 23:55:32 +05:30
Sriharsha Chintalapani
60fe344aaf
Fix #6657: Change PathParam ID type in Resources from String to UUID (#6658) 2022-08-09 05:57:39 -07:00
Milan Bariya
c38ab8c8aa
Support for AWS-delegated credentials for Elasticsearch (#6644)
* Support for AWS-delegated credentials for Elasticsearch

* FIX: Based on comments

* FIX: Based on comments

* FIX: Based on comments
2022-08-09 12:36:16 +05:30
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610)
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
2022-08-09 09:00:43 +02:00
Abhishek Pandey
77ad1b1c87
close-connection-after-each-batch-using-with (#6641) 2022-08-09 11:15:56 +05:30
Abhishek Pandey
a08201e383
multiple-database-service-names-added-in-dashboard (#6591)
* multiple-database-service-names-added-in-dashboard

* code-formatted

* code-smell-removed

* code-formatted

* improved-dbserviceName-handling-in-dashboard

* fix checks failing

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-08-08 22:21:28 +05:30
Mayur Singal
997486f1bb
Fix #6540: Added Exception handling in metadata_rest (#6650)
* Added Exception handling in metadata_rest

* Added HTTPException
2022-08-08 20:25:31 +05:30
Ayush Shah
a6db2e8a84
Fix for profiler: modified filter patterns and added error handling (#6608) 2022-08-08 10:43:17 +05:30
Milan Bariya
f3ec8c55a6
Clean col-level lineage internal logic (#6500)
* Clean col-level lineage internal logic

* Fix: The Python checkstyle failed

* Fix: SonarCloud Bugfix

* Add test

* Fix: Add multiple queries

* Fix: Based on review comments

* Fix: make py_format issue

* Fix: pytest issue

* Fix: BUG FIX
2022-08-05 20:48:28 +05:30