412 Commits

Author SHA1 Message Date
Mayur Singal
939adf887b
Fix #8973: Superset fetch metadata from DB (#9645)
* Docs #8973: Superset with SSO docs

* Superset support for mysql & postgres db

* remove unwanted field from yaml

* Fix pylint

* Fix superset tests

* Fix sample data connection

* ci fix & review comments

* cypress & defualt provider fix
2023-01-19 16:39:43 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package (#9521)
Fix #6562 - Sources have their own package (#9521)
2022-12-27 15:00:22 +01:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers (#9513)
Organise dynamic importers (#9513)
2022-12-23 12:17:06 +01:00
Sriharsha Chintalapani
b9242c1390
Fix topic indexes and migrations (#9439)
* Fix topic indexing

* Fix topic db migrations

* Fix topic db migrations

* Fix topic db migrations

* Fix test; add missing topic indexes on python side; fix ranking for tables using usage count
2022-12-21 10:33:12 +01:00
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test (#9363)
* Prep airflow operator integration tests

* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Onkar Ravgan
b539b299ee
Integrated schema parsers (#9305)
* Integrated schema parsers

* Addressed review comments

* fixed pytests
2022-12-15 16:54:55 +05:30
Sriharsha Chintalapani
38074f763b
Fix #8509: Add schema fields (#9209)
* Fix #8509: Add schema fields

* Fix #4675: Ingestion deployment from UI is broken

* Added sample data for topics

* Fixed pytests

* Address comments

* Refactored sampledata according to new schema

* Added return type

* Feat(ui)  : Add support for Avro editor (#9224)

* Feat(ui)  : Add support for Avro editor

* chore : add mock data for nested fields

* style: add group and opacity css

* add schema fields component to topic details and unit test

* Add locale keys

* Add support for edit description.

* refactor: expandableConfig

* test: add unit test for util method

* chore : make changes according to the schema change

* test : fix unit test and add new mock data

* chore : rename files

* Add row key

* chore : add default value for tags

* chore : update util method

* chore : add support for editing field tags

* chore : rename util files

* test : add unit test

* add comments

* addressing comments

* Address comments

* Added avro requirments

* Added requirement versions

* fixed versions

* protobuf version fix

* chore : rename util test file

* Fixed Dataype

* test: add unit test for schema component

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-14 19:56:37 +05:30
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state (#9206) 2022-12-10 19:54:41 +01:00
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159)
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Onkar Ravgan
23468fb868
Fixed Broken docs links (#9125) 2022-12-02 15:42:06 +01:00
Pere Miquel Brull
83ef3315c3
Fix query log parser (#8920) 2022-11-21 12:55:55 +00:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Mayur Singal
18cd1c99ca
Add queries in sample data for lineage (#8674) 2022-11-14 13:11:16 +05:30
Shailesh Parmar
0f0489a7d4
Feat: Added pipeline as edge from ui (#8503)
* Feat: Added pipeline as edge from ui

* fixed failing unit test

* removed pipeline

* added unit test for util function
2022-11-03 17:46:49 +05:30
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image (#8065)
Prepare Ingestion Base Docker image (#8065)
2022-10-11 07:50:49 +02:00
Sriharsha Chintalapani
612c083612
Fix Doc links (#7734)
* Fix Broken links

* Fix symlink

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-09-28 14:05:51 -07:00
NiharDoshi99
d9f3ab23d7
Fixed: SampleData Teams and teamTypes (#7682) 2022-09-26 18:22:41 +05:30
mohitdeuex
cceff48f0d
Default token (#7695) 2022-09-23 20:03:54 +05:30
Teddy
e22036cc09
Fixes #7504 Added migration steps to delete profiler pipeline from entity relationship (#7511)
* Added migration steps to delete profiler pipeline from entity relationship

* Added migration to remove rows from entity_relationship with profiler pipelines

* Fixed sample data TestDefinition name after renaming testDefinition

* Added user_tokens table to v005

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2022-09-20 08:14:29 +02:00
Nahuel
2a6c6134f4
Fix#7272: Improve logging when initializing workflow from CLI (#7522)
* Improve logging when initializing workflow from CLI

* Fix broken tests
2022-09-19 08:00:00 -07:00
Akash Jain
fc8312c5fb
feat: Refactor openmetadata/ingestion docker image (#7456)
* centralize openmetadata/ingestion docker image

* update volume mappings; fix ingestion tag!

* fix PR comments

* fix cypress mysql tests!
2022-09-19 09:20:54 +05:30
Nahuel
6f196a5c12
Added workflow config for redpanda (#7170) 2022-09-03 18:20:09 +02:00
Nahuel
ed58c27f76
Remove unused property extra_hosts from docker-compose files (#7096)
* Remove unused properties from docker-compose files

* Minor fix in run_local_docker script
2022-08-31 21:30:24 +02:00
Ashish Gupta
b59878e1b5
Fix (#6984) doc links in OpenMetaData (#6986)
* Fix doc links in OpenMetaData

* Link updated

* remove unwanted file

* changes as per comments

* Changes as per comments
2022-08-29 19:38:54 +05:30
Abhishek Pandey
fde53809eb
added-query-log-parser-in-postgres (#6750)
added-query-log-parser-in-postgres (#6750)
2022-08-29 06:48:29 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Teddy
afd12cff04
added testResultValue to testCaseResult (#6881) 2022-08-23 18:32:15 +02:00
Sriharsha Chintalapani
821d70eae4
Fix #6782: Separate TableProfile and ColumnProfile api calls (#6783)
* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6571: Add EntityLink for the testCase to ID columns

* Fix #6782: Separate TableProfile and ColumnProfile api calls

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix #6782: Separate TableProfile and ColumnProfile api calls - fix tests

* Fix setFields

* Fix tests

* Update pipeline status endpoint

* updated ui side as per new schema for profiler tab

* updated profiler details with new API

* Fix Profiler tests and validation errors (#6827)

* add profilerSample field in TableProfile

* add profilerSample field in TableProfile

* get columnProfile with field profile

* get columnProfile with field profile

* Fixed sample data and python tests

* fixed date range filter change issue

* handled empty profiler case

* Added column level test case and results

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-22 21:31:24 +05:30
Teddy
78b5f8c8e2
Part 1 of #5831 -- Profiler workflow implementation (#6809)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells
2022-08-19 10:52:08 +02:00
Onkar Ravgan
424239840f
Added result limits to usage queries (#6748)
* Added result limits to queries

* Fixed formatting

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-17 08:07:39 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added (#6513)
* dagster-connector-added

* code-formatted

* dagster-comments-removed

* dagster-comments-removed

* dagster-unittest-added

* code-formatted-pr-comments-resolved

* improved-code-quality

* increased-py-test-time-out

* improved-dagster-tests

* improved-dagster-tests

* reverted-pytests-timeout

* dagster-pytest-improved

* added-schema-in-omdcore

* updated-schema-in-omd-core

* ui-schema-updated-omd-core

* updated-dagster-schema

* dagter-schema-test-updated
2022-08-13 15:18:14 +05:30
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568)
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-12 13:24:58 +05:30
Shailesh Parmar
455c2d039e
(Feat) UI: Add Profiler Tab as per new mock (#6627)
* add initial setup and added profiler tab

* added unit test for tableprofiler component

* added more coverage for table profiler component

* added get and put request and functionality for profiler setting

* added unit test for new components

* miner fix

* added data-testid for profiler settings component

* changed input box to slider for profile sample

* updated setting model design

* replaced select with treeSelect

* added unit test for setting modal

* fixing flaky cypress related to resizeObserver loop limit

* fixed failed unit test and added selected sample profiler text

* addressing comments
2022-08-10 14:46:50 +05:30
Onkar Ravgan
af33cc0a31
Fixed Tableau ingestion issue (#6671)
* Fixed Tableau ingestion issue

* revmoved doc string

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-09 23:55:32 +05:30
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610)
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
2022-08-09 09:00:43 +02:00
Abhishek Pandey
a08201e383
multiple-database-service-names-added-in-dashboard (#6591)
* multiple-database-service-names-added-in-dashboard

* code-formatted

* code-smell-removed

* code-formatted

* improved-dbserviceName-handling-in-dashboard

* fix checks failing

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-08-08 22:21:28 +05:30
Ayush Shah
a6db2e8a84
Fix for profiler: modified filter patterns and added error handling (#6608) 2022-08-08 10:43:17 +05:30
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975: Oracle Connection Enhancements (#6579)
* Fix #6530 & Fix #4975: Oracle Connection Enhancements

* updated migrations & docs
2022-08-05 10:28:04 +05:30
Teddy
b37bca5471
Fixes #2954 -- Ingest sample data for profile, test suite and co. (#6586)
Fixes #2954 -- Ingest sample data for profile, test suite and co. (#6586)
2022-08-04 19:44:21 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416)
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Milan Bariya
45e6c2c7d7
Sample Data Ingestion use Create request (#6546)
* Sample Data Ingestion use Create request

* Fix: Code smell

* Fix: make_pyformat

* Fix: Changed based on comments
2022-08-04 11:13:48 +02:00
Mayur Singal
bafca3b7b6
Fix #6387: Clean Ingestion (#6405)
* Fix #6387: Clean Ingestion

* postgres fix

* Fixed Location in Sample Data
2022-07-29 08:51:58 +02:00
Mayur Singal
c6efe8c142
Fix #6158: Fivetran Connector Added (#6329)
* Fivetran Connector Added

* Fivetran unit test added

* Fixed Unit Tests

* Added Fivetran Docs & Added Fivetran in Pipeline Listing
2022-07-27 20:50:56 +05:30
Abhishek Pandey
f125c597a9
Atlas connector fix (#4112)
* refactored-atlas-schema

* atlas-source-connector-fixed

* code-formatted

* updated-service-creation

* code-formatted

* atlas-json-updated

* atlas-removed-from-database-service

* atlas-fqn-and-execution-flow-updated

* atlas-connector-fixed

* atlas--amundsen-removed-from-db-service

* code-cleanup

* removed-unused-codes

* code-smelled

* code-smell-removed
2022-07-25 11:23:36 +05:30
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
2022-07-19 14:51:44 +02:00
Shailesh Parmar
04db051197
Revert "Pipeline lineage edge UI (#5891)" (#6159)
This reverts commit 4d898a120fd4410a8ed660db252bc270710e9323.
2022-07-19 18:07:46 +05:30
Onkar Ravgan
4d898a120f
Pipeline lineage edge UI (#5891)
Pipeline lineage edge UI (#5891)
2022-07-17 18:25:37 +02:00
Milan Bariya
187de08d1f
dbServiceName field move from serviceConnection to sourceConfig (#6047) 2022-07-14 11:25:31 +05:30
Pere Miquel Brull
a6dd100be4
Fix metadata es DAG (#5933) 2022-07-07 12:15:09 -07:00
Pere Miquel Brull
9ca7a75e2d
Add col lineage in sample usage (#5790) 2022-06-30 12:16:06 +02:00