yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-12 19:48:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,870 Commits 675 Branches 144 Tags
Commit Graph

68 Commits

Author SHA1 Message Date
Pere Miquel Brull
7d2527bbeb
Fix #6961 - Fallback to another IP service (#7019) 
Fix #6961  - Fallback to another IP service (#7019)
 2022-09-01 06:39:12 +02:00
Shailesh Parmar
3b67cc824d
Data qaulity add test's workflow (#6957) 
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
 2022-08-29 17:26:59 +05:30
Nahuel
bdbbca0efe
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920) 
Fix#6027: Improve logging in OpenMetadata Airflow APIs (#6920)
 2022-08-26 07:29:38 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911) 
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
 2022-08-25 10:01:28 +02:00
Pere Miquel Brull
667c837ae0
Fix #5566 - Add Pipeline Service host IP endpoint (#6898) 
Fix #5566 - Add Pipeline Service host IP endpoint (#6898)
 2022-08-24 16:27:31 +02:00
Pere Miquel Brull
a975d5177d
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836) 
WIP - Fix #6582 - Validate Airflow REST versions on health endpoint (#6836)
 2022-08-23 08:59:44 +02:00
Nahuel
4e176fbc66
Fix#5917: Implementation of temp secret for testing connection (#6832) 
* Implementation of temp secret for testing connection

* Fix tests
 2022-08-22 08:43:23 +02:00
Nahuel
8691022d0f
AWS SSM secrets manager implementation on ingestion (#6805) 
* Implementation of AWS SSM as secrets manager

* Remove dead code

* Minor fixes

* Allow using default credentials in AWS client

* Fixed py style

* Fixed tests imports

* Minor changes

* Add patch to failing test

* Fix how we were storing dbtConfigSource

* Address PR comments
 2022-08-19 16:15:40 +02:00
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568) 
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-12 13:24:58 +05:30
Nahuel
66e0527a90
Fix: Profiler DAG takes loggerLevel from ingestion pipeline (#6685) 2022-08-10 17:02:16 +02:00
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610) 
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
 2022-08-09 09:00:43 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416) 
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
 2022-08-04 07:22:47 -07:00
Pere Miquel Brull
dc712e3179
Add executionDate to ingestionPipeline (#6481) 2022-08-03 08:13:55 +02:00
Pere Miquel Brull
049077e1b5
Airflow APIs coverage (#6501) 
Airflow APIs coverage (#6501)
 2022-08-02 14:02:44 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441) 
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
 2022-08-02 09:13:46 +02:00
Pere Miquel Brull
8ec2dbe6e4
Update vuln (#6432) 2022-07-29 13:08:12 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362) 
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
 2022-07-28 14:46:25 +02:00
Pere Miquel Brull
53930a9253
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288) 
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288)
 2022-07-27 07:47:25 +02:00
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager (#6330) 
* Implentation for retrieving auth provider config from Secret Manager

* Address PR comments

* Address code smells from SonarCloud

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-07-26 15:42:40 +02:00
Francisco J. Jurado Moreno
539da30ca0
Ingest triggered dags too (#6296) 2022-07-23 10:23:12 -07:00
Pere Miquel Brull
d89521cf52
Fix #6161 - kill endpoint for Airflow REST (#6244) 
Fix  #6161 - kill endpoint for Airflow REST (#6244)
 2022-07-21 19:04:02 +02:00
Pere Miquel Brull
385b7edac9
Prepare CI for Airflow APIs (#6240) 
Prepare CI for Airflow APIs (#6240)
 2022-07-21 12:29:57 +02:00
Pere Miquel Brull
36cd2933ae
Fix #5890 - Add health endpoint (#6030) 
* Add health endpoint

* Replace rest_status for health

* Update message
 2022-07-19 15:59:30 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128) 
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
 2022-07-19 14:51:44 +02:00
Pere Miquel Brull
7b6332edef
Fix #6045 - Parse gracefully the connection on Test Connection (#6149) 
* Parse gracefully test connection

* Test file renames
 2022-07-18 18:50:27 +02:00
Pere Miquel Brull
dc4579c564
Clean dag_id if ingestion pipeline name has weird characters (#6028) 2022-07-13 14:43:35 +02:00
Vj-L
011651f078
"Updated the tags with latest tag" (#5962) 
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
 2022-07-08 16:43:39 +05:30
Pere Miquel Brull
89a026b022
Fix #5688 - MlFlow deployment fails from UI (#5691) 
* Fix mlflow from UI

* format
 2022-06-28 14:58:38 +02:00
Mayur Singal
3a1dfbdbcb
Fix #5667: Fixed Build Usage Dag (#5670) 2022-06-27 09:54:20 -07:00
Pere Miquel Brull
816f8d49eb
Use source when creating and updating indexes (#5597) 2022-06-26 12:18:14 -07:00
Onkar Ravgan
4d693e40ff
removed json config from error string (#5584) 
* removed json config from error string

* Format changes

* Space formatting changes

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-06-23 09:19:19 +02:00
darth-coder00
4662fa8b6c
Fix start datetime and format for create ingestions (#5414) 
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
 2022-06-14 11:36:46 -07:00
Pere Miquel Brull
3cbd0bc1d8
Fix #5379 - Fix service type handling on REST APIs (#5380) 2022-06-08 11:36:10 -07:00
Pere Miquel Brull
8e9d0a73f6
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265) 
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
 2022-06-08 16:10:40 +02:00
Pere Miquel Brull
a57806553a
Fix #4889 - IngestionPipeline reads source connection dynamically; Fix #4888 - Remove force deploy (#5228) 2022-06-02 20:40:53 +02:00
Nahuel
628296d294
Fix #2725: Add IngestionPipeline Task logs (#5152) 2022-05-26 10:08:56 -07:00
Pere Miquel Brull
0f5777a9fa
Fix #4483 Airflow REST stage file loc; Fix #4738 disable rest_on_return (#5138) 
Fix #4483 Airflow REST stage file loc; Fix #4738 disable rest_on_return (#5138)
 2022-05-26 09:21:36 +02:00
Ayush Shah
93c32ad801
Removed Ingestion Core dependency (#5006) 2022-05-18 00:01:20 +05:30
Pere Miquel Brull
d8383fd0e5
Clean DagRun info (#4877) 2022-05-11 16:02:06 +02:00
Pere Miquel Brull
21ab0ef6f5
Fix #4886 - Overwrite dag generated configs JSON (#4887) 
* Update api docs

* Remove deployment check
 2022-05-11 15:39:30 +02:00
Pere Miquel Brull
23b95a267b
Fix #4828 - Remove lineage backend from airflow image (#4833) 
* Remove callbacks from DAG

* Remove lineage backend from image
 2022-05-11 07:36:33 +02:00
Pere Miquel Brull
ef9515bdf3
Fix #4793 - pipelineStatus for unscheduled ingestions (#4799) 
* Delete configs

* Check only None

* Format
 2022-05-09 15:24:58 +02:00
Pere Miquel Brull
3c4b582136
Revert "Configure entrypoint for airflow plugin (#4733)" (#4761) 
This reverts commit 2b1fca377d4c32010b1f2f67bc63bac2ba7ec4b0.
 2022-05-06 10:13:01 +02:00
codingwithabhi
ee1dda5366
postgres-selecting-multiple-databases-per-service (#4736) 2022-05-06 06:31:07 +02:00
Vitor Avancini
2b1fca377d
Configure entrypoint for airflow plugin (#4733) 
* Configure entrypoint for airflow plugin

* fix formatting
 2022-05-06 06:30:12 +02:00
Sriharsha Chintalapani
c0fa190c82
Update README.md 2022-05-04 19:11:02 -07:00
Pere Miquel Brull
3f8f1d1c05
Fix #4589 - Check Airflow REST status; Fix #4525 - Clean REST deps (#4606) 
* Add rest status

* Check REST status

* Clean REST deps
 2022-05-02 12:27:20 +02:00
Pere Miquel Brull
4ec9f3e117
Validate dag_id argument (#4593) 2022-04-29 09:20:29 -07:00
Pere Miquel Brull
34b90de388
Remove default logger from ingestionpipeline (#4584) 
* Remove default logger from ingestionpipeline

* Format
 2022-04-29 09:40:05 +02:00
Akash Jain
76b9ee6ee7
updated deploy docs (#4567) 2022-04-29 11:30:59 +05:30