yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-30 20:59:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,692 Commits 701 Branches 149 Tags
Commit Graph

60 Commits

Author SHA1 Message Date
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568) 
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-08-12 13:24:58 +05:30
Nahuel
66e0527a90
Fix: Profiler DAG takes loggerLevel from ingestion pipeline (#6685) 2022-08-10 17:02:16 +02:00
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml (#6610) 
* Add cluster name in the app configuration and start using it to create secrets id

* Update secret manager client in openmetadata for using default auth provider

* Add missing property in test config file
 2022-08-09 09:00:43 +02:00
Sriharsha Chintalapani
1a42428e42
Add time series extention (#6416) 
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
 2022-08-04 07:22:47 -07:00
Pere Miquel Brull
dc712e3179
Add executionDate to ingestionPipeline (#6481) 2022-08-03 08:13:55 +02:00
Pere Miquel Brull
049077e1b5
Airflow APIs coverage (#6501) 
Airflow APIs coverage (#6501)
 2022-08-02 14:02:44 +02:00
Nahuel
a878aa911c
Fix#6212: Retrieve connection params from secret manager in CLI commands (#6441) 
* Retrieve connection params from secret manager for database connectors

* Retrieve connection params from secret manager for all services except database connectors

* Stop retrieving connection from SM in Airflow rest plugin

* Retrieve connection params from secret manager for dashboard services

* Retrieve connection params when initializing Workflow/ProfilerWorkflow objects

* Align services topologies + comment changes in topology runner

* Address SonarCloud bug detected

* Update database service topology

* Address PR comments

* Address PR comments

* Address PR comments
 2022-08-02 09:13:46 +02:00
Pere Miquel Brull
8ec2dbe6e4
Update vuln (#6432) 2022-07-29 13:08:12 +02:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362) 
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
 2022-07-28 14:46:25 +02:00
Pere Miquel Brull
53930a9253
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288) 
Fix #6283 #6281 #6279 - Send compressed log, backend logs and Airflow fix (#6288)
 2022-07-27 07:47:25 +02:00
Nahuel
fdfdaa14aa
Fix#5921: Implentation for retrieving auth provider config from Secret Manager (#6330) 
* Implentation for retrieving auth provider config from Secret Manager

* Address PR comments

* Address code smells from SonarCloud

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-07-26 15:42:40 +02:00
Francisco J. Jurado Moreno
539da30ca0
Ingest triggered dags too (#6296) 2022-07-23 10:23:12 -07:00
Pere Miquel Brull
d89521cf52
Fix #6161 - kill endpoint for Airflow REST (#6244) 
Fix  #6161 - kill endpoint for Airflow REST (#6244)
 2022-07-21 19:04:02 +02:00
Pere Miquel Brull
385b7edac9
Prepare CI for Airflow APIs (#6240) 
Prepare CI for Airflow APIs (#6240)
 2022-07-21 12:29:57 +02:00
Pere Miquel Brull
36cd2933ae
Fix #5890 - Add health endpoint (#6030) 
* Add health endpoint

* Replace rest_status for health

* Update message
 2022-07-19 15:59:30 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128) 
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
 2022-07-19 14:51:44 +02:00
Pere Miquel Brull
7b6332edef
Fix #6045 - Parse gracefully the connection on Test Connection (#6149) 
* Parse gracefully test connection

* Test file renames
 2022-07-18 18:50:27 +02:00
Pere Miquel Brull
dc4579c564
Clean dag_id if ingestion pipeline name has weird characters (#6028) 2022-07-13 14:43:35 +02:00
Vj-L
011651f078
"Updated the tags with latest tag" (#5962) 
Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
 2022-07-08 16:43:39 +05:30
Pere Miquel Brull
89a026b022
Fix #5688 - MlFlow deployment fails from UI (#5691) 
* Fix mlflow from UI

* format
 2022-06-28 14:58:38 +02:00
Mayur Singal
3a1dfbdbcb
Fix #5667: Fixed Build Usage Dag (#5670) 2022-06-27 09:54:20 -07:00
Pere Miquel Brull
816f8d49eb
Use source when creating and updating indexes (#5597) 2022-06-26 12:18:14 -07:00
Onkar Ravgan
4d693e40ff
removed json config from error string (#5584) 
* removed json config from error string

* Format changes

* Space formatting changes

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-06-23 09:19:19 +02:00
darth-coder00
4662fa8b6c
Fix start datetime and format for create ingestions (#5414) 
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
 2022-06-14 11:36:46 -07:00
Pere Miquel Brull
3cbd0bc1d8
Fix #5379 - Fix service type handling on REST APIs (#5380) 2022-06-08 11:36:10 -07:00
Pere Miquel Brull
8e9d0a73f6
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265) 
Fix #3573 - Sample Data refactor & ORM converter improvements (#5265)
 2022-06-08 16:10:40 +02:00
Pere Miquel Brull
a57806553a
Fix #4889 - IngestionPipeline reads source connection dynamically; Fix #4888 - Remove force deploy (#5228) 2022-06-02 20:40:53 +02:00
Nahuel
628296d294
Fix #2725: Add IngestionPipeline Task logs (#5152) 2022-05-26 10:08:56 -07:00
Pere Miquel Brull
0f5777a9fa
Fix #4483 Airflow REST stage file loc; Fix #4738 disable rest_on_return (#5138) 
Fix #4483 Airflow REST stage file loc; Fix #4738 disable rest_on_return (#5138)
 2022-05-26 09:21:36 +02:00
Ayush Shah
93c32ad801
Removed Ingestion Core dependency (#5006) 2022-05-18 00:01:20 +05:30
Pere Miquel Brull
d8383fd0e5
Clean DagRun info (#4877) 2022-05-11 16:02:06 +02:00
Pere Miquel Brull
21ab0ef6f5
Fix #4886 - Overwrite dag generated configs JSON (#4887) 
* Update api docs

* Remove deployment check
 2022-05-11 15:39:30 +02:00
Pere Miquel Brull
23b95a267b
Fix #4828 - Remove lineage backend from airflow image (#4833) 
* Remove callbacks from DAG

* Remove lineage backend from image
 2022-05-11 07:36:33 +02:00
Pere Miquel Brull
ef9515bdf3
Fix #4793 - pipelineStatus for unscheduled ingestions (#4799) 
* Delete configs

* Check only None

* Format
 2022-05-09 15:24:58 +02:00
Pere Miquel Brull
3c4b582136
Revert "Configure entrypoint for airflow plugin (#4733)" (#4761) 
This reverts commit 2b1fca377d4c32010b1f2f67bc63bac2ba7ec4b0.
 2022-05-06 10:13:01 +02:00
codingwithabhi
ee1dda5366
postgres-selecting-multiple-databases-per-service (#4736) 2022-05-06 06:31:07 +02:00
Vitor Avancini
2b1fca377d
Configure entrypoint for airflow plugin (#4733) 
* Configure entrypoint for airflow plugin

* fix formatting
 2022-05-06 06:30:12 +02:00
Sriharsha Chintalapani
c0fa190c82
Update README.md 2022-05-04 19:11:02 -07:00
Pere Miquel Brull
3f8f1d1c05
Fix #4589 - Check Airflow REST status; Fix #4525 - Clean REST deps (#4606) 
* Add rest status

* Check REST status

* Clean REST deps
 2022-05-02 12:27:20 +02:00
Pere Miquel Brull
4ec9f3e117
Validate dag_id argument (#4593) 2022-04-29 09:20:29 -07:00
Pere Miquel Brull
34b90de388
Remove default logger from ingestionpipeline (#4584) 
* Remove default logger from ingestionpipeline

* Format
 2022-04-29 09:40:05 +02:00
Akash Jain
76b9ee6ee7
updated deploy docs (#4567) 2022-04-29 11:30:59 +05:30
Pere Miquel Brull
0c51ecde63
Fix #2830 - Centralise loggers and update format (#4570) 
Fix #2830 - Centralise loggers and update format (#4570)
 2022-04-29 06:54:30 +02:00
Pere Miquel Brull
a7328ccf1b
Fix #4491 - PipelineIngestion to support debug (#4564) 2022-04-28 15:50:03 +05:30
Akash Jain
55e1be9ca4
feat: Prepare 0.11.0-snapshot version changes (#4505) 
* updated version to 0.11.0-SNAPSHOT

* updated version to 0.11.0.dev* for ingestion

* docker version updates for metadata docker cli

* OM Server Env Variables from #4197 #4261 #4473
 2022-04-27 00:41:17 +05:30
Pere Miquel Brull
87a85b085e
Fix usage source type (#4471) 2022-04-25 11:55:12 +02:00
Pere Miquel Brull
dbdd3caf1d
Fix #4332 - Athena LEN & hist concat cast + disable (#4371) 2022-04-22 20:29:59 +05:30
Mayur Singal
db0e34c709
Fixing Test Connection for Dynamo & Glue (#4316) 
* Fixing Test Connection for Dynamo

* Fixed Glue Connector

* renamed engine to connection

* Fixed the return signature

* Added dataclass
 2022-04-22 11:30:59 +05:30
Pere Miquel Brull
2444d3de3d
Fix #4235 - Run data profiler workflows from Airflow REST (#4325) 
Fix #4235 - Run data profiler workflows from Airflow REST (#4325)
 2022-04-21 17:53:29 +02:00
Pere Miquel Brull
c3993119f6
Fix #4258 - Check dag bag and discard run id search (#4264) 
Fix #4258 - Check dag bag and discard run id search (#4264)
 2022-04-20 14:55:33 +02:00