yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-11 16:58:38 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,985 Commits 851 Branches 169 Tags
Commit Graph

44 Commits

Author SHA1 Message Date
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code (#5165) 2022-05-26 21:00:18 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125) 
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125)
 2022-05-25 08:35:16 +02:00
Pere Miquel Brull
0c51ecde63
Fix #2830 - Centralise loggers and update format (#4570) 
Fix #2830 - Centralise loggers and update format (#4570)
 2022-04-29 06:54:30 +02:00
Onkar Ravgan
148d7f47ec
Refactor DBT for schema changes (#4497) 
Refactor DBT for schema changes (#4497)
 2022-04-26 14:10:47 +02:00
Onkar Ravgan
c4aa07858d
ISSUE-2925: Added support to get dbt files via s3, gcs and http sources (#3736) 2022-04-19 19:36:24 +05:30
Pere Miquel Brull
2aecebd595
Fix #4189 - Fix airflow lineage callback (#4191) 
Fix #4189 - Fix airflow lineage callback (#4191)
 2022-04-18 15:13:26 +02:00
Pere Miquel Brull
43be01b92e
Fix #4091 & #3948 - Create services correctly (#4101) 2022-04-13 18:04:52 +02:00
Sriharsha Chintalapani
4a572713a7 Fix #3865: Service Connection & Schema Name - Fix ES indexes 2022-04-10 09:47:36 -07:00
Mayur Singal
42dcb383c0
Lineage Revamp for Schema changes (#3995) 
* Fix #3592: Refractored Lineage Ingestion
 2022-04-10 08:44:30 -07:00
Ayush Shah
3840eb77c0
Refactoring Connectors - Metabase, MariaDB, Looker (#3977) 
* Metabase Fix

* Looker, Metabase and Mariadb

* modified metabase
 2022-04-09 16:56:12 +05:30
Pere Miquel Brull
bd4071bd64
Fix #3826 & #3886 - Profiler workflow & filter pattern (#3893) 
Fix #3826 & #3886 - Profiler workflow & filter pattern (#3893)
 2022-04-06 17:05:00 +02:00
Mayur Singal
7292695bd3
Sample Data Fix (#3888) 
* Sample Data Fix
 2022-04-06 18:26:54 +05:30
Pere Miquel Brull
63533eb388
Fix for connectors based on refactoring of schemas V2 (#3870) 
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-04-05 18:33:25 -07:00
Pere Miquel Brull
b3480693e4
Fix #3824 - OMeta Schema and JSON Connections (#3861) 
Fix #3824 - OMeta Schema and JSON Connections (#3861)
 2022-04-05 21:20:39 +02:00
Pere Miquel Brull
e2539c5e83
Fix #3844 - First iteration for deprecating MetadataServerConfig (#3853) 
* Style

* deprecate MetadataServerConfig

* Remove audience from Okta
 2022-04-05 18:02:45 +05:30
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service (#3804) 
* Fix #3659 Refactor Service Connection String to be specific to per service

* Simplify and centralize Airflow Pipeline info for REST (#3740)

* Remove code

* Modified Configs based on refactoring schema (#3816)

* Clean WorkflowContext


Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-04-04 12:46:09 -07:00
Mayur Singal
a38b08882a
Fix #3566: ingesting sample data with space in name (#3586) 2022-03-22 18:29:03 +05:30
codingwithabhi
55c65b5c53
issue-3420: debug-logger-added (#3434) 2022-03-15 19:13:31 +05:30
codingwithabhi
2ef2edbf17
Bigquery lineage mapping (#2849) 
* lineage-boilerplate-code-added

* bigquery-lineage-completed

* lineage-code-optimised

* code-cleaned

* code-smell-fixed

* code-refined

* added-database-name-in-fqdn

* code-smell-removed

* exception-handled

* removed-print-statement

* formatted-setup.py

* added-sqllineage-version

* included-valid-type-none

* debug-added-in-logger

* lineage-logic-updated

* updated-lineage-params

* global-config-variables-removed

* return-type-updated-in-entity

* updated-conditionin-from-entity

* imported-sqllineage-inside-method

* code-smell-removed

* code-formatted
 2022-02-24 11:59:00 +05:30
Mayur Singal
4739aaa096
Fix #2905: Added MSSQL Usage Connector (#2948) 
* Fix #2509: added mssql usage connector

* removed comment

* removed unused variables

* fixed code smell

* renamed sql variable
 2022-02-24 01:01:03 +05:30
Mayur Singal
f9182a6244
Fix #2901: fixed error message (#2917) 
* Fix #2901: fixed error message

* moved error message to helper.py

* replaced error to http
 2022-02-23 00:20:36 +05:30
Pere Miquel Brull
76f4ccd590
Fix #2735 - Simplify Airflow properties extraction (#2749) 
* Simplify lineage properties extraction

* Add network name

* Format
 2022-02-14 16:53:42 +01:00
Pere Miquel Brull
1b2ea54d4f
Fix #2316 - Append Tasks & Add Status in Airflow (#2738) 
* Add license

* Add date to timestamp helper

* Prepare pipeline status operation

* Update and clean tasks from client

* Update tasks from client

* Check if pipeline is empty

* Keep all current pipeline info

* Append and clean tasks

* format

* Add status information

* Test pipelineStatus update

* Update task on clear

* Log status on callback

* Update lineage and status docs

* Update lineage docs

* Format

* Logic to handle DAG status

* Lint and format

* Update lineage tests
 2022-02-13 17:51:25 +01:00
Robert Pack
67a1c23482
Fix #2348: Harmonize request titles in json schemas 2022-01-31 16:29:56 -08:00
Ayush Shah
917ccd7147
Bigquery and Helpers fix - Ingestion (#2408) 
* Bigquery Ingestion fix - error handling and keyerror

* Update helpers.py and column_type_parser

* Update helpers.py
 2022-01-25 14:30:09 +05:30
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency (#2385) 
* Added pyarrow dependency

* Fix #2378

* Fix Looker

* Fix Looker, secretstr to str
 2022-01-24 12:29:54 -08:00
Sriharsha Chintalapani
10e71a7e09
Fix #2340: Ingestion pipeline deployment refactor (#2341) 
* Fix #2340: Ingestion pipeline deployment refactor
 2022-01-21 22:06:14 -08:00
Alberto Miorin
f9436c65f0
Fix #1306: Evolve Location entity integration (#1442) 2021-12-03 02:13:13 +01:00
parthp2107
a9e70e6a89
License header update (#1498) 
* updated liscense header

* updated license header

* updated license header

* updated license header

* addressing stylecheck findings
 2021-12-01 12:46:28 +05:30
Ayush Shah
219246b78e
Glue pagination added (#1282) 2021-11-20 12:46:18 -08:00
Pere Miquel Brull
bc45c33f4d
[Issue-998] - Deprecate OpenMetadataAPIClient (#1052) 
* Refactor OpenMetadata and remove unused task functionality

* Refactor OpenMetadata in helpers

* Refactor OpenMetadata in sample_data helpers

* Remove unused client

* Remove unused client

* Remove unused client

* Align fields signature

* Refactor source OpenMetadata

* Refactor metadata_rest sink OpenMetadata

* Add list tags

* Refactor PII OpenMetadata

* Refactor LDAP REST users OpenMetadata

* isort imports

* Fix metadata to ES pipeline

* Refactor sample_entity OpenMetadata

* Remove helpers test - covered by OMeta testing

* Refactor workflow_test OpenMetadata

* Refactor mysql integration test for OpenMetadata

* Remove tasks as their refactor to pipelines

* Refactor Hive integration tests for OpenMetadata

* Remove OpenMetadataAPIClient

* Remove OpenMetadataAPIClient
 2021-11-03 13:02:34 -07:00
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample (#797) 
* Fix typo

* Prepare sample data for Model Entity

* Clean print

* Prepare sample model data ingestion

* Allow fstring in logging for style unification

* Update sink status signature

* Update sink status signature

* Return Model in REST

* Update workflow status info for readability
 2021-10-16 09:59:32 -07:00
Pere Miquel Brull
9a362dafdc
[Issue 760] - Python tooling (#761) 
* Update python tooling with black

* black formatting

* Change ci config for python
 2021-10-14 07:48:42 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type (#560) 
* For hive's complex data types parse raw type

* Complex Data type logic modification

* Complex Data Type parsing implemented

* Raw Data type helper modification

* handling unnamed/anonymous struct

* Complex Nested structure implementation

* print statements removed and reverted to raw_data_type

* Complex Structure Array & MAP logic implemented

* Raw Data Type Logic revamped

* Redshift Integration

* MAP and UnionType support added

* Redshift Pypi package updated

* dataLength validationError fix

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-10-04 23:36:35 +05:30
Sriharsha Chintalapani
08cefee56a
Fix #636: Pipeline/TaskEntity capture more metadata like start, end time, type etc.. (#637) 2021-10-02 17:12:18 -07:00
Sriharsha Chintalapani
8f5c5021d7
Fix redshift complex types (#619) 2021-09-30 11:02:52 +05:30
Sriharsha Chintalapani
eff1de87ef
Fix #381: SQLAlchemy URL is causing issues in creating service (#383) 2021-09-02 20:36:18 -07:00
Suresh Srinivas
bd0c65f3e4 Fix #331: Integrate superset connector to OpenMetadata APIs 2021-08-27 21:33:42 -07:00
Suresh Srinivas
2af624a2d8 Refactor client rest apis 2021-08-24 08:37:36 -07:00
Suresh Srinivas
19151dcac7 Ingestion: Add Kafka Connector 2021-08-21 17:52:24 -07:00
Suresh Srinivas
2d70350742
Update python json schema classes; Add presto connector (#213) 
* Update python json schema classes; Add presto connector

* Modification in Postgres, pylint and presto support

Co-authored-by: Suresh Srinivas <srini3005@gmail.com>
Co-authored-by: Ayush Shah <ayush02shah12@gmail.com>
 2021-08-17 21:15:46 +05:30
Suresh Srinivas
4371e7a6f0 Fix: usage connectors 2021-08-10 15:00:41 -07:00
parthp2107
b5baa9f2db applied license to the backend files 2021-08-02 15:08:30 +05:30
sureshms
8ce7c879e8 OpenMetadata snapshot release 0.3 2021-08-01 14:27:44 -07:00