yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-10-31 18:48:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
9,823 Commits 873 Branches 177 Tags
Commit Graph

2275 Commits

Author SHA1 Message Date
Pere Miquel Brull
67ac286f56
[Issue-1285] - Update ingestion package generation (#1295) 
* Remove generated sources

* Ignore python generated sources

* Keep the generated dir

* Prepare publish recipe

* Simplify recipe

* Update README with generation

* Update docs with python local setup information

* Generate python sources in the CI
 2021-11-21 08:02:31 -08:00
Alberto Miorin
00fd7f4175 Fix #1291: Python Inspect Code errors 2021-11-21 15:51:51 +01:00
Alberto Miorin
3638786162 Fix warning in pytest unit tests (use raw string for regex) 2021-11-21 08:54:22 +01:00
Sriharsha Chintalapani
e4fa0247f5
Fix #1280: Amundsen connector to import metadata (#1281) 
* Fix #1280: Amundsen connector to import metadata
 2021-11-20 14:08:27 -08:00
Ayush Shah
219246b78e
Glue pagination added (#1282) 2021-11-20 12:46:18 -08:00
Pere Miquel Brull
ea79b47c45
[Issue-1268] - MlModel generated sources (#1288) 
* Update ML model naming for generated python src

* Update MlModel id and title

* Update MlModel descriptions

* Generate pydantic models

* Update OpenMetadata with new MlModel naming

* Update MlModel naming

* Update MlModel naming

* Update MlModel namings for backend sources
 2021-11-20 08:26:45 -08:00
Sriharsha Chintalapani
c078200f7c
Fix #1276: Add entity type to ES index and aggregations (#1278) 2021-11-19 12:23:35 -08:00
Sriharsha Chintalapani
a3e9c8f844
Fix #1273: Add service_category to ES index (#1274) 
* Fix #1273: Add service_category to ES index
 2021-11-19 11:31:14 -08:00
Sriharsha Chintalapani
eea60a8696
Fix #1264: Pipelines: remove 64 character constraint in names (#1265) 
* Fix #1264: Pipelines: remove 64 character constraint in names
 2021-11-18 16:27:43 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968: Add DBT Connector (#1200) 2021-11-16 01:02:45 -08:00
Ayush Shah
2111f6c938
Docker Commands Modified (#1199) 
* Docker ingestion modified

* Command line commands updated
 2021-11-16 00:25:40 -08:00
Sriharsha Chintalapani
f018664e0c
Issue-1197: Add MariaDB Connector (#1198) 2021-11-16 00:25:24 -08:00
AviG
f5a7c193e3
ISSUE-1012 support dashboard charts of redash (widgets) (#1177) 
* ISSUE-1012 support dashboard charts of redash (widgets)

* ISSUE-1012 add comma
 2021-11-13 19:38:44 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added (#1158) 
* Script modified - supports running from different locations

* Docker support from Python CLI

* Docker plugin setup.py

* Paths for latest and local dockers updated

* Resolved Comments - Docker CLI optimized, timestamp added

* help attribute added to options

* Docker clean code refactored
 2021-11-12 10:30:28 -08:00
Sriharsha Chintalapani
ab836ac26f
Fix #1151: Refactor Model to MLModel (#1152) 
* Fix #1151: Refactor Model to MLModel

* Fix #1151: Refactor Model to MLModel

* Fix #1151: Refactor Model to MLModel

* Fix #1151: Refactor Model to MLModel
 2021-11-11 12:44:03 -08:00
Sriharsha Chintalapani
6c9202db3d
Fix #1149: Fix Airflow Lineage backend to use new APIs to publish pip… (#1150) 
* Fix #1149: Fix Airflow Lineage backend to use new APIs to publish pipeline and tasks

* Fix #1149: Fix Airflow Lineage backend to use new APIs to publish pipeline and tasks

* Fix #1149: Fix Airflow Lineage backend to use new APIs to publish pipeline and tasks
 2021-11-11 00:06:47 -08:00
Sriharsha Chintalapani
b29a6c3699
Fix #1127: Terms in descriptions for fields of struct column not searchable (#1132) 
* Fix #1127: Terms in descriptions for fields of struct column not searchable

* Fix #1127: Terms in descriptions for fields of struct column not searchable
 2021-11-10 11:33:08 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation (#1124) 
* Glue Tables and Pipeline workflows implemented

* Glue Config Added

* Relative imports changed to absolute

* Resolving Comments - Changed Imports, serviceTypes

* Type fixed in setup.py
 2021-11-10 07:28:13 -08:00
Sriharsha Chintalapani
cc539f37fd
Fix #1116: make elastic search database not include FQN (#1126) 2021-11-10 13:04:56 +05:30
Sriharsha Chintalapani
58460f3f2b
Fix #1034: workflow/bot operations API (#1053) 
* [WIP] workflow/bot operations API

* workflow/bot operations API

* workflow/bot operations API- Fix tests

* workflow/bot operations API- Fix styling issues

* workflow/bot operations API- Fix styling issues

* workflow/bot operations API- Add list runs api

* workflow/bot operations API - Fix styling issues; address reviews

* workflow/bot operations API - Fix styling issues; address reviews

* workflow/bot operations API - Fix styling issues; address reviews
 2021-11-09 20:56:45 -08:00
Pere Miquel Brull
a41e1d751e
[Issue-1119] - Generate Fix (#1120) 
* Remove additionalProperties from table definitions

* Regenerate pydantic models
 2021-11-09 15:08:21 -08:00
vijaypm
d5758a0fbb
Issue 1023 (#1088) 
* fixed bug that was setting to lowercase property names that were already in camelCase

* added datetime prefix to allow integration tests to be run repeatedly

* fixed test to use the ldap entries from ldap service in docker to test the service

* updated instructions for ldap test

* added comments to clarify the fix done in snake_to_camel method

Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
 2021-11-06 16:42:23 -07:00
Ayush Shah
bb3a7a57b8
Issue 1000 (#1083) 
* Sample Users fix #1000

* User Entity Implemented

* Teams Entity Implemented

* Added displayName Param
 2021-11-06 08:52:02 -07:00
Pere Miquel Brull
bc45c33f4d
[Issue-998] - Deprecate OpenMetadataAPIClient (#1052) 
* Refactor OpenMetadata and remove unused task functionality

* Refactor OpenMetadata in helpers

* Refactor OpenMetadata in sample_data helpers

* Remove unused client

* Remove unused client

* Remove unused client

* Align fields signature

* Refactor source OpenMetadata

* Refactor metadata_rest sink OpenMetadata

* Add list tags

* Refactor PII OpenMetadata

* Refactor LDAP REST users OpenMetadata

* isort imports

* Fix metadata to ES pipeline

* Refactor sample_entity OpenMetadata

* Remove helpers test - covered by OMeta testing

* Refactor workflow_test OpenMetadata

* Refactor mysql integration test for OpenMetadata

* Remove tasks as their refactor to pipelines

* Refactor Hive integration tests for OpenMetadata

* Remove OpenMetadataAPIClient

* Remove OpenMetadataAPIClient
 2021-11-03 13:02:34 -07:00
Tom Vijlbrief
20887c92dd
Map PostGIS spatial types to GEOGRAPHY (#1009) 
* Map PostGIS spatial types to GEOGRAPHY

* Use engine.raw_connection()
 2021-11-03 08:19:33 -07:00
Pere Miquel Brull
87fd93a3d0
Add GET with fields param (#1033) 2021-11-02 13:04:55 -07:00
Pere Miquel Brull
11f9ddae3c
Issue 954 (#1020) 
* Prepare mixins

* Remove tasks

* Prepare table mixin

* Clean tasks from sample_data

* Add table mixin tests

* Prepare compute percentile

* Refactor OpenMetadata
 2021-11-01 08:57:53 -07:00
Tom Vijlbrief
0c23b68d40
Handle . in column names (#989) 
* Handle . in column names

* Handle . for profiling
 2021-11-01 07:22:40 -07:00
Sriharsha Chintalapani
9b7d3e313f
Fix #1004: Update Pipeline Ingestion and Indexing to accommodate new API changes (#1005) 2021-11-01 07:22:01 -07:00
AviG
f105f35e58
ISSUE-995: fix redash dashboard slug (#996) 
Co-authored-by: user <user@users-MacBook-Pro.local>
 2021-10-31 11:57:41 -07:00
Pere Miquel Brull
423e81ee68
Generate python models (#990) 
* Mark internal functions

* Regenerate python models

* Prepare health check
 2021-10-31 08:23:01 -07:00
Tom Vijlbrief
e6f6d9c2bb
Map SQL columns with unknown type to VARCHAR (#980) 
Change pydantic version requirement
 2021-10-30 21:07:39 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes (#983) 
* Docker fix - Architecture, MySQL

* Docker Airflow API Dagrun support

* Docker latest changes modified
 2021-10-30 09:05:30 -07:00
Pere Miquel Brull
42db57e3b8
Issue 953 (#972) 
* Rename docstring

* Fix typo

* Prepare linage API

* Centralise GET API

* Fix delete test by FQDN

* Add hints

* Add lineage tests

* Migrate lineage ops to OpenMetadata API
 2021-10-28 11:31:38 -07:00
Sriharsha Chintalapani
7b09571a22
Profiler: Module to calculate measurements for a table (#933) 
* Profiler: Module to calculate measurements for a table

* Profiler: refactor sql expressions

* Profiler: fix supported types

* Update README.md

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2021-10-27 08:31:10 +05:30
Pere Miquel Brull
710675d51a
OpenMetadata High Level API (#942) 
* update arg name for fqdn

* fix suite setup and teardown

* Update list logic

* Add Chart tests

* Add Dashboard tests

* Add pipeline tests

* Add table tests

* Simplify create API

* Formatting tests

* Update create API

* Add Tasks and Topics tests

* Rename OpenMetadata

* Add client close

* Use Optional for GET id and name

* Fix #931 fetching dashboards using client

* Add tests

* Fix sample users ingestion

* Remove lineage test

* Rename services for independence
 2021-10-26 14:18:43 -07:00
Sriharsha Chintalapani
bb7899638f
Fix #941: Fix Topic FQDN (#943) 2021-10-26 11:15:38 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas (#935) 
* Sample Profile Data for Sample Tables (#815)

* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables

* Sample Tables fixed (#850)

* Pydantic fix, Docker update (#860)

* Setup.py Modified with openmetadata-airflow package, docker update

* Setup.py Modified

* Update setup.py

* Removed Pandas from Sample Data

* Sample Users added under sample data

* Sample User Standalone pipelines and modules removed

* Docker release package updated

* Dockerfile updated, removed redundant files

* Setup.py removed from ingestion src directory

* User Resource failing check resolved

* Modifying Usage Columns Datatype
 2021-10-26 09:14:24 -07:00
parthp2107
7b05ad7161
changed elasticsearch username and password from required to optional (#922) 2021-10-25 10:25:49 +05:30
rong fengliang
8e06fa25b6
ISSUE-861: add elasticsearch username & password (#894) 
* ISSUE-861: add elasticsearch username & password

* ISSUE-861:  python elasticsearch sink add username & password

* ISSUE-861: bugfix

* format code

* format code
 2021-10-23 10:03:24 -07:00
Ayush Shah
7bf6de0b40
Sample Usage modified as per Model (#889) 
* Sample Usage modified as per Model

* Resolved - Named Parameters
 2021-10-22 08:11:57 -07:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API (#890) 
* First approach

* Test model

* Validate endpoints

* Update schema codegen

* Prepare reqs dev

* Rename titles and codegen

* Update README

* Keep lineage title as addLineage

* Update pydantic classes

* Update API wrapper & tests

* Handle services methods

* Prepare tests
 2021-10-21 14:51:38 -07:00
Ayush Shah
13e78a74f3
Usage Pipeline fix (#883) 
* Snowflake Usage query modified

* Snowflake and Bigquery Usage pipelines added

* Update bigquery_usage.py
 2021-10-21 15:20:06 +05:30
Sriharsha Chintalapani
29c87f77d0
Improve Redshift Usage query (#881) 
* Improve Redshift Usage query

* Improve Redshift Usage query

* Improve Redshift Usage query
 2021-10-20 21:50:02 -07:00
Sriharsha Chintalapani
39a7b3e8c6
Fix #845: Fix FQN when the service is passed with Entity ID and Type only (#880) 2021-10-20 17:03:01 -07:00
Sriharsha Chintalapani
ee1ec15ef9
Fix #853: Ingestion: handle profiler returning None (#857) 
* Fix #853: Ingestion: handle profiler returning None

* Fix #853: Ingestion: handle profiler returning None
 2021-10-19 10:08:35 +05:30
Sriharsha Chintalapani
1db18a50cf
Issue-823: data profiler should have an option of configuring a profiler date (#831) 
* Issue-823: data profiler should have an option of configuring a profiler date

* Fix error logging, remove stack trace

* Issue-823: data profiler should have an option of configuring a profiler date

* Issue-823: Fix formatting
 2021-10-18 15:00:19 -07:00
Ayush Shah
9cf3a4045f
Sample Profile Data for Sample Tables (#815) 
* Sample Profile Data for Sample Tables

* Disabling Profile as Default

* Added Sample Profile Data to 3 additional sample tables
 2021-10-17 17:24:39 -07:00
Sriharsha Chintalapani
f43bf000cd
Issue-787: sql_source mark entities failed to ingest in warning (#796) 2021-10-16 22:50:36 +05:30
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample (#797) 
* Fix typo

* Prepare sample data for Model Entity

* Clean print

* Prepare sample model data ingestion

* Allow fstring in logging for style unification

* Update sink status signature

* Update sink status signature

* Return Model in REST

* Update workflow status info for readability
 2021-10-16 09:59:32 -07:00