Ayush Shah
5dc3bb9297
Docker Support from Python added ( #1158 )
...
* Script modified - supports running from different locations
* Docker support from Python CLI
* Docker plugin setup.py
* Paths for latest and local dockers updated
* Resolved Comments - Docker CLI optimized, timestamp added
* help attribute added to options
* Docker clean code refactored
2021-11-12 10:30:28 -08:00
Sriharsha Chintalapani
ab836ac26f
Fix #1151 : Refactor Model to MLModel ( #1152 )
...
* Fix #1151 : Refactor Model to MLModel
* Fix #1151 : Refactor Model to MLModel
* Fix #1151 : Refactor Model to MLModel
* Fix #1151 : Refactor Model to MLModel
2021-11-11 12:44:03 -08:00
Sriharsha Chintalapani
6c9202db3d
Fix #1149 : Fix Airflow Lineage backend to use new APIs to publish pip… ( #1150 )
...
* Fix #1149 : Fix Airflow Lineage backend to use new APIs to publish pipeline and tasks
* Fix #1149 : Fix Airflow Lineage backend to use new APIs to publish pipeline and tasks
* Fix #1149 : Fix Airflow Lineage backend to use new APIs to publish pipeline and tasks
2021-11-11 00:06:47 -08:00
Sriharsha Chintalapani
b29a6c3699
Fix #1127 : Terms in descriptions for fields of struct column not searchable ( #1132 )
...
* Fix #1127 : Terms in descriptions for fields of struct column not searchable
* Fix #1127 : Terms in descriptions for fields of struct column not searchable
2021-11-10 11:33:08 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation ( #1124 )
...
* Glue Tables and Pipeline workflows implemented
* Glue Config Added
* Relative imports changed to absolute
* Resolving Comments - Changed Imports, serviceTypes
* Type fixed in setup.py
2021-11-10 07:28:13 -08:00
Sriharsha Chintalapani
cc539f37fd
Fix #1116 : make elastic search database not include FQN ( #1126 )
2021-11-10 13:04:56 +05:30
Sriharsha Chintalapani
58460f3f2b
Fix #1034 : workflow/bot operations API ( #1053 )
...
* [WIP] workflow/bot operations API
* workflow/bot operations API
* workflow/bot operations API- Fix tests
* workflow/bot operations API- Fix styling issues
* workflow/bot operations API- Fix styling issues
* workflow/bot operations API- Add list runs api
* workflow/bot operations API - Fix styling issues; address reviews
* workflow/bot operations API - Fix styling issues; address reviews
* workflow/bot operations API - Fix styling issues; address reviews
2021-11-09 20:56:45 -08:00
Pere Miquel Brull
a41e1d751e
[Issue-1119] - Generate Fix ( #1120 )
...
* Remove additionalProperties from table definitions
* Regenerate pydantic models
2021-11-09 15:08:21 -08:00
vijaypm
d5758a0fbb
Issue 1023 ( #1088 )
...
* fixed bug that was setting to lowercase property names that were already in camelCase
* added datetime prefix to allow integration tests to be run repeatedly
* fixed test to use the ldap entries from ldap service in docker to test the service
* updated instructions for ldap test
* added comments to clarify the fix done in snake_to_camel method
Co-authored-by: Vijay Mariadassou <vijay@mariadassou.com>
2021-11-06 16:42:23 -07:00
Ayush Shah
bb3a7a57b8
Issue 1000 ( #1083 )
...
* Sample Users fix #1000
* User Entity Implemented
* Teams Entity Implemented
* Added displayName Param
2021-11-06 08:52:02 -07:00
Pere Miquel Brull
bc45c33f4d
[Issue-998] - Deprecate OpenMetadataAPIClient ( #1052 )
...
* Refactor OpenMetadata and remove unused task functionality
* Refactor OpenMetadata in helpers
* Refactor OpenMetadata in sample_data helpers
* Remove unused client
* Remove unused client
* Remove unused client
* Align fields signature
* Refactor source OpenMetadata
* Refactor metadata_rest sink OpenMetadata
* Add list tags
* Refactor PII OpenMetadata
* Refactor LDAP REST users OpenMetadata
* isort imports
* Fix metadata to ES pipeline
* Refactor sample_entity OpenMetadata
* Remove helpers test - covered by OMeta testing
* Refactor workflow_test OpenMetadata
* Refactor mysql integration test for OpenMetadata
* Remove tasks as their refactor to pipelines
* Refactor Hive integration tests for OpenMetadata
* Remove OpenMetadataAPIClient
* Remove OpenMetadataAPIClient
2021-11-03 13:02:34 -07:00
Tom Vijlbrief
20887c92dd
Map PostGIS spatial types to GEOGRAPHY ( #1009 )
...
* Map PostGIS spatial types to GEOGRAPHY
* Use engine.raw_connection()
2021-11-03 08:19:33 -07:00
Pere Miquel Brull
87fd93a3d0
Add GET with fields param ( #1033 )
2021-11-02 13:04:55 -07:00
Pere Miquel Brull
11f9ddae3c
Issue 954 ( #1020 )
...
* Prepare mixins
* Remove tasks
* Prepare table mixin
* Clean tasks from sample_data
* Add table mixin tests
* Prepare compute percentile
* Refactor OpenMetadata
2021-11-01 08:57:53 -07:00
Tom Vijlbrief
0c23b68d40
Handle . in column names ( #989 )
...
* Handle . in column names
* Handle . for profiling
2021-11-01 07:22:40 -07:00
Sriharsha Chintalapani
9b7d3e313f
Fix #1004 : Update Pipeline Ingestion and Indexing to accommodate new API changes ( #1005 )
2021-11-01 07:22:01 -07:00
AviG
f105f35e58
ISSUE-995: fix redash dashboard slug ( #996 )
...
Co-authored-by: user <user@users-MacBook-Pro.local>
2021-10-31 11:57:41 -07:00
Pere Miquel Brull
423e81ee68
Generate python models ( #990 )
...
* Mark internal functions
* Regenerate python models
* Prepare health check
2021-10-31 08:23:01 -07:00
Tom Vijlbrief
e6f6d9c2bb
Map SQL columns with unknown type to VARCHAR ( #980 )
...
Change pydantic version requirement
2021-10-30 21:07:39 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes ( #983 )
...
* Docker fix - Architecture, MySQL
* Docker Airflow API Dagrun support
* Docker latest changes modified
2021-10-30 09:05:30 -07:00
Pere Miquel Brull
42db57e3b8
Issue 953 ( #972 )
...
* Rename docstring
* Fix typo
* Prepare linage API
* Centralise GET API
* Fix delete test by FQDN
* Add hints
* Add lineage tests
* Migrate lineage ops to OpenMetadata API
2021-10-28 11:31:38 -07:00
Sriharsha Chintalapani
7b09571a22
Profiler: Module to calculate measurements for a table ( #933 )
...
* Profiler: Module to calculate measurements for a table
* Profiler: refactor sql expressions
* Profiler: fix supported types
* Update README.md
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-27 08:31:10 +05:30
Pere Miquel Brull
710675d51a
OpenMetadata High Level API ( #942 )
...
* update arg name for fqdn
* fix suite setup and teardown
* Update list logic
* Add Chart tests
* Add Dashboard tests
* Add pipeline tests
* Add table tests
* Simplify create API
* Formatting tests
* Update create API
* Add Tasks and Topics tests
* Rename OpenMetadata
* Add client close
* Use Optional for GET id and name
* Fix #931 fetching dashboards using client
* Add tests
* Fix sample users ingestion
* Remove lineage test
* Rename services for independence
2021-10-26 14:18:43 -07:00
Sriharsha Chintalapani
bb7899638f
Fix #941 : Fix Topic FQDN ( #943 )
2021-10-26 11:15:38 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas ( #935 )
...
* Sample Profile Data for Sample Tables (#815 )
* Sample Profile Data for Sample Tables
* Disabling Profile as Default
* Added Sample Profile Data to 3 additional sample tables
* Sample Tables fixed (#850 )
* Pydantic fix, Docker update (#860 )
* Setup.py Modified with openmetadata-airflow package, docker update
* Setup.py Modified
* Update setup.py
* Removed Pandas from Sample Data
* Sample Users added under sample data
* Sample User Standalone pipelines and modules removed
* Docker release package updated
* Dockerfile updated, removed redundant files
* Setup.py removed from ingestion src directory
* User Resource failing check resolved
* Modifying Usage Columns Datatype
2021-10-26 09:14:24 -07:00
parthp2107
7b05ad7161
changed elasticsearch username and password from required to optional ( #922 )
2021-10-25 10:25:49 +05:30
rong fengliang
8e06fa25b6
ISSUE-861: add elasticsearch username & password ( #894 )
...
* ISSUE-861: add elasticsearch username & password
* ISSUE-861: python elasticsearch sink add username & password
* ISSUE-861: bugfix
* format code
* format code
2021-10-23 10:03:24 -07:00
Ayush Shah
7bf6de0b40
Sample Usage modified as per Model ( #889 )
...
* Sample Usage modified as per Model
* Resolved - Named Parameters
2021-10-22 08:11:57 -07:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API ( #890 )
...
* First approach
* Test model
* Validate endpoints
* Update schema codegen
* Prepare reqs dev
* Rename titles and codegen
* Update README
* Keep lineage title as addLineage
* Update pydantic classes
* Update API wrapper & tests
* Handle services methods
* Prepare tests
2021-10-21 14:51:38 -07:00
Ayush Shah
13e78a74f3
Usage Pipeline fix ( #883 )
...
* Snowflake Usage query modified
* Snowflake and Bigquery Usage pipelines added
* Update bigquery_usage.py
2021-10-21 15:20:06 +05:30
Sriharsha Chintalapani
29c87f77d0
Improve Redshift Usage query ( #881 )
...
* Improve Redshift Usage query
* Improve Redshift Usage query
* Improve Redshift Usage query
2021-10-20 21:50:02 -07:00
Sriharsha Chintalapani
39a7b3e8c6
Fix #845 : Fix FQN when the service is passed with Entity ID and Type only ( #880 )
2021-10-20 17:03:01 -07:00
Sriharsha Chintalapani
ee1ec15ef9
Fix #853 : Ingestion: handle profiler returning None ( #857 )
...
* Fix #853 : Ingestion: handle profiler returning None
* Fix #853 : Ingestion: handle profiler returning None
2021-10-19 10:08:35 +05:30
Sriharsha Chintalapani
1db18a50cf
Issue-823: data profiler should have an option of configuring a profiler date ( #831 )
...
* Issue-823: data profiler should have an option of configuring a profiler date
* Fix error logging, remove stack trace
* Issue-823: data profiler should have an option of configuring a profiler date
* Issue-823: Fix formatting
2021-10-18 15:00:19 -07:00
Ayush Shah
9cf3a4045f
Sample Profile Data for Sample Tables ( #815 )
...
* Sample Profile Data for Sample Tables
* Disabling Profile as Default
* Added Sample Profile Data to 3 additional sample tables
2021-10-17 17:24:39 -07:00
Sriharsha Chintalapani
f43bf000cd
Issue-787: sql_source mark entities failed to ingest in warning ( #796 )
2021-10-16 22:50:36 +05:30
Pere Miquel Brull
3bf5999224
[Issue-759] - Model Entity Sample ( #797 )
...
* Fix typo
* Prepare sample data for Model Entity
* Clean print
* Prepare sample model data ingestion
* Allow fstring in logging for style unification
* Update sink status signature
* Update sink status signature
* Return Model in REST
* Update workflow status info for readability
2021-10-16 09:59:32 -07:00
Pere Miquel Brull
1804de6d73
[Issue-760] - Precommit & isort ( #772 )
...
* Update configs
* Update configs
* Update precommit black exclude
* Update precommit exclude isort
* Test precommit
* Revert test commit
* isort & black
* Test precommit
* Revert test precommit
* Update docs
* Apply recipe update
* Fix black_check recipe
* Grammar
2021-10-14 15:22:59 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs ( #773 )
...
* Removing Cron from all configs
* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Pere Miquel Brull
9a362dafdc
[Issue 760] - Python tooling ( #761 )
...
* Update python tooling with black
* black formatting
* Change ci config for python
2021-10-14 07:48:42 -07:00
Sriharsha Chintalapani
5644addc9a
Fix #748 : Airflow lineage add inlet , outlet parsing to construct lineage ( #749 )
2021-10-13 09:33:15 +05:30
Sriharsha Chintalapani
c28665bca7
Sample lineage ( #735 )
...
* Fix #727 : Add sample lineage data and ingestion support
2021-10-11 20:12:40 -07:00
Matt
78e97c81ca
Add exponentional backoff for ometa REST Client ( #711 )
...
Co-authored-by: Mithun Mathew <matt@Mithuns-MacBook-Pro.local>
2021-10-11 11:20:36 -07:00
Ayush Shah
323513755f
register_custom_type import path changed ( #722 )
2021-10-09 11:06:25 -07:00
Ayush Shah
1650a4ef4a
Added support for struct in bigquery, pyhive and bigquery pypi packag… ( #717 )
...
* Added support for struct in bigquery, pyhive and bigquery pypi packages modified
* Versions added, Naming changed, Newlines removed
2021-10-09 07:15:41 -07:00
Ayush Shah
7ef490ed2d
Sample Data set to generate proper data for Currency column ( #713 )
2021-10-09 07:15:09 -07:00
Ayush Shah
a19c5bceca
Trino fix #705 ( #706 )
...
* Trino fix #705
* Update trino.json to point to default database
2021-10-07 12:25:03 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata ( #697 )
...
* issue-696: Added trino support for Openmetadata
* issue-696: fixed linting issues
* issue-696: not mentioning Trino for now as it will be part of 0.5 release
Co-authored-by: jbuoncri <jbuoncri@cisco.com>
2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type ( #560 )
...
* For hive's complex data types parse raw type
* Complex Data type logic modification
* Complex Data Type parsing implemented
* Raw Data type helper modification
* handling unnamed/anonymous struct
* Complex Nested structure implementation
* print statements removed and reverted to raw_data_type
* Complex Structure Array & MAP logic implemented
* Raw Data Type Logic revamped
* Redshift Integration
* MAP and UnionType support added
* Redshift Pypi package updated
* dataLength validationError fix
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 23:36:35 +05:30
Sriharsha Chintalapani
1361bace11
sample entity generation ( #646 )
...
* 1M entities to test scalability
* 1M entities to test scalability
* 1M entities test
* column name modification
* Fix service add and tags
* Add sample entity generation
Co-authored-by: parthp2107 <parth@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 22:56:28 +05:30
