Mayur Singal
2d25da149d
Fix #2626 : add db2 connector ( #2661 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-07 10:49:59 -08:00
Pere Miquel Brull
08a46814e6
Fix #2625 : Use pytest 7.0.0 ( #2640 )
2022-02-06 14:41:56 -08:00
Ayush Shah
21491559de
Fix #2587 : Bigquery ADC and Policy Tag implementation
2022-02-06 10:36:10 -08:00
Pere Miquel Brull
c2ad7f6373
Fix #2317 : Override Pipeline tasks from client
2022-02-06 10:17:08 -08:00
Pere Miquel Brull
6ea6f0c43d
Fix #2383 - Multiple validation workflows ( #2500 )
...
* Clean duplicated validation
* Move black dep to dev
* Fix black version for ingestion core
* Validate core generated in the same workflow
2022-01-30 12:26:41 -08:00
Pere Miquel Brull
d10639958c
Fix black version 21.12b0 ( #2499 )
2022-01-30 06:54:11 -08:00
Ayush Shah
02f3678f2a
Fixed Mysql profiler + removed additional dependency for profiler ( #2447 )
...
* Fixed Mysql profiler + removed additional dependency for profiler
* Update sql_source.py
* Update sql_source.py
* Update setup.py
2022-01-26 16:45:28 +05:30
Akash Jain
20309cde64
fix:cherry pick from 0.8.0 and miscellaneous fixes ( #2445 )
...
* updating ingestion-core version to 0.8.0
* updating ingestion version to 0.8.0
* fixing file copy path
* update docker images to point to 0.8.0
* set up new release version
* reset ingestion version to 0.9.0.dev0
* reset ingestion-core version to 0.9.0
2022-01-26 14:00:27 +05:30
Ayush Shah
b4d63bd951
Fix: Typo in setting up pyarrow dependency ( #2398 )
2022-01-25 11:58:10 +05:30
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency ( #2385 )
...
* Added pyarrow dependency
* Fix #2378
* Fix Looker
* Fix Looker, secretstr to str
2022-01-24 12:29:54 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI ( #2349 )
...
* Prepare cli module
* Check call
* Prepare backup
* Run pytest on schema changes
* Remove traces
* Add backup plugin
* run generate during CI
* Install after generate
* Address logger hotspot
2022-01-22 12:08:14 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration ( #2235 )
...
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements ( #1895 )
...
* revamped docker commands
* update metadata docker commands in comments
* Added Docker Compose Check
* code refactored to docker.py
* fix py-checkstyle lints
* Adding clean functionality; refactor code
* pylint checkstyle fixes
* version string revised
2022-01-13 20:39:46 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation ( #2016 )
...
* Reflection Cache for Bigquery and Redshift
* Overrided few sqlalchemy packages
* Added Geography Support
* Reformatted files
* DBT models error handling implemented
* Geography type added as a custom sqlalchemy datatype
* GEOGRAPHY and VARIANT added as custom sql types
* Implemented file formatting using black
* Implemented file formatting using black
2022-01-11 14:58:03 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel ( #2119 )
...
* Move staticmethods to utils
* Use functions from utils
* Convert sklearn to MlModel
* merge main
2022-01-10 09:36:08 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud ( #2085 )
...
* Update path
* Prepare sonar properties
* Prepare coverage recipes
* Add coverage
* Simplify pytest
* Organise integration tests
* Update path
* Use setup instead of reqs
* Update recipes
* Fix PR event to target
* Update event_name
* Prepare sonar
* Run tests & sonar
* Use sonarcloud host
* Fix compose
* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
c081eabb7e
Fix #1868 : Add openmetadata-ingestion-core as dependency ( #1869 )
2021-12-21 17:41:20 +01:00
Sriharsha Chintalapani
7111267669
Update version to 0.8 and ingestion to 0.6 ( #1859 )
2021-12-20 22:02:59 -08:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values ( #1830 )
...
* Prepare tests for basic config cases
* Add test cases
* Use os expandvars
* Add missing commas
* Add missing commas
* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Sriharsha Chintalapani
5a8c70bbe9
Fix #1747 : Ingestion: parse connection config method ( #1748 )
...
* Fix #1747 : Ingestion: parse connection config method
* Fix #1747 : Ingestion: parse connection config method
* Fix #1747 : Ingestion: parse connection config method
2021-12-15 00:58:57 +05:30
Ayush Shah
14982501c6
Snowflake Version Upgrade and Connection string change ( #1707 )
2021-12-12 08:26:15 -08:00
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector ( #1677 )
...
* Prepare infra
* Store experiment information in MySQL & MinIO
* Use CreateMlModelEntityRequest instead of MlModel for PUT operations
* Update MlFlow infra
* Prepare MlFlow source
* Prepare Mlflow workflow
* Simplify test and prepare README
* Revert compose
* Fix compose
* Prepare warnings and fix features
* Use non-default port for integration test
* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00
Ayush Shah
65fb71f819
Update setup.py - removed upper click version ( #1638 )
...
fix #1636
Removed click upper limit version as upgraded click package doesn't conflict if installed with openmetadata
2021-12-09 07:58:09 -08:00
Ayush Shah
67433f559c
Trino JDBC updated ( #1614 )
2021-12-08 13:14:38 +05:30
Ayush Shah
85f132738b
Remove faker from Sample-data ( #1536 )
...
* Removed Faker as a dependency
* Added ModuleNotFoundError
* Removed faker from requirements.txt
* Removed SampleData plugin from setup.py
2021-12-06 10:22:27 -08:00
Ayush Shah
341d2d5bf8
Update setup.py ( #1542 )
2021-12-03 08:56:29 -08:00
parthp2107
a9e70e6a89
License header update ( #1498 )
...
* updated liscense header
* updated license header
* updated license header
* updated license header
* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Sriharsha Chintalapani
7009a0bc12
Fix #1073 : Apache Druid Support ( #1424 )
...
* Fix #1073 : Apache Druid Support
2021-11-29 10:59:15 -08:00
Akash Jain
96c65e7ebd
fix: remove spaCy dependencies from setup.py ( #1362 )
...
* remove spaCy dependencies from setup.py
* Spacy, PII and Processor dependencies removed
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-11-24 12:23:17 -08:00
Ayush Shah
eb34d04285
Setup.py modified to support 3.10
2021-11-23 21:50:44 -08:00
Alberto Miorin
00fd7f4175
Fix #1291 : Python Inspect Code errors
2021-11-21 15:51:51 +01:00
Sriharsha Chintalapani
e4fa0247f5
Fix #1280 : Amundsen connector to import metadata ( #1281 )
...
* Fix #1280 : Amundsen connector to import metadata
2021-11-20 14:08:27 -08:00
Ayush Shah
219246b78e
Glue pagination added ( #1282 )
2021-11-20 12:46:18 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968 : Add DBT Connector ( #1200 )
2021-11-16 01:02:45 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added ( #1158 )
...
* Script modified - supports running from different locations
* Docker support from Python CLI
* Docker plugin setup.py
* Paths for latest and local dockers updated
* Resolved Comments - Docker CLI optimized, timestamp added
* help attribute added to options
* Docker clean code refactored
2021-11-12 10:30:28 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation ( #1124 )
...
* Glue Tables and Pipeline workflows implemented
* Glue Config Added
* Relative imports changed to absolute
* Resolving Comments - Changed Imports, serviceTypes
* Type fixed in setup.py
2021-11-10 07:28:13 -08:00
Tom Vijlbrief
e6f6d9c2bb
Map SQL columns with unknown type to VARCHAR ( #980 )
...
Change pydantic version requirement
2021-10-30 21:07:39 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes ( #983 )
...
* Docker fix - Architecture, MySQL
* Docker Airflow API Dagrun support
* Docker latest changes modified
2021-10-30 09:05:30 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas ( #935 )
...
* Sample Profile Data for Sample Tables (#815 )
* Sample Profile Data for Sample Tables
* Disabling Profile as Default
* Added Sample Profile Data to 3 additional sample tables
* Sample Tables fixed (#850 )
* Pydantic fix, Docker update (#860 )
* Setup.py Modified with openmetadata-airflow package, docker update
* Setup.py Modified
* Update setup.py
* Removed Pandas from Sample Data
* Sample Users added under sample data
* Sample User Standalone pipelines and modules removed
* Docker release package updated
* Dockerfile updated, removed redundant files
* Setup.py removed from ingestion src directory
* User Resource failing check resolved
* Modifying Usage Columns Datatype
2021-10-26 09:14:24 -07:00
Ayush Shah
0eb3e7b964
Changing Pydantic from 1.8.2 to 1.7.4 resolves the conflict ( #788 )
2021-10-16 07:58:55 -07:00
Ayush Shah
85b6b72848
Airflow docker ( #762 )
...
* Airflow Docker implementation - Ingestion
* Dockerfiles modified
2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
c28665bca7
Sample lineage ( #735 )
...
* Fix #727 : Add sample lineage data and ingestion support
2021-10-11 20:12:40 -07:00
Ayush Shah
1650a4ef4a
Added support for struct in bigquery, pyhive and bigquery pypi packag… ( #717 )
...
* Added support for struct in bigquery, pyhive and bigquery pypi packages modified
* Versions added, Naming changed, Newlines removed
2021-10-09 07:15:41 -07:00
James
d455409cc9
issue-696: Added trino support for Openmetadata ( #697 )
...
* issue-696: Added trino support for Openmetadata
* issue-696: fixed linting issues
* issue-696: not mentioning Trino for now as it will be part of 0.5 release
Co-authored-by: jbuoncri <jbuoncri@cisco.com>
2021-10-07 11:15:34 -07:00
Sriharsha Chintalapani
e3cfb4dc65
For hive's complex data types parse raw type ( #560 )
...
* For hive's complex data types parse raw type
* Complex Data type logic modification
* Complex Data Type parsing implemented
* Raw Data type helper modification
* handling unnamed/anonymous struct
* Complex Nested structure implementation
* print statements removed and reverted to raw_data_type
* Complex Structure Array & MAP logic implemented
* Raw Data Type Logic revamped
* Redshift Integration
* MAP and UnionType support added
* Redshift Pypi package updated
* dataLength validationError fix
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-10-04 23:36:35 +05:30
Sriharsha Chintalapani
bfec0bfbed
Ingestion: Airflow integration to ingest metadata about pipelines and tasks ( #609 )
...
* [WIP] Airlfow integration
* [WIP] Airlfow integration
* [WIP] airflow integration
* [WIP] Airflow
* [WIP] Airflow
* Fix #608 : Ingestion: Airflow integration to ingest metadata about pipelines and tasks
* Fix #608 : Ingestion: Airflow integration to ingest metadata about pipelines and tasks
* Update DashboardServiceResource.java
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-09-29 11:32:09 -07:00
Sriharsha Chintalapani
eb2717b0e3
Fix #587 : Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server ( #588 )
...
* Fix #587 : Ingestion: Add standalone report process to generate datasets, usage & profile and serve from standalone server
* add localhost
2021-09-27 08:43:38 -07:00
Ayush Shah
627481f181
Status record Json encoding bug fixed and pandas not found fixed ( #584 )
2021-09-25 13:54:04 -07:00
Sriharsha Chintalapani
745ae0c253
Fix #577 : Users API should support put op ( #578 )
...
* Fix #577 : Users API should support put op
2021-09-24 17:55:26 -07:00
