Pere Miquel Brull
a9c840778a
[issue-2116] - Add target property to MlModel ( #2134 )
...
* Add target property to MlModel
* Use pull_request_target for forks
* Simplify str
* Update ingestion-core setup
* Bump ingestion-core version
* Update install
2022-01-18 16:27:35 +01:00
Sriharsha Chintalapani
b4983764e5
Fix 2232: Presto - connection formatting issues when username/password configured ( #2233 )
...
* Fix 2232: Presto - connection formatting issues when username/password configured
* Fix 2232: Presto - connection formatting issues when username/password configured
2022-01-17 11:03:17 +05:30
Matt
276128e8b0
Add MetadataOperations RBAC support for PATCH APIs ( #2128 )
...
* Add RBAC for PATCH APIs
* Expand scope to all resources except a few (Policy, User, Role, Team, resources that doesn't support PATCH)
* Fix code smells
2022-01-16 09:24:26 -08:00
Pere Miquel Brull
ebda556adc
[issue-2223] - Lineage Callback for any operator ( #2225 )
...
* Fix logging
* Reorganise lineage
* Add callback
* Add metadata config
* Fix type smell
2022-01-15 18:57:48 +01:00
Pere Miquel Brull
2728041d0f
[issue-2213] - Airflow lineage refactor ( #2218 )
...
* Fix ometa logging
* Refactor airflow lineage
* Refactor airflow lineage
* Add airflow lineage integration tests
* Fix circular dependency
2022-01-14 13:00:33 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements ( #1895 )
...
* revamped docker commands
* update metadata docker commands in comments
* Added Docker Compose Check
* code refactored to docker.py
* fix py-checkstyle lints
* Adding clean functionality; refactor code
* pylint checkstyle fixes
* version string revised
2022-01-13 20:39:46 -08:00
codingwithabhi
5714c4122b
Move sql queries ( #2183 )
...
* sql-queries-moved-separate-file
* sql-queries-moved-separate-file
* redshift-and-neo4j-queries-added
* redshift-and-neo4j-queries-added
* neo4j-query-imported
* code-formatted
Co-authored-by: = <=>
2022-01-13 14:16:54 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack ( #2118 )
...
* ElasticSearch & Slack Event publishers
* Fix #2165 : Support for deleted entities in ElasticSearch
2022-01-12 16:35:27 -08:00
Matt
f323afe0cd
Support role ingestion from sample data ( #2167 )
2022-01-11 16:51:18 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation ( #2016 )
...
* Reflection Cache for Bigquery and Redshift
* Overrided few sqlalchemy packages
* Added Geography Support
* Reformatted files
* DBT models error handling implemented
* Geography type added as a custom sqlalchemy datatype
* GEOGRAPHY and VARIANT added as custom sql types
* Implemented file formatting using black
* Implemented file formatting using black
2022-01-11 14:58:03 +05:30
Sriharsha Chintalapani
cf6f438531
Fix #1994 : Add support for marking dataset entities as deleted ( #2141 )
...
* Fix #1994 : Add support for marking dataset entities as deleted
* Fix #1994 : Add support for marking dataset entities as deleted
* Fix #1994 : Add support for marking dataset entities as deleted
2022-01-11 11:42:53 +05:30
Pere Miquel Brull
bc05e4945d
update name ( #2138 )
2022-01-10 08:56:12 -08:00
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel ( #2119 )
...
* Move staticmethods to utils
* Use functions from utils
* Convert sklearn to MlModel
* merge main
2022-01-10 09:36:08 +01:00
Pere Miquel Brull
178315d68a
[issue-2112] - Add helper to get EntityReference in OMeta API ( #2114 )
...
* Use entity list from mixin
* Add entity reference helper
* Add tests for retrieving the entity reference
* Add missing space
* Fix shadowing
* Use get entity ref
* Use get entity ref
2022-01-10 09:07:39 +01:00
Pere Miquel Brull
74e9038b92
[issue-997] - More pylint fixes ( #2107 )
...
* Update naming and disable false positives
* Rename for naming convention
* Fix linting issues
* Moved pylint to pylintrc
* Moved pylint to pylintrc
* Rename CI
* Check sonar properties
* Fix pull_request_target checkout
2022-01-07 19:50:53 +01:00
Teddy
dbcc5e9dd2
ISSUE-1048: Add Version Mixin ( #2018 )
...
* Added entity version mixins logics to ometa API
* remove logging in line 385 used for testing
* Fixed black error + ran isort
* remove extra underscore in
* Added integration tests for OpenMetadata versions methods
* Fixed linting errors in versionMixin.py
2022-01-07 10:37:56 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud ( #2085 )
...
* Update path
* Prepare sonar properties
* Prepare coverage recipes
* Add coverage
* Simplify pytest
* Organise integration tests
* Update path
* Use setup instead of reqs
* Update recipes
* Fix PR event to target
* Update event_name
* Prepare sonar
* Run tests & sonar
* Use sonarcloud host
* Fix compose
* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
c8de136cca
First part of #2091 : Python code in branch main is always formatted correctly ( #2093 )
2022-01-07 06:17:48 +01:00
Alberto Miorin
34a9d31240
Fix #2052 : Revert the reverted pre-commit ( #2053 )
2022-01-07 03:53:21 +01:00
Pere Miquel Brull
eb01c55676
Update naming ( #2080 )
2022-01-06 14:42:50 +01:00
Ayush Shah
860f0ae8db
Iterable from Collections not working ( #2019 )
2022-01-04 06:48:18 -08:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink ( #1956 )
...
* Query Usage fixed
* Usage updated
* Reformatted change
* Usage files reformatted
2022-01-04 17:12:10 +05:30
Akash Jain
348e23ca6f
fix/ci-workflows ( #1997 )
...
* add pytest and pytest-cov
* add timeout; update make command usages
2022-01-04 12:49:22 +05:30
Ayush Shah
6175053dd4
Sample Topics fixed ( #2005 )
2022-01-03 11:00:04 -08:00
Matt
ab4c9ede25
Add ingestion for access_control_policies to support RBAC ( #1969 )
2021-12-30 08:25:09 -08:00
Ayush Shah
5d6f385a75
Added Filter Params for Table and Schema ( #1954 )
...
* Added Filter Params for table and Schema
* Bigquery Doc changes
* Doc Changes for databases
* Filter Pattern Changes
* Table Filter Pattern Example Changes
* Filter Pattern Example Changes
2021-12-29 09:13:09 -08:00
Pere Miquel Brull
1e334af89c
Fix linting ( #1958 )
2021-12-29 17:33:40 +01:00
Sriharsha Chintalapani
30eeec498f
Fix #1952 : Airflow Openmetadata lineage allow config to be read from … ( #1953 )
...
* Fix #1952 : Airflow Openmetadata lineage allow config to be read from env variable
* Fix #1952 : Airflow Openmetadata lineage allow config to be read from env variable
* Fix #1952 : Airflow Openmetadata lineage allow config to be read from env variable
* Fix #1952 : Airflow Openmetadata lineage allow config to be read from env variable
* Fix #1952 : Airflow Openmetadata lineage allow config to be read from env variable
2021-12-29 11:31:18 +05:30
Matt
a2b1d96636
Address code smells within ingestion source ( #1951 )
2021-12-28 17:40:55 -08:00
Ayush Shah
431f47fb3c
Kafka and Oracle issues fixed ( #1917 )
...
* Kafka and Oracle issues fixed
* Kafka failure status updated
2021-12-27 08:04:56 -08:00
Matt
9ff0fdc533
Refactor Glue ingestion to use common config and client ( #1925 )
...
- Use AWSClientConfigModel and AWSClient for glue.py
- Use helpers.get_storage_service_or_create for s3.py
2021-12-26 20:27:08 -08:00
Matt
f7653f4122
Add support for GCS Lifecycle Policy Ingestion ( #1923 )
...
- Add example gcs.json
- Add support for MoveAction and DeleteAction
2021-12-26 20:26:33 -08:00
Matt
4588121520
Add support for DeleteAction in S3 Lifecycle Policy ( #1916 )
...
- Support Delete Action for S3
- Add Example s3.json
- Create AWSClient util
- Use AWSClient util in S3 ingestion source
- Remove ambiguity in policy filters by removing array with different types
2021-12-25 16:29:10 -08:00
Sriharsha Chintalapani
bb847bc004
Revert "Use defaults of google java format action ( #1845 )" ( #1912 )
...
This reverts commit 782209ead58f4cd82882872e9f8f40779d8b6e54.
2021-12-23 19:22:47 -08:00
Matt
f1a8a7886e
Link Lifecycle Policy to specific Location ( #1720 )
...
- Amend ingestion to support Location reference from Policy
- Amend PolicyResource and PolicyRepository to accommodate Location reference
2021-12-23 11:06:15 -08:00
Alberto Miorin
a9e2df8993
Fix #1821 : PyHive registers the trino:// schema too ( #1835 )
2021-12-22 20:46:01 +01:00
Pere Miquel Brull
1f01081698
Clean DBT Model ( #1888 )
2021-12-22 17:17:05 +01:00
Pere Miquel Brull
d303be847e
Prepare MlModel lineage ( #1879 )
2021-12-21 17:44:29 -08:00
Alberto Miorin
782209ead5
Use defaults of google java format action ( #1845 )
...
* Use defaults of google java format action
* Fix the first Java file
* Add lint-staged
* Remove husky
2021-12-21 17:51:18 +01:00
Alberto Miorin
c081eabb7e
Fix #1868 : Add openmetadata-ingestion-core as dependency ( #1869 )
2021-12-21 17:41:20 +01:00
Sriharsha Chintalapani
7111267669
Update version to 0.8 and ingestion to 0.6 ( #1859 )
2021-12-20 22:02:59 -08:00
Alberto Miorin
ec1e3a2d61
Fix #1344 : Remove requirements.txt ( #1588 )
2021-12-20 21:36:23 -08:00
Sriharsha Chintalapani
485fdea7b1
Fix #1854 : Support DBT 1.0 format ( #1855 )
...
* Fix #1854 : Support DBT 1.0 format
2021-12-20 12:25:26 -08:00
Sriharsha Chintalapani
3c254440a5
Fix #1841 : Presto Connector issues ( #1842 )
2021-12-18 23:27:51 -08:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values ( #1830 )
...
* Prepare tests for basic config cases
* Add test cases
* Use os expandvars
* Add missing commas
* Add missing commas
* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Carsten Steckel
1bd1f99a47
fix for #1796 : hard-coded /tmp folder removed ( #1804 )
...
* fix for #1796
* fix for #1796
* missed some merge trouble artifacts
* reformat to pass black py-code-style-check
2021-12-17 18:59:17 +05:30
Ayush Shah
be1c4d4833
Stopping Docker containers and network with container name and not project name ( #1819 )
2021-12-17 18:58:54 +05:30
Sriharsha Chintalapani
df6232ef52
Fix #1808 : check for None in description ( #1809 )
2021-12-16 21:15:42 +01:00
Ayush Shah
d71c75aaa1
Update snowflake.py ( #1803 )
...
* Update snowflake.py
* snowflake needs database and warehouse
2021-12-16 10:46:49 -08:00
