62 Commits

Author SHA1 Message Date
Pere Miquel Brull
b0058bdb65
Fix #9707 - Install instantclient and enable thick mode (#9761)
* Install instantclient

* Prep instant client

* Update docs

* Amend comments
2023-01-19 08:08:56 +01:00
Sriharsha Chintalapani
9702e70ea4
Minor: Delete generated directory to delete older files (#9643)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-08 21:32:16 -08:00
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test (#9363)
* Prep airflow operator integration tests

* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849)
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
2022-11-17 16:44:26 +00:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Nahuel
fbc7b245c1
Fix SonarCloud coverage XML file (#8177)
* Fix SonarCloud coverage XML file

* Add minor change on Python code

* Update coverage config

* Update coverage config

* Remove test output

* Remove sonar property

* Update Makefile
2022-10-17 17:38:19 +02:00
Pere Miquel Brull
d48fd468d4
Tag PATCH + Cleanup of helpers methods (#8150)
* cleanup

* lint

* Add tag patch

* Fix rename

* Dont kill tests
2022-10-15 14:56:30 +02:00
Nahuel
b69581299f
Fix SonarCloud coverage for Python code (#8144) 2022-10-14 15:17:40 +02:00
Ayush Shah
58765379a1
Fix Policy Tags from diff project ids (#8004) 2022-10-12 15:17:21 +05:30
Pere Miquel Brull
4b3ca065c4
Fix #3394 - Pylint fixes and add it to CI (#8084)
Fix #3394 - Pylint fixes and add it to CI (#8084)
2022-10-11 17:15:02 +02:00
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image (#8065)
Prepare Ingestion Base Docker image (#8065)
2022-10-11 07:50:49 +02:00
Pere Miquel Brull
f031293af0
Fix Pylint ingestion/bulksink and ingestion/stage (#8030)
* Pylint ingestion bulksink

* Pylint stage

* Remove test

* Remove stage
2022-10-10 14:06:43 +02:00
Sriharsha Chintalapani
656b50dd3a
Fix #7469: Refactor OpenMetadata code modules (#7474) 2022-09-14 23:14:02 -07:00
Pere Miquel Brull
15c83e3f2d
Docs - Update images and clean files (#7005)
* Update image links

* Remove external YAMLs
2022-08-29 08:59:41 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Nahuel
e321008cd0
Snyk report uploaded as HTML files (#6674) 2022-08-09 21:54:08 +02:00
Pere Miquel Brull
049077e1b5
Airflow APIs coverage (#6501)
Airflow APIs coverage (#6501)
2022-08-02 14:02:44 +02:00
Pere Miquel Brull
8ec2dbe6e4
Update vuln (#6432) 2022-07-29 13:08:12 +02:00
Pere Miquel Brull
ea918495ff
Fix #6382 - Snyk workflows (#6413)
Fix #6382 - Snyk workflows (#6413)
2022-07-28 23:06:33 +02:00
Suresh Srinivas
a23a9c84c1
Fixes #6336 - Add API to list of operations per resource for policy authoring (#6337)
* Fixes #6336 - Add API to list of operations per resource to be used for writing policies

* Fixing UI build errors

* Include only JSON schema file and not JSON data files in python code generation

* Fix generate

* Update operations list

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 12:08:01 -07:00
Pere Miquel Brull
385b7edac9
Prepare CI for Airflow APIs (#6240)
Prepare CI for Airflow APIs (#6240)
2022-07-21 12:29:57 +02:00
Nahuel
f0aab78295
Fix workflow GH for py3.9 (#6198) 2022-07-20 07:07:04 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
2022-07-19 14:51:44 +02:00
Pere Miquel Brull
0793e69752
Fix automated schema generation & Add Docs CI (#6084)
Fix automated schema generation & Add Docs CI (#6084)
2022-07-14 14:13:00 +02:00
Pere Miquel Brull
1651a6fec0
Update docs recipe (#5886)
* Update docs recipe

* Add local docs recipe
2022-07-07 16:23:39 +02:00
Pere Miquel Brull
0bb246dadb
Init docs and documentation sync CI (#5662)
* Prep docs migration

* Fix destination username
2022-06-27 15:14:04 +02:00
Teddy
5067e24374
[ISSUE-4723] Fix Snowflake Case Sensitive Error with Profiler (#5533)
* Fixed snowflake profiler + enabled profiler integration tests

* Fixed py formating
2022-06-20 22:23:17 +02:00
Pere Miquel Brull
02fece727b
Fix #5307 - Define a better topology for Ingestion Sources (#5366)
Fix #5307 - Define a better topology for Ingestion Sources (#5366)
2022-06-13 12:17:54 +02:00
Pere Miquel Brull
cb414d5768
Fix BigQuery LENGTH (#5216)
* BigQuery LENGTH

* cat coverage

* Fix coverage source
2022-05-30 16:12:45 +02:00
Pere Miquel Brull
04ede3b05b
Fix #3395 - Profiler linting (#5177)
Fix #3395 - Profiler linting (#5177)
2022-05-27 09:50:08 +02:00
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code (#5165) 2022-05-26 21:00:18 +02:00
Pere Miquel Brull
cd35447792
Fix #4688 - Python FQN with ANTRL & ES (#5073)
Fix #4688 - Python FQN with ANTRL & ES (#5073)
2022-05-23 09:51:34 +02:00
Pere Miquel Brull
d6b49f070c
Fix #3951 - OpenMetadata Airflow REST (#4034)
Fix #3951 - OpenMetadata Airflow REST (#4034)
2022-04-11 18:38:26 +02:00
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field (#3909)
* Fix sample data DAG

* Fix callback imports

* Use --set-default-enum-member in generate

* Format

* Add faulty merge hard_delete

* Fix airflow lineage, improve naming and fix lineage tests

* Add mysql url test

* Add mysql url test

* Update CI name

* Fix test ometa endpoint

* Format

* Fix metadata config
2022-04-07 18:22:50 +05:30
codingwithabhi
2006b2b8ac
bigquery-unittest-added (#3479)
* bigquery-unittest-added

* mock-added-in-create_bigquery_client

* complex-struct-added

* makefile-updated-to-install-all-packages-for-unittest

* bigquery-unittest-updated-via-file-sink
2022-03-31 14:27:51 +05:30
codingwithabhi
ccb96b3ea7
makefile-updated-to-install-all-packages-for-unittest (#3550) 2022-03-21 16:02:34 +05:30
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln (#3370)
Fix #3104 - Remove unused imports with pycln (#3370)
2022-03-14 06:59:15 +01:00
Ayush Shah
7c84f063e6
Minor Fixes - Ingestion - Datetime format, Looker fix (#3365)
* Minor Fixes - UTC timezone for usage, tag method modification, looker fix

* Modified core version

* Snowflake Usage Result limit and Makefile ignore env fix
2022-03-11 13:31:42 +05:30
Pere Miquel Brull
71207de362
Fix #2875 - Profiler API Sink (#3011)
Fix #2875 - Profiler API Sink
2022-03-02 16:46:28 +01:00
Pere Miquel Brull
01a975e414
Update py format recipes to include more paths (#3043) 2022-03-01 15:50:23 +01:00
darth-coder00
8431a23e8c
Updated make recipe for core_install_dev to use python3 (#2695) 2022-02-09 08:55:47 -08:00
Pere Miquel Brull
08a46814e6
Fix #2625: Use pytest 7.0.0 (#2640) 2022-02-06 14:41:56 -08:00
Teddy
4f3e330dd8
Issue 997-01 (#2476)
* Fix linting

* Fixed pyformat errors

* Address comments from PR review

* Added back phony in makefile

* Added back comment
2022-01-27 18:45:45 -08:00
Akash Jain
a850628593
update make core_generate to run on ubuntu as well (#2373) 2022-01-24 14:04:27 +05:30
Akash Jain
657edfeb9a
use venv instead (#2370) 2022-01-24 12:21:35 +05:30
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359)
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
2022-01-23 09:47:06 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349)
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
2022-01-22 12:08:14 -08:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085)
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
2022-01-07 10:28:38 +01:00