48 Commits

Author SHA1 Message Date
Pere Miquel Brull
15c83e3f2d
Docs - Update images and clean files (#7005)
* Update image links

* Remove external YAMLs
2022-08-29 08:59:41 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Nahuel
e321008cd0
Snyk report uploaded as HTML files (#6674) 2022-08-09 21:54:08 +02:00
Pere Miquel Brull
049077e1b5
Airflow APIs coverage (#6501)
Airflow APIs coverage (#6501)
2022-08-02 14:02:44 +02:00
Pere Miquel Brull
8ec2dbe6e4
Update vuln (#6432) 2022-07-29 13:08:12 +02:00
Pere Miquel Brull
ea918495ff
Fix #6382 - Snyk workflows (#6413)
Fix #6382 - Snyk workflows (#6413)
2022-07-28 23:06:33 +02:00
Suresh Srinivas
a23a9c84c1
Fixes #6336 - Add API to list of operations per resource for policy authoring (#6337)
* Fixes #6336 - Add API to list of operations per resource to be used for writing policies

* Fixing UI build errors

* Include only JSON schema file and not JSON data files in python code generation

* Fix generate

* Update operations list

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 12:08:01 -07:00
Pere Miquel Brull
385b7edac9
Prepare CI for Airflow APIs (#6240)
Prepare CI for Airflow APIs (#6240)
2022-07-21 12:29:57 +02:00
Nahuel
f0aab78295
Fix workflow GH for py3.9 (#6198) 2022-07-20 07:07:04 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
2022-07-19 14:51:44 +02:00
Pere Miquel Brull
0793e69752
Fix automated schema generation & Add Docs CI (#6084)
Fix automated schema generation & Add Docs CI (#6084)
2022-07-14 14:13:00 +02:00
Pere Miquel Brull
1651a6fec0
Update docs recipe (#5886)
* Update docs recipe

* Add local docs recipe
2022-07-07 16:23:39 +02:00
Pere Miquel Brull
0bb246dadb
Init docs and documentation sync CI (#5662)
* Prep docs migration

* Fix destination username
2022-06-27 15:14:04 +02:00
Teddy
5067e24374
[ISSUE-4723] Fix Snowflake Case Sensitive Error with Profiler (#5533)
* Fixed snowflake profiler + enabled profiler integration tests

* Fixed py formating
2022-06-20 22:23:17 +02:00
Pere Miquel Brull
02fece727b
Fix #5307 - Define a better topology for Ingestion Sources (#5366)
Fix #5307 - Define a better topology for Ingestion Sources (#5366)
2022-06-13 12:17:54 +02:00
Pere Miquel Brull
cb414d5768
Fix BigQuery LENGTH (#5216)
* BigQuery LENGTH

* cat coverage

* Fix coverage source
2022-05-30 16:12:45 +02:00
Pere Miquel Brull
04ede3b05b
Fix #3395 - Profiler linting (#5177)
Fix #3395 - Profiler linting (#5177)
2022-05-27 09:50:08 +02:00
Pere Miquel Brull
02c2dd9494
Fix #4689 - Rename FQDN to FQN and clean unused code (#5165) 2022-05-26 21:00:18 +02:00
Pere Miquel Brull
cd35447792
Fix #4688 - Python FQN with ANTRL & ES (#5073)
Fix #4688 - Python FQN with ANTRL & ES (#5073)
2022-05-23 09:51:34 +02:00
Pere Miquel Brull
d6b49f070c
Fix #3951 - OpenMetadata Airflow REST (#4034)
Fix #3951 - OpenMetadata Airflow REST (#4034)
2022-04-11 18:38:26 +02:00
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field (#3909)
* Fix sample data DAG

* Fix callback imports

* Use --set-default-enum-member in generate

* Format

* Add faulty merge hard_delete

* Fix airflow lineage, improve naming and fix lineage tests

* Add mysql url test

* Add mysql url test

* Update CI name

* Fix test ometa endpoint

* Format

* Fix metadata config
2022-04-07 18:22:50 +05:30
codingwithabhi
2006b2b8ac
bigquery-unittest-added (#3479)
* bigquery-unittest-added

* mock-added-in-create_bigquery_client

* complex-struct-added

* makefile-updated-to-install-all-packages-for-unittest

* bigquery-unittest-updated-via-file-sink
2022-03-31 14:27:51 +05:30
codingwithabhi
ccb96b3ea7
makefile-updated-to-install-all-packages-for-unittest (#3550) 2022-03-21 16:02:34 +05:30
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln (#3370)
Fix #3104 - Remove unused imports with pycln (#3370)
2022-03-14 06:59:15 +01:00
Ayush Shah
7c84f063e6
Minor Fixes - Ingestion - Datetime format, Looker fix (#3365)
* Minor Fixes - UTC timezone for usage, tag method modification, looker fix

* Modified core version

* Snowflake Usage Result limit and Makefile ignore env fix
2022-03-11 13:31:42 +05:30
Pere Miquel Brull
71207de362
Fix #2875 - Profiler API Sink (#3011)
Fix #2875 - Profiler API Sink
2022-03-02 16:46:28 +01:00
Pere Miquel Brull
01a975e414
Update py format recipes to include more paths (#3043) 2022-03-01 15:50:23 +01:00
darth-coder00
8431a23e8c
Updated make recipe for core_install_dev to use python3 (#2695) 2022-02-09 08:55:47 -08:00
Pere Miquel Brull
08a46814e6
Fix #2625: Use pytest 7.0.0 (#2640) 2022-02-06 14:41:56 -08:00
Teddy
4f3e330dd8
Issue 997-01 (#2476)
* Fix linting

* Fixed pyformat errors

* Address comments from PR review

* Added back phony in makefile

* Added back comment
2022-01-27 18:45:45 -08:00
Akash Jain
a850628593
update make core_generate to run on ubuntu as well (#2373) 2022-01-24 14:04:27 +05:30
Akash Jain
657edfeb9a
use venv instead (#2370) 2022-01-24 12:21:35 +05:30
Pere Miquel Brull
b39b698eb7
Fix #2345 - Make bump & unify (#2359)
* Add installing instructions

* Unify makefile and automate bump version setup

* Update submake call

* Mark targets as phony

* Add help to recipes

* Update core publish

* Do not trigger install from echo

* Run make from root

* Prepare py_format from isort and black

* Run py_format

* Check black and isort formatting

* Update comment msg
2022-01-23 09:47:06 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349)
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
2022-01-22 12:08:14 -08:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085)
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
34a9d31240
Fix #2052: Revert the reverted pre-commit (#2053) 2022-01-07 03:53:21 +01:00
darth-coder00
5fb63c3019
Minor changes with webpack optimisation (#1949) 2021-12-28 23:16:52 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Ayush Shah
07924f80c3
Added Metabase Connector (#1726)
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
2021-12-15 09:53:55 -08:00
Alberto Miorin
f9436c65f0
Fix #1306: Evolve Location entity integration (#1442) 2021-12-03 02:13:13 +01:00
Akash Jain
e288a64781
feat: Adding CI for python integration tests (#1368)
* Adding CI for python integration tests

* Adding Paths
2021-11-24 22:20:36 -08:00
Ayush Shah
7156bf60a0
Import Error fix - Okta (#1315)
* Import Error fix

* Import errors fixed

* Make file updated as per comments
2021-11-22 15:06:05 -08:00
Pere Miquel Brull
67ac286f56
[Issue-1285] - Update ingestion package generation (#1295)
* Remove generated sources

* Ignore python generated sources

* Keep the generated dir

* Prepare publish recipe

* Simplify recipe

* Update README with generation

* Update docs with python local setup information

* Generate python sources in the CI
2021-11-21 08:02:31 -08:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API (#890)
* First approach

* Test model

* Validate endpoints

* Update schema codegen

* Prepare reqs dev

* Rename titles and codegen

* Update README

* Keep lineage title as addLineage

* Update pydantic classes

* Update API wrapper & tests

* Handle services methods

* Prepare tests
2021-10-21 14:51:38 -07:00
Pere Miquel Brull
1804de6d73
[Issue-760] - Precommit & isort (#772)
* Update configs

* Update configs

* Update precommit black exclude

* Update precommit exclude isort

* Test precommit

* Revert test commit

* isort & black

* Test precommit

* Revert test precommit

* Update docs

* Apply recipe update

* Fix black_check recipe

* Grammar
2021-10-14 15:22:59 -07:00
Pere Miquel Brull
9a362dafdc
[Issue 760] - Python tooling (#761)
* Update python tooling with black

* black formatting

* Change ci config for python
2021-10-14 07:48:42 -07:00
sureshms
8ce7c879e8 OpenMetadata snapshot release 0.3 2021-08-01 14:27:44 -07:00