44 Commits

Author SHA1 Message Date
Teddy
449a5f2de3
FIX #11951 - ingestion logic for global profiler config (#15948)
* feat: add global metric configuration for the profiler

* style: ran java linting

* fix: renamed disable to disabled

* style: ran java linting

* feat: ometa sdk for profiler setting

* test: ingestion profiler global config tests

* fix: update metric name to use MetricType Enum

* fix: allow bot to retrieve settings

* fix: exclude GX artifacts

* feat: implement global profiler setting logic for ingestion side

* fix: exclude metrics if Metric is empty

* style: ran python linting

* style: ran python linting

* fix: skip empty metrics

* style: ran python linting

* fix: moved GET profiler config to seperate endpoint in system resource

* fix: moved compute metric filter to MetricFilter + renamed container

* fix: test failures

* fix: profiler test case
2024-04-22 22:35:37 +02:00
Teddy
a424c37a34
[Proposal] Playwright CI (#13248)
* feat: playwright CI tests

* feat: ran python linting

* feat: updated workflow to run in PR

* feat: updated server start step

* feat: fixed playwright run command

* feat: fix matrix value

* feat: removed browser option in test run

* feat: updated typo in make file

* feat: move playwritgh installation

* feat: fail CI on error

* feat: save tracing on failure

* feat: enforce expect on .fill

* feat: change tracing to video

* feat: clean up code and update CI workflow for main branch
2023-09-19 14:44:36 +02:00
Sachin Chaurasiya
7619ce0a19
chore:(ui): parse ingestion workflow schemas (#12477)
* chore:(ui): parse ingestion workflow schemas

* chore: remove dereference part from script

* chore: address comments
2023-08-05 12:04:58 +05:30
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Ayush Shah
785142d86a
Add policy tags from Bigquery (#10189) 2023-02-20 19:13:45 +00:00
Deepa Rao
d2e0e75ac4
Fixes #9930 - Add API support for delete entities by name (#9931) 2023-01-25 21:03:40 -08:00
Nahuel
5608c2fde0
Add E2E CLI tests coverage to Sonar (#9908) 2023-01-25 11:51:25 +01:00
NiharDoshi99
34aacb20d6
Fix: atlas connector (#9034) 2022-11-29 23:55:05 +05:30
Mayur Singal
cc8ce8a13b
Ignore Local Docker Volume Files (#8913) 2022-11-21 15:48:40 +05:30
Nahuel
37105e09df
Add CLI e2e tests workflow dispatch (#8053)
* Add CLI E2E tests for MySQL

* Fix setup.py and pylint

* Add missing doc and update code after pylint refactor

* Add manual trigger for py-cli-e2e-tests GH action
2022-10-10 11:50:20 +02:00
Sriharsha Chintalapani
656b50dd3a
Fix #7469: Refactor OpenMetadata code modules (#7474) 2022-09-14 23:14:02 -07:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Nahuel
e321008cd0
Snyk report uploaded as HTML files (#6674) 2022-08-09 21:54:08 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128)
2022-07-19 14:51:44 +02:00
Teddy
7431c1c226
[Issue-5188] Implement User Query for Sampler and Profiler (#5578)
* Added custom query sample for sample data ingestion

* Added logic to run table profiling against user's query

* Added tests for user query logic in profiler and sampler

* Added user profiling to tableProfile + fixed format

* staging commit

* Added logic to add profileQuery to table entity

* Added limit to sample rows
2022-06-24 14:46:34 +02:00
Teddy
28336b0a43
Fix #3575 Data Quality - Partitioned Tables (#5441)
Co-authored-by: Teddy Crepineau <teddycrepineau@Teddys-MacBook-Pro.local>
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
2022-06-14 12:37:44 -07:00
Teddy
bbe5cfa9e7
[ISSUE-3450] openmetadata-ingestion great-expectations submodule (#4752)
[ISSUE-3450] openmetadata-ingestion great-expectations submodule (#4752)
2022-05-26 15:16:03 +02:00
darth-coder00
372bc340c7
Fix: UI add generated TS to ignored files (#5049)
* Ignored files

* Add generated TS to ignored files
2022-05-20 00:18:58 +05:30
darth-coder00
9fc612e013
Optimising generation of typescripts from schema and as prebuild script (#5037)
* Adding json2ts as prebuild script

* Add generated TS to gitignore

* Fixing CI errors

* Fix Sonar CI

* Bumping metadata version

* Fixing sonar CI

* Fix json2ts script

* Revert gitignore

* prettify

* Fix CI build failures

* Removing unnecessary changes

* Add pre-script for coverage
2022-05-19 21:23:39 +05:30
Vivek Ratnavel Subramanian
af8b95091a
Cypress Integration (#4859) 2022-05-10 22:47:04 -07:00
darth-coder00
148c96c0a4
Fix #4455: UI Support for Optional DBT fields (#4489) 2022-05-01 22:06:30 -07:00
darth-coder00
fb2de6125a
Fix #4207: json schema files for connections generated during build time (#4339) 2022-04-21 17:40:48 -07:00
Alberto Miorin
60c452cb7e
Add JavaScript to antlr and clean up the Java code (#4099)
* Add JavaScript to antlr and clean up the Java code
2022-04-14 10:18:54 +02:00
darth-coder00
241525000d
Added UI Error handling for All service catagories and auth config apis (#3767)
* Added UI Error handling for All service catagories and auth config apis
2022-04-04 19:53:11 +05:30
Teddy
6ee31eb21f
Issue-2538: Add Iceberg type for Glue Table + Location Entity (#3210) 2022-03-07 08:27:14 -08:00
Vivek Ratnavel Subramanian
64c157498d
Fix #2609: Ignore .idea from git (#2610) 2022-02-03 15:02:37 -08:00
darth-coder00
92d6e943db
Added unit test coverage config (#2513) 2022-01-31 22:09:00 +05:30
Sriharsha Chintalapani
e5fdf77d8e
Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service (#2357)
* Fix #2340: Ingestion pipeline deployment refactor - update ingestion

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service

* Fix #2355: Upgrade scripts, handle entity_extension updatedAt and dbservice jdbc field , Fix #2353: Add API to AirflowPipelines filter based on service
2022-01-22 21:12:25 -08:00
Akash Jain
00ccd612ee
fix: Ingestion Core CI to Publish PyPi packages (#2187)
* Remove gitignore on _versions.py

CI commits and pushes the new version in _versions.py and since the folder is marked under gitignore, CI fails to commit and push changes to the branch.

* bump version to 0.8.0.dev0 as already published
2022-01-13 12:40:47 +05:30
Sriharsha Chintalapani
c4d7c76198
Event publishers to ElasticSearch and Slack (#2118)
* ElasticSearch & Slack Event publishers

* Fix #2165: Support for deleted entities in ElasticSearch
2022-01-12 16:35:27 -08:00
Sriharsha Chintalapani
cf6f438531
Fix #1994: Add support for marking dataset entities as deleted (#2141)
* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted

* Fix #1994: Add support for marking dataset entities as deleted
2022-01-11 11:42:53 +05:30
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud (#2085)
* Update path

* Prepare sonar properties

* Prepare coverage recipes

* Add coverage

* Simplify pytest

* Organise integration tests

* Update path

* Use setup instead of reqs

* Update recipes

* Fix PR event to target

* Update event_name

* Prepare sonar

* Run tests & sonar

* Use sonarcloud host

* Fix compose

* Use ingestion token
2022-01-07 10:28:38 +01:00
Akash Jain
5685efd651
feat: openmetadata-ingestion-core pypi publish dev packages (#2023)
* update requirements-dev.txt

* integrate pypi incremental

* Add CI for #1875

* gitignore generated dir in ingestion-core

* update version to match ingestion package
2022-01-04 21:11:33 -08:00
Alberto Miorin
ec1e3a2d61
Fix #1344: Remove requirements.txt (#1588) 2021-12-20 21:36:23 -08:00
Sriharsha Chintalapani
5a2c97724f
Fix #1824: ES index live check if tableType is set (#1825)
* Fix #1824: ES index live check if tableType is set
2021-12-17 11:06:41 -08:00
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector (#1677)
* Prepare infra

* Store experiment information in MySQL & MinIO

* Use CreateMlModelEntityRequest instead of MlModel for PUT operations

* Update MlFlow infra

* Prepare MlFlow source

* Prepare Mlflow workflow

* Simplify test and prepare README

* Revert compose

* Fix compose

* Prepare warnings and fix features

* Use non-default port for integration test

* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00
Alberto Miorin
249643b79f
Small fix to .gitignore (#1524) 2021-12-02 10:58:17 -08:00
parthp2107
a9e70e6a89
License header update (#1498)
* updated liscense header

* updated license header

* updated license header

* updated license header

* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Sriharsha Chintalapani
a595332701
Fix #1471: Add Intellij idea settings to repo (#1472)
* Fix #1471: Add Intellij idea settings to repo
2021-11-30 12:14:54 -08:00
Sriharsha Chintalapani
a180adc39c
Fix #1299: Move UI module into its own maven module (#1351)
* Fix #1299: Move UI module into its own maven module

* Fix #1299: fix indentation in selenium tests

* Fix #1299: fix indentation in selenium tests

* Fix #1299: fix indentation in selenium tests

* Fix: Updated json2ts command script

* Removed ignored file for bundle analyzer

* Updated gitignore

* Cleaning up the gitignore

Co-authored-by: darth-coder00 <aashit@getcollate.io>
2021-11-24 13:18:54 +05:30
Pere Miquel Brull
67ac286f56
[Issue-1285] - Update ingestion package generation (#1295)
* Remove generated sources

* Ignore python generated sources

* Keep the generated dir

* Prepare publish recipe

* Simplify recipe

* Update README with generation

* Update docs with python local setup information

* Generate python sources in the CI
2021-11-21 08:02:31 -08:00
Alberto Miorin
4cc922c885
Add storage service (S3, GCS, HDFS, and so on) (#866)
Moved to the PR.
2021-11-03 13:21:36 +05:30
parthp2107
b5baa9f2db applied license to the backend files 2021-08-02 15:08:30 +05:30
parthp2107
ed6a74b052 added git files 2021-08-02 11:07:13 +05:30