88 Commits

Author SHA1 Message Date
Shailesh Parmar
e947f4cf77
playwright: add make command to run playwright test (#16575)
* playwright: add make command to run playwright test

* chore: Remove unused yarn command and update Makefile
2024-06-08 08:34:57 +00:00
IceS2
b67de741e4
MINOR: Small refactor to the Make update_all scripts (#16204)
* Small refactor to the Make update_all scripts

* Extract regex update method

* decouple release from workflows

* Fix issue with docker-openmetadata-db workflow referencing non existent action
2024-05-10 11:40:30 +02:00
Akash Jain
22b7ac3a0b
chore(Makefile): Update Make Recipes for 1.3.0 changes (#14951) 2024-02-05 23:45:21 +05:30
sushi30
93c8ce3998
MINOR - Docs prerequisites recipe (#15014)
* docs(check-prerequisites): prerequisites recipe

- added scripts/check_prerequisites.sh.
- added usage in docs.
- added prerequisites to Makefile.

* docs(check-prerequisites): fix for docker version

* docs(check-prerequisites): fix for docker version

* docs(check-prerequisites): fix for docker version

* docs(check-prerequisites): revert docker-compose.yml
2024-02-05 07:04:40 +01:00
Akash-Jain
16ff7be4f9 fix: Security Scan CI file path 2024-01-19 16:12:57 +05:30
Akash Jain
72fc0cf685
fix: Security Scan CI + Makefiles (#14790) 2024-01-19 14:18:07 +05:30
Akash Jain
4e8042c9a8
fix(CI): Enable Security Scan Reports for ingestion base slim docker image (#14788) 2024-01-19 12:38:02 +05:30
Akash Jain
ab3440938a
fix: make ingestion build docker image using Dockerfile.ci (#14080) 2023-11-23 07:12:20 +01:00
Pere Miquel Brull
caaf0e7a1d
Fix #12436 - Migrate to pyproject.toml (#14025)
* test

* Use pyproject.toml

* Fix pylint

* makefile

* makefile

* Fix pylint

* isort

* pyproject

* Airflow apis pyproject

* Remove ingestion core

* isort

* Fix makefile help
2023-11-22 07:10:37 +01:00
Mohit Yadav
3f8a931e39
Bump Pom Version to 1.3.0-SNAPSHOT (#14008)
* Bump Pom Version to 1.3.0-SNAPSHOT

* chore: Fix Makefile recipe

* fix: Prepare Main Branch for Next Feature Release

* fix: Syntax issue

---------

Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-11-17 11:33:47 +05:30
Tom Bushell
33fe3b5ee5
Fixes #7858: Moved ingestion related make commands into Makefile in ingestion directory (#13677)
* Moved more recipes into ingestion/Makefile

* Removed some recipes into ingestion/Makefile and added import statement

* Modified file paths so that 'make generate' works from the ingestion directory

* Modified checks for current directory

* Fixed function names to be in snake case

* Reverted function names back to camel case

* Reverted changes to js_antlr and py_antlr and moved generate command back into root directory Makefile

* Updated run_ometa_integration_testsrecipe in ingestion/Makefile

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-30 08:23:47 +01:00
Aniket Katkar
9e773ff1a8
chore(docs): updated the metadata standard schemas and menu.md for documentation (#13745)
* updated the metadata standard schemas and menu.md

* updated the schema docs with better parsing
2023-10-27 11:37:31 -07:00
Teddy
452a33b1a0
Fixes Druid Profiler failures (#13700)
* fix: updated playwrigth test structure

* fix: druid profiler queries

* fix: python linting

* fix: python linting

* fix: do not compute random sample if profile sample is 100

* fix: updated workflow to test on push

* fix: move connector config to category folder

* fix: updated imports

* fix: added pytest-dependency package

* fix: updated readme.md

* fix: python linting

* fix: updated profile doc for Druid sampling

* fix: empty commit for CI

* fix: added workflow constrain back

* fix: sonar code smell

* fix: added secrets to container

* Update openmetadata-docs/content/v1.2.x-SNAPSHOT/connectors/ingestion/workflows/profiler/index.md

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Update openmetadata-docs/content/v1.2.x-SNAPSHOT/connectors/ingestion/workflows/profiler/index.md

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>

* Update ingestion/tests/e2e/entity/database/test_redshift.py

* fix: ran pylint

* fix: updated redshift env var.

* fix: import linting

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-10-25 20:47:51 +02:00
Pere Miquel Brull
0282574bdd
Create ometa client once and pass it around & improve pycln config (#13310)
* Create ometa client once and pass it around & improve pycln config

* Fix

* Fix

* Fix tests

* Fix maven ci

* Fix tests

* Fix tests

* Fix tests

* Format

* Fix DI
2023-10-04 09:14:03 +02:00
Teddy
a424c37a34
[Proposal] Playwright CI (#13248)
* feat: playwright CI tests

* feat: ran python linting

* feat: updated workflow to run in PR

* feat: updated server start step

* feat: fixed playwright run command

* feat: fix matrix value

* feat: removed browser option in test run

* feat: updated typo in make file

* feat: move playwritgh installation

* feat: fail CI on error

* feat: save tracing on failure

* feat: enforce expect on .fill

* feat: change tracing to video

* feat: clean up code and update CI workflow for main branch
2023-09-19 14:44:36 +02:00
Pere Miquel Brull
5a6310eff9
[Docs] - Yarn dev on local docker (#13169)
* [Docs] - Yarn dev on local docker

* added env variable values for docker-docs command

* reverted the changes in makefile

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-09-13 14:56:45 +02:00
Anuj359
9d649964fd
Creating make script for code freeze automation (#12976)
* Disk space test

* Disk space test with other action

* Disk space test with other action

* Disk space test with first action

* checking labeler

* updating version of wait action

* xvfb-run check

* Update disk-space-test.yml

* xvfb-run check using sudo

* checking large packages

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Update disk-space-test.yml

* Checking the logic

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update update_version.py

* Update Makefile

* Update update_version.py

* Update Makefile

* permission issue check

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update Makefile

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update release-code-freeze.yml

* Update Makefile

* Creating make script for code freeze automation

* updating dockerfile version with main branch

* Updating paths in makefile

* Updating the python code with logger

* Using argparse to pass the arguments in python code

* feat:Updating script path

* Delete update_version.py from root directory
2023-09-13 15:10:28 +05:30
Pere Miquel Brull
5da63c5aed
Fix #12734 - Do not publish .egg to pypi (#12799)
* Fix #12734 - Do not publish .egg to pypi

* Remove python 3.7 tests support
2023-08-10 07:30:22 +02:00
Aniket Katkar
cf2aafa770
changes in script to generate schemas documentation (#12325)
* changes in script to generate schemas documentation

* generated schema doc changes

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-07-11 09:59:51 +02:00
Sriharsha Chintalapani
6ca1ec6fbe
Delete old docs (#11627)
* Delete old docs and rename the openmetadata-docs-v1 to openmetadata-docs

* Delete old docs and rename the openmetadata-docs-v1 to openmetadata-docs

* Delete old docs and rename the openmetadata-docs-v1 to openmetadata-docs
2023-05-17 07:04:56 +02:00
Pere Miquel Brull
26bd1f4618
Add .snyk (#11614) 2023-05-16 11:03:07 +02:00
Pere Miquel Brull
463f242d6b
Add S3 Storage docs & CI validation (#11120)
* Prep CI

* Update Athena

* Update docs for S3 Storage

* Add manifest information
2023-04-19 06:31:55 +02:00
Onkar Ravgan
c9feba826f
Kinesis bug fixed (#10578)
* Kinesis bug fixed

* Added sample data fix

* Fixed typo

* Added constansts

* Added try catch for sample data

* Added fixes on review comments

* Modified models

* Added sample data py models

* Fix pylint

* fixed code smell

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-03-23 13:38:56 +05:30
Sriharsha Chintalapani
de10cd254b
New docker structure (#10438)
* Adding the different docker-compose file openmetadata and ingestion

* Added two different env files for mysql and postgres

* Updated the docker file path

* Updated the path of docker folder structure

* Fix docker

* Updating the PR with necessary changes required

---------

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: Akash-Jain <Akash.J@deuexsolutions.com>
2023-03-06 08:58:53 -08:00
Nahuel
a3490093df
Generate PDF for Snyk security report (#10086)
* Generate PDF for Snyk security report

* Add missing doc

* Minor change
2023-02-02 17:10:35 +01:00
Nahuel
d7b40162fd
Doc: Update security docs related to SSL configuration (#10053) 2023-02-02 11:32:54 +00:00
Pere Miquel Brull
b0058bdb65
Fix #9707 - Install instantclient and enable thick mode (#9761)
* Install instantclient

* Prep instant client

* Update docs

* Amend comments
2023-01-19 08:08:56 +01:00
Sriharsha Chintalapani
9702e70ea4
Minor: Delete generated directory to delete older files (#9643)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-08 21:32:16 -08:00
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test (#9363)
* Prep airflow operator integration tests

* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849)
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
2022-11-17 16:44:26 +00:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Nahuel
fbc7b245c1
Fix SonarCloud coverage XML file (#8177)
* Fix SonarCloud coverage XML file

* Add minor change on Python code

* Update coverage config

* Update coverage config

* Remove test output

* Remove sonar property

* Update Makefile
2022-10-17 17:38:19 +02:00
Pere Miquel Brull
d48fd468d4
Tag PATCH + Cleanup of helpers methods (#8150)
* cleanup

* lint

* Add tag patch

* Fix rename

* Dont kill tests
2022-10-15 14:56:30 +02:00
Nahuel
b69581299f
Fix SonarCloud coverage for Python code (#8144) 2022-10-14 15:17:40 +02:00
Ayush Shah
58765379a1
Fix Policy Tags from diff project ids (#8004) 2022-10-12 15:17:21 +05:30
Pere Miquel Brull
4b3ca065c4
Fix #3394 - Pylint fixes and add it to CI (#8084)
Fix #3394 - Pylint fixes and add it to CI (#8084)
2022-10-11 17:15:02 +02:00
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image (#8065)
Prepare Ingestion Base Docker image (#8065)
2022-10-11 07:50:49 +02:00
Pere Miquel Brull
f031293af0
Fix Pylint ingestion/bulksink and ingestion/stage (#8030)
* Pylint ingestion bulksink

* Pylint stage

* Remove test

* Remove stage
2022-10-10 14:06:43 +02:00
Sriharsha Chintalapani
656b50dd3a
Fix #7469: Refactor OpenMetadata code modules (#7474) 2022-09-14 23:14:02 -07:00
Pere Miquel Brull
15c83e3f2d
Docs - Update images and clean files (#7005)
* Update image links

* Remove external YAMLs
2022-08-29 08:59:41 +02:00
Teddy
ce578e73d4
Fixes #5831 by implenting testSuite workflow logic (#6911)
* Added database filter in workflow

* Removed association between profiler and data quality

* fixed tests with removed association

* Fixed sonar code smells and bugs

* Updated profiler workflow to:
- support only running profiler (removed test run)
- support column inclusion and exclusion
- added back support for partitioned table and sample

* moved status to workflow

* Fixed tests

* removed test logic from profiler sink

* Added logic to return sample from workflow sample value

* Added profiler examples

* Updated documentation for profiler

* Fixed code smells

* commited changed to profiler

* initial commit of the revamp workflow

* Fixed python formating

* cleaned up profiler submodule by removing test related files and functions

* Added airflow DAG logic for testSuite workflow

* Fixed code smells + added airflow ingestion tests + fixed comments
2022-08-25 10:01:28 +02:00
Ayush Shah
36338eb727
Antlr Grammar implemented, Imports fixed (#6673) 2022-08-18 18:32:28 +05:30
Nahuel
e321008cd0
Snyk report uploaded as HTML files (#6674) 2022-08-09 21:54:08 +02:00
Pere Miquel Brull
049077e1b5
Airflow APIs coverage (#6501)
Airflow APIs coverage (#6501)
2022-08-02 14:02:44 +02:00
Pere Miquel Brull
8ec2dbe6e4
Update vuln (#6432) 2022-07-29 13:08:12 +02:00
Pere Miquel Brull
ea918495ff
Fix #6382 - Snyk workflows (#6413)
Fix #6382 - Snyk workflows (#6413)
2022-07-28 23:06:33 +02:00
Suresh Srinivas
a23a9c84c1
Fixes #6336 - Add API to list of operations per resource for policy authoring (#6337)
* Fixes #6336 - Add API to list of operations per resource to be used for writing policies

* Fixing UI build errors

* Include only JSON schema file and not JSON data files in python code generation

* Fix generate

* Update operations list

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-07-26 12:08:01 -07:00
Pere Miquel Brull
385b7edac9
Prepare CI for Airflow APIs (#6240)
Prepare CI for Airflow APIs (#6240)
2022-07-21 12:29:57 +02:00
Nahuel
f0aab78295
Fix workflow GH for py3.9 (#6198) 2022-07-20 07:07:04 +02:00