2916 Commits

Author SHA1 Message Date
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Mayur Singal
3dc357561e
Fix #8868: Support json.gz files for datalake (#8869)
* Fix #8868: Support  files

* Review comments
2022-11-18 10:31:25 +00:00
Teddy
d680d5d2d2
added logic for modulo function for pyodbc driver (#8853) 2022-11-18 08:41:36 +00:00
Ayush Shah
be59a78a49
Fix Great Expectation Import Error (#8861) 2022-11-18 07:04:40 +00:00
Pere Miquel Brull
b203a92565
Add config file env var loading tests (#8799)
* Add env var load test

* Add env var load test
2022-11-17 16:52:42 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849)
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
2022-11-17 16:44:26 +00:00
Nahuel
626702f612
'get_table_entities_from_query' try with uppercase table name (#8839) 2022-11-17 14:27:07 +01:00
Mayur Singal
da490cf44b
Fix #8658: Redshift Data Type Incorrect (#8808) 2022-11-17 17:01:23 +05:30
Ayush Shah
7224145446
Fix and add Workflow Failures (#8838) 2022-11-17 10:45:17 +00:00
Teddy
4ab6dbc201
Data insight tests addition (#8814)
* Added additional tests around web analytics and data insight

* Added playwright dependency installation

* removed -m no-ui option

* Removed playwright tests

* Added event ingestion in setUp class
2022-11-17 10:44:52 +01:00
Pere Miquel Brull
fe16dea584
Fix #8794 - Separate DL requirements and lazy imports (#8806) 2022-11-17 09:11:54 +00:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Onkar Ravgan
c56004c395
Skipping analysis nodes (#8812) 2022-11-17 09:24:51 +01:00
Mayur Singal
b74fb82c15
Fix #8750: Superset Owner Fix (#8805)
* Fix #8750: Superset Owner Fix

* Refactor User Ingestion for Dashbaords

* User ES to Search User

* Pylint Fix

* Removed Unsed Import
2022-11-17 08:48:10 +01:00
Sriharsha Chintalapani
422eccf6e7
Fix #8738: Improve Search Relevancy (#8739) 2022-11-16 15:43:41 -08:00
Teddy
88c0ecb579
Remove count logic in custom sql test as 1) create complex logic and 2) multiple users got confused with expected tests results (#8819) 2022-11-16 18:10:55 +01:00
Pere Miquel Brull
d1cc3e880c
Fix e2e test and add slack alert (#8817)
* Fix mysql test

* Run alarm on failed test

* Force failure for testing
2022-11-16 17:43:48 +01:00
Ayush Shah
44613b1532
Fix Profiler issue (#8796) 2022-11-16 17:13:34 +05:30
Onkar Ravgan
93dc01fbb4
Fixed filterpattern and lineage (#8802) 2022-11-16 11:42:59 +01:00
Teddy
62bc104cab
Fix KPI Check and ES deletion logic (#8797)
* - limit check for existing es data to first iteration
- converted fraction to percentage for kpi success check

* Reverted value * 100, should be handle when we send the KPI definition request
2022-11-16 11:02:35 +01:00
Pere Miquel Brull
440e9ada75
Improve ES logging (#8793) 2022-11-16 10:05:40 +01:00
NiharDoshi99
d783869ed6
Fix: Added metadata service (#8759)
* Fix: Added metadata service

* Fix as per comment

* Fix changes for py-test

* Fix changes for py-test

* Fix py-checkstyle
2022-11-16 06:41:54 +01:00
Teddy
4ccdce50e0
Fix data insight bugs (#8761)
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day

* Added time utils

* fixed timezone
2022-11-15 16:10:30 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler (#8694)
* DQ commit

* Add DL Profiler

* Fix Ingestion and Profliing pylint checks

* Fix Tests

* PyFormat files

* Fix Tests

* Resolve Comments

* Fix Tests and Format Files

* Resolve Comments

* Fix Pylint and Code smells

* Resolve Comments

* Fix S3 parquet

* Fix Metrics Code Smell
2022-11-15 16:01:10 +01:00
Nahuel
f57fe1d507
Fix#8577: Ingestion Pipelines sensitive passwords fields are stored in secrets store (#8725)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test

* Bots sensitive passwords fields are stored in secrets store

* Ingestion pipeline sensitive passwords fields are stored in secrets store

* Fix Java style

* Add update for ingestion pipelines on start up

* Add pylint exception

* Fix broken test

* Update UI references
2022-11-15 06:34:49 +01:00
Teddy
3dbaa69978
Data insight workflow (#8729) 2022-11-15 05:44:25 +01:00
Pere Miquel Brull
f33003485e
ES reindex Airflow workflow (#8723)
* Added support for 'Metadata Service' connectors

* Fixed failing unit tests

* Removed delete button for 'Metadata Service' connector of type OpenMetadataServer

* Code optimization for SelectServiceType component

* Fixed errors arrived due to backend changes related to OpenMetadata Services

* init reindex

* Added additional step for adding Metadata to ES configs while creating ingestion pipeline for Metadata Service

* Add ES reindex

* Add ES reindex

* Format

* Format

* Rename service

* Pick service name from IngestionPipeline

* fix ui conflicts

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-14 18:59:56 +01:00
Mohit Yadav
db0215f964
Mapping from kpi to dichart(One to one) (#8707)
* Mapping from kpi to dichart(One to one)

* Mapping from kpi to dichart(One to one)

* Removed second tests creating KPI

* Fix some ui improve as well

* review comment changed function name

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-11-14 19:44:01 +05:30
Nahuel
e5abdc8c5c
Fix#8577: Services sensitive passwords fields are stored in secrets store (#8671)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test
2022-11-14 13:48:50 +01:00
Teddy
4212663765
Removed User and Test Suite (#8699) 2022-11-14 09:05:03 +01:00
Mayur Singal
18cd1c99ca
Add queries in sample data for lineage (#8674) 2022-11-14 13:11:16 +05:30
Mayur Singal
01bc9f1cfe
Fix PyMSSQL Version (#8696) 2022-11-14 08:40:41 +01:00
Milan Bariya
ebfb872e16
Fix: Filter Datalake empty files (#8677) 2022-11-13 22:09:29 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Mohit Yadav
4dc24b904f
python fix circular import (#8670)
python fix circular import (#8670)
2022-11-11 17:03:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas (#8654)
* Added topic parsers

* Fixed pylint

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-11 16:35:09 +05:30
Pere Miquel Brull
ff028a6eb0
Fix Mysql bigint (#8665)
* Fix mysql bigint cast

* Remove upgrade from dockerfiles
2022-11-11 10:12:23 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mayur Singal
5476879c51
Fix #8651: Fix Hive Table Descriptions (#8652)
* Fix #8651: Fix Hive Table Descriptions

* Removed pandas
2022-11-11 09:57:32 +01:00
Mayur Singal
8c7c08ae5d
Fix py-cli-e2e-tests (#8662) 2022-11-11 08:55:34 +01:00
Teddy
9d594efcb5
Fixes #8473 - Implements KPI Workflow (#8657)
* Added KPI workflow logic

* Update ingestion/src/metadata/data_insight/runner/run_result_registry.py

* ran py format

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-11 06:35:33 +01:00
Sam Firke
6ba5f7ec90
cast to bigint to avoid overflow when summing (#8655)
* cast to bigint to avoid overflow when summing

fixes #8430

* Update ingestion/src/metadata/orm_profiler/orm/functions/sum.py

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-11 06:33:38 +01:00
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler (#8623)
* Added logic to handle partitioning config in profiler

* extracted get_partition_details out of workflow classes
2022-11-10 10:54:31 +01:00
Sachin Chaurasiya
43ea44a0f9
Feat (#8161) Implement logic to fetch report data part-2 (#8605)
* Feat (#8161) Implement logic to fetch report data part-2

* Fix _get_user_details should return dict not None Type

* Format average session

* Make user name clickable

* Add provision for showing percentage symbol in graph tooltip

* Add summary support for entities

* Fix format issue

* Fix unit tests

* Fix labeling issue

* Add PageViewsByEntities chart

* Add DailyActiveUsers chart

* Add description to charts

* Fix unit tests

* remove startTs Overriding

* Address review comment
2022-11-10 09:56:04 +01:00
Mayur Singal
5d4f4a819f
Fix #8600: Add SQL query in lineage (#8603) 2022-11-10 08:25:23 +01:00
NiharDoshi99
283200449b
changed logic for parsing version (#8606) 2022-11-10 12:17:33 +05:30
Pere Miquel Brull
600aecbdf8
Bump bind9 bullseye version for ingestion container (#8619)
* Bump bind9 bullseye version

* Remove bind9 fixed version
2022-11-09 21:17:20 -08:00
NiharDoshi99
4ed088f55e
WIP: Fixed amundsen ingestion (#8544) 2022-11-09 14:21:44 +05:30
Mayur Singal
1d93d0e650
Fix #8570: Parsing Workflow Gracefully on Deploy (#8585)
* Fix #8570: Parsing Workflow Gracefully on Deploy

* use entity_class

* py format
2022-11-09 08:30:22 +01:00