1614 Commits

Author SHA1 Message Date
Ayush Shah
89d89b13cb
Fix BQ ingestion (#9015) 2022-11-25 12:51:24 +00:00
Teddy
afca46d45b
Added limit for number of entities fetch because of gateway time out + handled exception when fetching user details (#8999) 2022-11-24 16:10:14 +01:00
Sriharsha Chintalapani
95f0f4d9e1
Fix column name indexes (#8990) 2022-11-23 19:08:44 -08:00
Sriharsha Chintalapani
285e7dc1e7
Fix arg parsing (#8989)
* Fix arg parsing

* fixed checkstyle python

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-11-23 14:50:44 -08:00
Teddy
8789bdb847
Ometa mixin for ingestion pipeline (#8931)
* Added sleep between data processing ingestion and KPI processing

* Added ometa mixin for ingestion pipeline and test suite
2022-11-23 09:46:56 +00:00
Teddy
b8d99fc275
increase size of es results when checking for existing data (#8966) 2022-11-23 09:44:24 +00:00
Ayush Shah
9d25781481
Add DL profiler fix (#8957) 2022-11-22 18:15:06 +00:00
Ashish Gupta
36f27e947d
UI : Mask the JWT token in Metadata service (#8842)
* Mask the Jwt token in Metadata service

* minor fix

* fix the icon alignment in Add Ingestion button

* disable the test connection for metadata service OpenMetadata type

* change the css name

* fix unit test issue

* Fix Auth Provider

* Fix add ingestion dropdown icon alignment

* Fix formatting

* Do not encryt JWT auth mechanism with secrets manager

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2022-11-22 14:17:40 +00:00
Pere Miquel Brull
d6f255b61c
Add ingestion workflow reporter (#8943)
* Add ingestion workflow reporter
2022-11-22 12:11:32 +00:00
Nahuel
5151db4ae0
Fix bug in secrets manager with SSO configuration (#8932)
* Fix bug in secrets manager with SSO configuration

* Remove comment from imports

* Sort imports

* Add missing decrypt when updating

* Add minor change
2022-11-21 22:37:20 +01:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change (#8909) 2022-11-21 12:35:23 +01:00
Abhishek Pandey
4427fe487f
added-exception-handling-in-redash-dashboard-url-creation (#8763)
* added-exception-handling-in-redash-dashboard-url-creation

* code-reformatted

* exception-error-msg-updated

* error-msg-updated-in-redash

* error-msg-updated-in-redash

* error-msg-length-reduced

* code-formatted

* main-rebases
2022-11-21 16:53:55 +05:30
Onkar Ravgan
5aa44e1cf0
Fix #8778 and #8494: Tableau Parser issue and API docs (#8875)
* Tmp fixed parser and added doc for API

* fixed pycheckstyle

* Addressed review comments
2022-11-21 12:50:25 +05:30
Teddy
89348b8622
Added sleep between data processing ingestion and KPI processing (#8876) 2022-11-21 07:28:15 +01:00
Milan Bariya
bc215515d8
Fix: ignored query with om-version and dbt (#8851)
* Fix: ignored query with om-version and dbt

* Change based on comments
2022-11-18 14:22:11 +01:00
Vj-L
2c055641be
Added host volumes for Database service (#8722)
* Added host volumes for Database service

* Formated the python file

* Done changes according to the comments on PR

* Changes done as per the comments on Draft PR

* Fix PyLint

* Changes done as per the comments on Draft PR

* Updated the path of volume for Database

* Added cleanup state in workflows

* Added cleanup state in workflows

Co-authored-by: “Vijay” <“vijay.l@deuexsolutions.com”>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-18 14:05:33 +01:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Mayur Singal
3dc357561e
Fix #8868: Support json.gz files for datalake (#8869)
* Fix #8868: Support  files

* Review comments
2022-11-18 10:31:25 +00:00
Teddy
d680d5d2d2
added logic for modulo function for pyodbc driver (#8853) 2022-11-18 08:41:36 +00:00
Ayush Shah
be59a78a49
Fix Great Expectation Import Error (#8861) 2022-11-18 07:04:40 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849)
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
2022-11-17 16:44:26 +00:00
Nahuel
626702f612
'get_table_entities_from_query' try with uppercase table name (#8839) 2022-11-17 14:27:07 +01:00
Mayur Singal
da490cf44b
Fix #8658: Redshift Data Type Incorrect (#8808) 2022-11-17 17:01:23 +05:30
Ayush Shah
7224145446
Fix and add Workflow Failures (#8838) 2022-11-17 10:45:17 +00:00
Teddy
4ab6dbc201
Data insight tests addition (#8814)
* Added additional tests around web analytics and data insight

* Added playwright dependency installation

* removed -m no-ui option

* Removed playwright tests

* Added event ingestion in setUp class
2022-11-17 10:44:52 +01:00
Pere Miquel Brull
fe16dea584
Fix #8794 - Separate DL requirements and lazy imports (#8806) 2022-11-17 09:11:54 +00:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Onkar Ravgan
c56004c395
Skipping analysis nodes (#8812) 2022-11-17 09:24:51 +01:00
Mayur Singal
b74fb82c15
Fix #8750: Superset Owner Fix (#8805)
* Fix #8750: Superset Owner Fix

* Refactor User Ingestion for Dashbaords

* User ES to Search User

* Pylint Fix

* Removed Unsed Import
2022-11-17 08:48:10 +01:00
Sriharsha Chintalapani
422eccf6e7
Fix #8738: Improve Search Relevancy (#8739) 2022-11-16 15:43:41 -08:00
Teddy
88c0ecb579
Remove count logic in custom sql test as 1) create complex logic and 2) multiple users got confused with expected tests results (#8819) 2022-11-16 18:10:55 +01:00
Ayush Shah
44613b1532
Fix Profiler issue (#8796) 2022-11-16 17:13:34 +05:30
Onkar Ravgan
93dc01fbb4
Fixed filterpattern and lineage (#8802) 2022-11-16 11:42:59 +01:00
Teddy
62bc104cab
Fix KPI Check and ES deletion logic (#8797)
* - limit check for existing es data to first iteration
- converted fraction to percentage for kpi success check

* Reverted value * 100, should be handle when we send the KPI definition request
2022-11-16 11:02:35 +01:00
Pere Miquel Brull
440e9ada75
Improve ES logging (#8793) 2022-11-16 10:05:40 +01:00
NiharDoshi99
d783869ed6
Fix: Added metadata service (#8759)
* Fix: Added metadata service

* Fix as per comment

* Fix changes for py-test

* Fix changes for py-test

* Fix py-checkstyle
2022-11-16 06:41:54 +01:00
Teddy
4ccdce50e0
Fix data insight bugs (#8761)
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day

* Added time utils

* fixed timezone
2022-11-15 16:10:30 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler (#8694)
* DQ commit

* Add DL Profiler

* Fix Ingestion and Profliing pylint checks

* Fix Tests

* PyFormat files

* Fix Tests

* Resolve Comments

* Fix Tests and Format Files

* Resolve Comments

* Fix Pylint and Code smells

* Resolve Comments

* Fix S3 parquet

* Fix Metrics Code Smell
2022-11-15 16:01:10 +01:00
Nahuel
f57fe1d507
Fix#8577: Ingestion Pipelines sensitive passwords fields are stored in secrets store (#8725)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test

* Bots sensitive passwords fields are stored in secrets store

* Ingestion pipeline sensitive passwords fields are stored in secrets store

* Fix Java style

* Add update for ingestion pipelines on start up

* Add pylint exception

* Fix broken test

* Update UI references
2022-11-15 06:34:49 +01:00
Teddy
3dbaa69978
Data insight workflow (#8729) 2022-11-15 05:44:25 +01:00
Pere Miquel Brull
f33003485e
ES reindex Airflow workflow (#8723)
* Added support for 'Metadata Service' connectors

* Fixed failing unit tests

* Removed delete button for 'Metadata Service' connector of type OpenMetadataServer

* Code optimization for SelectServiceType component

* Fixed errors arrived due to backend changes related to OpenMetadata Services

* init reindex

* Added additional step for adding Metadata to ES configs while creating ingestion pipeline for Metadata Service

* Add ES reindex

* Add ES reindex

* Format

* Format

* Rename service

* Pick service name from IngestionPipeline

* fix ui conflicts

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-14 18:59:56 +01:00
Nahuel
e5abdc8c5c
Fix#8577: Services sensitive passwords fields are stored in secrets store (#8671)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test
2022-11-14 13:48:50 +01:00
Teddy
4212663765
Removed User and Test Suite (#8699) 2022-11-14 09:05:03 +01:00
Mayur Singal
18cd1c99ca
Add queries in sample data for lineage (#8674) 2022-11-14 13:11:16 +05:30
Milan Bariya
ebfb872e16
Fix: Filter Datalake empty files (#8677) 2022-11-13 22:09:29 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Mohit Yadav
4dc24b904f
python fix circular import (#8670)
python fix circular import (#8670)
2022-11-11 17:03:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas (#8654)
* Added topic parsers

* Fixed pylint

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-11 16:35:09 +05:30
Pere Miquel Brull
ff028a6eb0
Fix Mysql bigint (#8665)
* Fix mysql bigint cast

* Remove upgrade from dockerfiles
2022-11-11 10:12:23 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00