1489 Commits

Author SHA1 Message Date
Mohit Yadav
db0215f964
Mapping from kpi to dichart(One to one) (#8707)
* Mapping from kpi to dichart(One to one)

* Mapping from kpi to dichart(One to one)

* Removed second tests creating KPI

* Fix some ui improve as well

* review comment changed function name

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-11-14 19:44:01 +05:30
Nahuel
e5abdc8c5c
Fix#8577: Services sensitive passwords fields are stored in secrets store (#8671)
* Services sensitive passwords fields are stored in secrets store

* Update services on application startup in case of changes in the JSON Schema

* Minor changes after manual test
2022-11-14 13:48:50 +01:00
Teddy
4212663765
Removed User and Test Suite (#8699) 2022-11-14 09:05:03 +01:00
Mayur Singal
18cd1c99ca
Add queries in sample data for lineage (#8674) 2022-11-14 13:11:16 +05:30
Mayur Singal
01bc9f1cfe
Fix PyMSSQL Version (#8696) 2022-11-14 08:40:41 +01:00
Milan Bariya
ebfb872e16
Fix: Filter Datalake empty files (#8677) 2022-11-13 22:09:29 +05:30
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682)
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Mohit Yadav
4dc24b904f
python fix circular import (#8670)
python fix circular import (#8670)
2022-11-11 17:03:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas (#8654)
* Added topic parsers

* Fixed pylint

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-11 16:35:09 +05:30
Pere Miquel Brull
ff028a6eb0
Fix Mysql bigint (#8665)
* Fix mysql bigint cast

* Remove upgrade from dockerfiles
2022-11-11 10:12:23 +01:00
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617)
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
2022-11-11 09:59:15 +01:00
Mayur Singal
5476879c51
Fix #8651: Fix Hive Table Descriptions (#8652)
* Fix #8651: Fix Hive Table Descriptions

* Removed pandas
2022-11-11 09:57:32 +01:00
Mayur Singal
8c7c08ae5d
Fix py-cli-e2e-tests (#8662) 2022-11-11 08:55:34 +01:00
Teddy
9d594efcb5
Fixes #8473 - Implements KPI Workflow (#8657)
* Added KPI workflow logic

* Update ingestion/src/metadata/data_insight/runner/run_result_registry.py

* ran py format

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-11 06:35:33 +01:00
Sam Firke
6ba5f7ec90
cast to bigint to avoid overflow when summing (#8655)
* cast to bigint to avoid overflow when summing

fixes #8430

* Update ingestion/src/metadata/orm_profiler/orm/functions/sum.py

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-11-11 06:33:38 +01:00
Suresh Srinivas
86c3ae30f1
Fixes #8596 Introduce mutually exclusive tags and glossary terms (#8597) 2022-11-10 16:47:21 -08:00
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler (#8623)
* Added logic to handle partitioning config in profiler

* extracted get_partition_details out of workflow classes
2022-11-10 10:54:31 +01:00
Sachin Chaurasiya
43ea44a0f9
Feat (#8161) Implement logic to fetch report data part-2 (#8605)
* Feat (#8161) Implement logic to fetch report data part-2

* Fix _get_user_details should return dict not None Type

* Format average session

* Make user name clickable

* Add provision for showing percentage symbol in graph tooltip

* Add summary support for entities

* Fix format issue

* Fix unit tests

* Fix labeling issue

* Add PageViewsByEntities chart

* Add DailyActiveUsers chart

* Add description to charts

* Fix unit tests

* remove startTs Overriding

* Address review comment
2022-11-10 09:56:04 +01:00
Mayur Singal
5d4f4a819f
Fix #8600: Add SQL query in lineage (#8603) 2022-11-10 08:25:23 +01:00
NiharDoshi99
283200449b
changed logic for parsing version (#8606) 2022-11-10 12:17:33 +05:30
Pere Miquel Brull
600aecbdf8
Bump bind9 bullseye version for ingestion container (#8619)
* Bump bind9 bullseye version

* Remove bind9 fixed version
2022-11-09 21:17:20 -08:00
NiharDoshi99
4ed088f55e
WIP: Fixed amundsen ingestion (#8544) 2022-11-09 14:21:44 +05:30
Mayur Singal
1d93d0e650
Fix #8570: Parsing Workflow Gracefully on Deploy (#8585)
* Fix #8570: Parsing Workflow Gracefully on Deploy

* use entity_class

* py format
2022-11-09 08:30:22 +01:00
Teddy
dcd0bbb566
Fixes 8470 -- Implements ES aggregation for web analytic data (#8566)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Added dailyActiveUsers aggregation

* Added page views entities aggregation and active users aggregation
2022-11-08 17:25:31 +01:00
Milan Bariya
4d16be2608
Fix: Hive connection issue (#8583)
* Fix: Hive connection issue

* Change based on comments
2022-11-08 16:22:37 +01:00
Teddy
2ffb88f6aa
Added bit -> binary column type conversion for mssql (#8581) 2022-11-08 16:21:55 +01:00
michizhou
da06d183db
feat: add QuickSight connector (#8442) 2022-11-08 19:54:49 +05:30
Mayur Singal
710e3d785f
Fix #8495: Update pipeline sources for pipeline as edge lineage (#8556) 2022-11-08 14:13:49 +05:30
Abhishek Pandey
73b370b5e2
schema-filter-added-in-datalake-for-bucket (#8516)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-11-08 10:57:16 +05:30
Mayur Singal
83102d6697
Fix #8455: fix dbt test errors when run not passed (#8545) 2022-11-08 10:50:27 +05:30
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events (#8528)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Fixed typos brought up during reviews
2022-11-07 17:08:20 +01:00
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387)
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
2022-11-03 18:19:50 +01:00
Tushar Mittal
6f2c93089c
feat: add SageMaker connector (#8435)
* feat: add sagemaker connector

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* fix: fix linting errors and update imports

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* test: add unit tests for sagemake source

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-03 18:19:20 +01:00
Milan Bariya
6a7704a746
Fix: Filter Databricks history by date (#8520)
* Fix: Filter Databricks history by date

* Fix: Filter Databricks history by date

* Change based on comment
2022-11-03 22:08:56 +05:30
Shailesh Parmar
0f0489a7d4
Feat: Added pipeline as edge from ui (#8503)
* Feat: Added pipeline as edge from ui

* fixed failing unit test

* removed pipeline

* added unit test for util function
2022-11-03 17:46:49 +05:30
Pere Miquel Brull
c326cc1370
Move logging to DEBUG (#8519) 2022-11-03 13:04:33 +01:00
Mayur Singal
1386b43607
Fix #6141: Ingestion Pipeline Status Updates (#8216)
* Fix #6141: Ingestion Pipeline Status Updates

* List Pipeline Status API & Improvements

* Rename State field to PipelineState in UI

* Convert Pipeline Status array to single object

* fix braking UI

* Rebase Fixes

* Profiler, TestStuite & DataInsigts Pipeline

* py_format

* fix logs page not loading
add pipelineStatus endpoint

* fix recent run changes

* Fix Tests

* address review comments for ui

* fix failing checks

* fix unit tests

* fix cypress test

* remove loader test as not using it

* wait for API when we click on the tab

* fix cypress waitFor api

* fix failed cypress tests

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-03 14:37:26 +05:30
Onkar Ravgan
8273197b48
Fix #7767: Added new PowerBi APIs and lineage (#8485)
* Added new APIs and lineage

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-03 08:31:31 +01:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector (#8452)
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-02 16:12:45 +05:30
Pere Miquel Brull
119763afc4
Bump datamodel-code-generator (#8492) 2022-11-02 10:31:44 +01:00
Mayur Singal
fa68a1f18f
Fix #6754: Added User Details for Usage (#8449) 2022-11-02 14:07:41 +05:30
Milan Bariya
fbe9e59dcd
Fix: Athena connector fails to ingest complex types (#8453)
* Fix: Athena connector fails to ingest complex types

* Fix: Athena connector fails to ingest complex types

* Fix: Athena connector fails to ingest complex types

* Fix: Change based on comments
2022-11-02 07:06:39 +01:00
Teddy
989f2911c2
Fixes #7810 - Allow to only pass min or max (#8474)
* ISSUE-7810 Added default values for min and max
For all data validations on columns:-
min_bound is set to float("-inf"), if there is no next value
max_bound is set to float("inf"), if there is no next value

* Fixed PR errors by removing tuple + added tests

Co-authored-by: demi <deepak1212365@gmail.com>
2022-11-01 13:26:51 +01:00
Onkar Ravgan
5819c45937
Databricks + DBT issue (#8391)
* Databricks dbt issue fixed

* fixed typo

* Added optimizations

* skipping flaky dashboard test

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-11-01 13:41:41 +05:30
Onkar Ravgan
a6d6cff0d5
Fixed abstract init issue (#8446)
* abstract init issue fixed

* abstract init issue fixed messaging service

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-01 11:40:08 +05:30
Onkar Ravgan
ddc66c8392
vertica query syntax issue fixed (#8463) 2022-11-01 00:16:45 +05:30
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse (#8182) 2022-10-31 18:12:26 +05:30
Pere Miquel Brull
f4c20ded32
Ingestion Image - Remove expat version release and CI (#8447)
* Remove expat version

* Remove expat version
2022-10-31 10:01:05 +01:00
Pere Miquel Brull
bc0ce0cec1
Remove expat version (#8445) 2022-10-31 13:55:44 +05:30
Nahuel
3ed6ad8456
Update GID used in ingestion docker images (#8434) 2022-10-29 13:03:37 -07:00