yujunjun/OpenMetadata
1
0
Fork 0
mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-08-11 18:46:56 +00:00
Code Issues Packages Projects Releases Wiki Activity
12,418 Commits 725 Branches 152 Tags
Commit Graph

723 Commits

Author SHA1 Message Date
Ayush Shah
f11cdbfab1
Profile sample rows + Randomization of chunks for datalake (#9595) 2023-01-16 22:17:46 +05:30
NiharDoshi99
029dbe892e
Fix: added test case for atlas (#9678) 
* Fix: added test case for atlas

* Fix: resolved conflict

* Fix: changing back neo4j to old version

* Fix: changing back neo4j to old version

* Fix: changes as per comment

* Fix: changes as per comment

* Fix: python checkstyle
 2023-01-13 16:07:29 +05:30
Pere Miquel Brull
bf753a4dee
Fix #7768 - Update and organize versions (#9664) 
Fix #7768 - Update and organize versions (#9664)
 2023-01-11 07:05:12 +01:00
Pere Miquel Brull
84348d4748
Fix #8866 - bump datamodel-codegen (#9623) 
* Fix #8866 - bump datamodel-codegen

* Update connection options and arguments structure

* Add builders test

* Format

* Allow Any values in componentConfig

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
 2023-01-09 13:20:32 +01:00
NiharDoshi99
27c68fb208
test case for DomoDatabase (#9630) 
* test case for domo-database

* test case for domo-database

* Fix: pytest
 2023-01-06 12:02:01 +01:00
Mayur Singal
9ef32eef99
Fix #6829: Improve usage filtering (#9613) 2023-01-05 15:07:03 +00:00
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector (#9554) 
* feat: update databricks_client

* feat: create pipelineconnection & connection

* test: add unit test

* feat: add examples file

* fix: remove unused imports

* style: fix python style

* fix: code review adjustments

* fix: remove unneeded schema ref

* fix: format requirements

* fix: change context calls

* test: add pipeline_status test

* format: pylint format change

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2023-01-05 14:14:12 +01:00
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588) 
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588)
 2023-01-04 08:50:53 +01:00
8vicat
7e7528e13b
Improve SQL lineage parsing (#9537) 
* CTEs support
* Supports inline comments
* Expands `*` in sources
 2023-01-03 09:27:45 +01:00
NiharDoshi99
8f69386076
Fix: refactor get_column_type (#9559) 
* Fix: refactor get_column_type

* Fix: added changes as per comments

* Fix: pylint

* Fix: minor changes

* Fix: minor changes
 2023-01-03 10:28:38 +05:30
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients (#9545) 2023-01-02 13:52:27 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package (#9521) 
Fix #6562 - Sources have their own package (#9521)
 2022-12-27 15:00:22 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs (#9260) 2022-12-26 12:32:17 -08:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers (#9513) 
Organise dynamic importers (#9513)
 2022-12-23 12:17:06 +01:00
Teddy
9e01fe0636
feat(CLI): added migration logic cmd (#9437) 2022-12-21 07:55:18 +00:00
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number (#9104) 2022-12-20 14:55:11 +05:30
Milan Bariya
a3495c67f0
Databricks improve code and doc update (#9394) 
* Databricks improve code and doc update

* Change base on comments

* Fix pytest fail
 2022-12-19 18:53:10 +05:30
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler (#9340) 
* feat(Profiler): Added timeout logic in the profiler workflow

* feat(Profiler): Added additional input in workflow UI

* fix(Profiler): timeout text blob in UI

* fix(Profiler): Return profile_results outside the context manager

* Fix(Profiler): Added localizer for timeout label and message

* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py

Co-authored-by: Nahuel <nahuel@getcollate.io>

Co-authored-by: Nahuel <nahuel@getcollate.io>
 2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field (#9314) 2022-12-16 14:17:47 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update (#9322) 
* DB2 connection config and ingestion update

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py

Co-authored-by: Ayush Shah <ayush@getcollate.io>

* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql

Co-authored-by: Ayush Shah <ayush@getcollate.io>
 2022-12-16 07:43:18 +01:00
Onkar Ravgan
b539b299ee
Integrated schema parsers (#9305) 
* Integrated schema parsers

* Addressed review comments

* fixed pytests
 2022-12-15 16:54:55 +05:30
Ayush Shah
a6ae9fd11a
Add Test Suite Implementation for Datalake (#9235) 2022-12-14 21:14:51 +05:30
Nahuel
9a4e3a7a46
Fix#8424: Remove brackets from tables and schemas on lineage (#9257) 
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test

* Remove brackets from tables and schemas on lineage
 2022-12-13 06:40:37 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection (#9196) 
* Add Test Connection to Profiler

* remove Code Lint

* Fix AttributeError

* Fix Pytest

* Fix Bigquery Partition tests

* Fix Lint
 2022-12-10 17:16:28 +01:00
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler (#9159) 
* refactor(profiler): integrated getter func.

Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.

* feature(profiler): added systemProfiler schema

* feat(profiler): workflow fresh. & snflk impl.

* feat(profiler): freshness endpoint for put and get

* feat(profiler): added system met. for redshift

* feat(profiler): freshness met. for bigquery

* fix(profiler): keyword not found in func

* feat(profiler): Added sample data for freshness

* fix(profiler): fetch previous day for BQ

* fix(profiler): sonar + data fetching logic

* fix: typo in SystemMetric Class

* fix: linting

* fix: extracted out EntityList class into models.py
 2022-12-07 14:33:30 +01:00
Nahuel
00375341f7
Fix#7690: Use ES to build Tables FQN when processing Lineage (#9079) 
* Use ES to build Tables FQN when processing Lineage

* Fix pylent check

* Fix test failing

* Remove warning when calling ES with empty query

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-12-01 09:56:42 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use (#9073) 
* Refactor LineageRunner use

* Address PR comments

* Address pylint errors

* Fix failing test
 2022-11-30 16:02:21 +01:00
Teddy
3cad959e44
Fixes #6760 -- Implements REGEX for regex test (#9033) 
feat(testCase): impelemented regex logic for test suite
 2022-11-29 13:00:28 +01:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change (#8909) 2022-11-21 12:35:23 +01:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas (#8818) 2022-11-18 11:35:08 +01:00
Pere Miquel Brull
b203a92565
Add config file env var loading tests (#8799) 
* Add env var load test

* Add env var load test
 2022-11-17 16:52:42 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required (#8849) 
* Always run the checkstyle

* Add print checker lint plugin

* Typos

* Add print checker

* Fix e2e
 2022-11-17 16:44:26 +00:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id (#8708) 2022-11-17 14:26:37 +05:30
Teddy
4ccdce50e0
Fix data insight bugs (#8761) 
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day

* Added time utils

* fixed timezone
 2022-11-15 16:10:30 +01:00
Ayush Shah
5be0f8ee76
Dl Profiler (#8694) 
* DQ commit

* Add DL Profiler

* Fix Ingestion and Profliing pylint checks

* Fix Tests

* PyFormat files

* Fix Tests

* Resolve Comments

* Fix Tests and Format Files

* Resolve Comments

* Fix Pylint and Code smells

* Resolve Comments

* Fix S3 parquet

* Fix Metrics Code Smell
 2022-11-15 16:01:10 +01:00
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras (#8682) 
* Always run python tests

* Fix athena tests and types

* Update deployed prop in IngestionPipeline

* Fix #8554

* Format

* Use true as default deployed migration

* Remove repeated req

* Pydantic wiggle room
 2022-11-13 11:59:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas (#8654) 
* Added topic parsers

* Fixed pylint

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
 2022-11-11 16:35:09 +05:30
Nahuel
91d0460b27
Fix#8577: Refactor part of the secrets manager implementation (#8617) 
* Removed part of the secrets manager implementation in the server side

* Removed part of the secrets manager implementation in the openmetadata python library side

* Remove deprecated test

* Address pylint checks

* Address new pylint checks

* Address PR comments

* Fix import on airflows apis

* Clear singleton instances for running Ometa secrets manager test
 2022-11-11 09:59:15 +01:00
Teddy
9d594efcb5
Fixes #8473 - Implements KPI Workflow (#8657) 
* Added KPI workflow logic

* Update ingestion/src/metadata/data_insight/runner/run_result_registry.py

* ran py format

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
 2022-11-11 06:35:33 +01:00
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler (#8623) 
* Added logic to handle partitioning config in profiler

* extracted get_partition_details out of workflow classes
 2022-11-10 10:54:31 +01:00
NiharDoshi99
4ed088f55e
WIP: Fixed amundsen ingestion (#8544) 2022-11-09 14:21:44 +05:30
Milan Bariya
4d16be2608
Fix: Hive connection issue (#8583) 
* Fix: Hive connection issue

* Change based on comments
 2022-11-08 16:22:37 +01:00
michizhou
da06d183db
feat: add QuickSight connector (#8442) 2022-11-08 19:54:49 +05:30
Abhishek Pandey
73b370b5e2
schema-filter-added-in-datalake-for-bucket (#8516) 
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
 2022-11-08 10:57:16 +05:30
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events (#8528) 
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Fixed typos brought up during reviews
 2022-11-07 17:08:20 +01:00
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387) 
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
 2022-11-03 18:19:50 +01:00
Tushar Mittal
6f2c93089c
feat: add SageMaker connector (#8435) 
* feat: add sagemaker connector

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* fix: fix linting errors and update imports

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* test: add unit tests for sagemake source

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
 2022-11-03 18:19:20 +01:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector (#8452) 
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
 2022-11-02 16:12:45 +05:30
Teddy
989f2911c2
Fixes #7810 - Allow to only pass min or max (#8474) 
* ISSUE-7810 Added default values for min and max
For all data validations on columns:-
min_bound is set to float("-inf"), if there is no next value
max_bound is set to float("inf"), if there is no next value

* Fixed PR errors by removing tuple + added tests

Co-authored-by: demi <deepak1212365@gmail.com>
 2022-11-01 13:26:51 +01:00
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse (#8182) 2022-10-31 18:12:26 +05:30