2826 Commits

Author SHA1 Message Date
Imri Paran
c61ab94ff4
Update conn_test.py (#15385) 2024-02-28 14:21:30 +01:00
IceS2
418e281daa
Fixes 15375: Metabase metadata extraction fix (#15376) 2024-02-28 13:23:53 +05:30
Mayur Singal
8571ab87e8
MINOR: Fix mongodb profiler imports (#15383) 2024-02-28 13:05:01 +05:30
Teddy
056e6368d0
Issue #14765 - Preparatory Work (#15312)
* refactor!: change partition metadata structure for table entities

* refactor!: updated json schema for TypeScript code gen

* chore: migration of partition for table entities

* style: python & java linting

* updated ui side change for table partitioned key

* miner fix

* addressing comments

* fixed ci error

---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2024-02-28 07:11:00 +01:00
Mayur Singal
1987a161f3
MINOR: Fix oracle lineage query (#15372) 2024-02-27 15:34:34 +00:00
Imri Paran
f6e1f0d9c0
Fixes #15366: add profiler_data_time_series to TABLES_DUMP_ALL (#15369)
* add profiler_data_time_series to TABLES_DUMP_ALL
2024-02-27 15:52:53 +01:00
Mayur Singal
d2879ae232
MINOR: Improve Databricks Tags Ingestion (#15248) 2024-02-27 16:57:11 +05:30
Imri Paran
50b2709e94
MINOR: Mongodb column profile (#15252)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.
2024-02-26 07:38:38 +01:00
Teddy
16fdc249b7
fix: pin pandas version to 2.1.x (#15333) 2024-02-24 23:12:22 +05:30
Teddy
ba8208222e
MINOR - Fix column to match set test (#15186)
* fix: column value test for SQA types

* style: ran python linting
2024-02-23 16:35:58 +01:00
Imri Paran
bdf27458e5
MINOR: modified nosql factory to not use pymongo (#15316) 2024-02-23 16:48:59 +05:30
Onkar Ravgan
b78e43e8d7
fixed looker and optional proj filters (#15289) 2024-02-23 16:01:47 +05:30
Imri Paran
ff2ecc56f2
MINOR: add MongoDB sample data (#15237)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format

* feat(nosql-profiler): added sample data

1. Implemented the NoSQL sampler.
2. Some naming changes to the NoSQL adaptor to avoid fixing names with the profiler interface.
3. Tests.

* added default sample limit
2024-02-22 16:31:58 +01:00
Imri Paran
4967e091e6
MINOR: add test for sqla compiler (#15275)
* add test for sqla compiler
2024-02-22 14:45:47 +01:00
Imri Paran
18c22c4178
Fixes #10013: Implement first stage of NoSQL profiler (#15189)
* feat(nosql-profiler): row count

1. Implemented the NoSQLProfilerInterface as an entrypoint for the nosql profiler.
2. Added the NoSQLMetric as an abstract class.
3. Implemented the interface for the MongoDB database source.
4. Implemented an e2e test using testcontainers.

* added profiler support for mongodb connection

* doc

* use int_admin_ometa in test setup

* - fixed linting issue in gx
- removed unused inheritance

* moved the nosql function into the metric class

* formatting

* validate_compose: raise exception for bad status code.

* fixed import

* format
2024-02-22 11:46:19 +01:00
Onkar Ravgan
8a0ab20890
changed list_all_entities default value (#15307) 2024-02-22 14:27:00 +05:30
Imri Paran
6c0f85cc98
add compile decorator for ibmi ConnTestFn (#15273) 2024-02-20 09:35:27 -08:00
Onkar Ravgan
dfc7662449
Fix #15247: Fixed ingesting dbt owners with dot in name (#15261)
* Fixed dbt owners with dot

* fixed pytests

* Fixed pytest2

* rmv type
2024-02-20 16:06:54 +05:30
Mayur Singal
eac40862ec
MINOR: Kafka handle test connection if Schema Registry not passed (#15250) 2024-02-20 12:39:01 +05:30
Pere Miquel Brull
62c0cc7563
#13985 - Azure KV Secrets Manager (#15192)
* #13985 - Azure KV Secrets Manager

* Format

* #13985 - Azure KV Secrets Manager

* #13985 - Azure KV Secrets Manager

* Simplify credentials loading

* Simplify credentials loading

* Simplify credentials loading
2024-02-20 07:18:35 +01:00
Mayur Singal
dbb888d962
MINOR: Fix CLI E2E Tests (#15253) 2024-02-19 23:04:44 +05:30
Mayur Singal
64e3cedeea
MINOR: Improve Kafka Test Connection (#15207) 2024-02-19 10:11:52 +05:30
IceS2
7b20ed2f34
FIXES 15215: Add execution time tracker for ingestion (#15013)
* Initial idea on how to track execution times

* Fix linters

* Add missing decorators on the API Client

* Improve where the output is being handled
2024-02-16 09:58:20 +01:00
Onkar Ravgan
90b4933f96
Added sql to tableau datamodels (#15197) 2024-02-16 09:57:15 +05:30
Onkar Ravgan
ecdcf517ac
Fix #13790: Cost analysis Data Insights Optimisations (#15147)
* Add optimizations

* optimise and add caching for DI: CA

* formatting

* Added cache cleanup
2024-02-15 19:20:49 +05:30
Mayur Singal
4441b2763b
Fix #15112: Fix metabase lineage via api (#15165) 2024-02-15 16:04:51 +05:30
Onkar Ravgan
cdbcea11f6
fixed e2e counts (#15171) 2024-02-14 06:33:17 +01:00
Imri Paran
aeb5fbe303
fixes #12591: add BigTable (#15122)
* feat(connector): add BigTable

* bigtable work

1. docstrings
2. tests
3. created a Row BaseModel
4. implemented a ClassConverter

* docs moved to separate PR

* format files

* minor cosmetic

- removed TODO
- changed headers' year to 2024 for new files
- fixed typos

* format

* formatting and comments

1. added missing docstrings.
2. abstracted the _find_instance method.
3. aliased the IDs used in the BigTable connection

* added comment regarding private key

* added comments regarding column families

* enclose get_schema_name_list in `try/except/else`

* format

* streamlined get_schema_name_list to include all logic in the try block
2024-02-13 08:28:01 +01:00
Onkar Ravgan
3cc074ba5a
Fixed sagemaker either (#15156) 2024-02-13 10:17:30 +05:30
Mayur Singal
79d724fc7a
MINOR: Fix superset query to fetch columns (#15153) 2024-02-12 21:23:00 +05:30
Mayur Singal
1564a7494d
MINOR: Fix MSSQL SP Lineage Issue (#15155) 2024-02-12 21:22:45 +05:30
Onkar Ravgan
2ab5a70dce
Fixed service entities topology (#15127) 2024-02-12 21:18:52 +05:30
Akash Jain
95bb221025
feat: Prepare main for next release cycle (#15139) 2024-02-12 16:56:50 +05:30
Mayur Singal
96de1f5402
MINOR: Fix Sample Data Deserialization (#15079) 2024-02-09 11:25:53 +05:30
Teddy
11b82d4eea
fix: add table_name arg. for runtime batch (#15078) 2024-02-07 17:40:27 +05:30
NiharDoshi99
2b56e34b19
#14930 bigquery support for pk, fk and column view description (#15042) 2024-02-07 16:49:27 +05:30
Mayur Singal
331c687625
MINOR: Fix mysql e2e count (#15064) 2024-02-06 18:08:12 +00:00
Mayur Singal
d76809801d
MINOR: Fix Databricks SDK Breaking Change (#15037) 2024-02-06 10:42:53 +05:30
Onkar Ravgan
edb9c21bfd
Added /view to tableau dashboard url (#15031) 2024-02-05 20:18:02 +05:30
Mayur Singal
a9fc51ec8b
MINOR: Change sqllineage import to collate_sqllineage (#14870) 2024-02-05 19:44:08 +05:30
IceS2
c2bb1e5004
MINOR: Fix 'calculate_execution_time_generator' (#14987)
* Refactor the calculate_execution_time for generators decorator in order to fix it

* Add small comment in the code
2024-02-02 14:23:09 +01:00
Mayur Singal
7197356101
Minor: Fix pyarrow import error (#15004) 2024-02-02 16:28:40 +05:30
Onkar Ravgan
131e5f5767
Modified snowflake view def query (#14994) 2024-02-02 14:34:43 +05:30
Onkar Ravgan
25c6260bd7
MINOR: Optimised dbt topology and better exception handling for athena LF tags (#14978)
* dbt top and athena excp

* Added dbt excp handling

* sourcehash in patch
2024-02-02 09:21:15 +01:00
Ayush Shah
a4f3ed8eec
Minor: Fix E2E Powerbi (#14991) 2024-02-01 19:31:38 +05:30
Akash Jain
7a90de2e9e
fix(docker): K8s requires apache-airflow-providers-cncf-kubernetes pip package (#14989) 2024-02-01 18:20:45 +05:30
Teddy
9a4a9df836
Fix #14895 - Get Metadata from Parquet Schema (#14956)
* linting: fix python linting

* fix: get column types from parquet schema for parquet files

* style: python linting

* fix: remove displayType check in test as variation depending on OS
2024-02-01 09:02:52 +01:00
Sriharsha Chintalapani
2e95fcb98d
Fix #14786: Suggestions API (#14821)
* Fix #14786: Suggestions API

* Handle suggestions in ometa

* Minor: Optimise Databricks Client (#14776)

* MINOR - Fix SP topology context & Looker usage context (#14816)

* MINOR - Fix SP topology context & Looker usage context

* MINOR - Fix SP topology context & Looker usage context

* Fix tests

* Fixes #14598: Fix Tags / Labels ingestion on includeTags as False (#14782)

* fix(ui): password error message for char limits (#14808)

* fix(ui): password error message for char limits

* fix java side code

* Fixes #13556: Support for Salesforce table description ingestion (#14733)

* ISSUE-13556: Add suport for Salesforce table description ingestion

* ISSUE-13556: Remove unnecessary blank line

* ISSUE-13556: Fix to get description for each table

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

* MINOR - Better handling of Ingestion Pipeline Status (#14792)

* MINOR - Better handling of Ingestion Pipeline Status

* format

* format

* MINOR: Added table validation for cost analysis data (#14793)

* Added validation for cost analysis source

* centralized life cycle logic

* CYPRESS: simplify side navigation click in cypress (#14818)

* simplify side navigation click in cypress

* make sidbar item uses common enum

* fix cypress failure of outside import

* fix(#14326): tier dropdown is not working in advance search (#14780)

* improvement in advance search based on custom property

* fix a reading undefined property issue

* wip: advance search based on tier

* some code cleanup and improvement

* some fixes

* fix: ui flicker when advanceSearched is apply and refresh the page

* some cleanup

* no need to call customproperty api call, if entity not suppport customProperties

* minor change

* fix: autocomplete not working in tier search option in advance search modal

* added unit test for advance search provider component

* some cleanup

* added testcase for open modal

* added testcase for resetAllFilters method

* removed unwanted code

* added e2e test for testing tier advance search

* fix: e2e search flow for single field

* fix: string field not working after giving listValues in TierSearch

* fix: group query e2e test fix

* used asyncFetch way to get the tierOptions synchronously

* some cleanup

* remove unwanted lines

* some cleanup

* fix: selected option show option value instead of option title

* fix(minor): update skip icon for executions (#14809)

* Fixes #14803: ignore capitalization when confirming deletes  (#14804)

* ignore case when confirming deletes

* Test confirmation of deletes works when case differs 

Added test case for 'delete' as the confirmation text.

* minor(config): update openmetadata-ui code reviewers (#14823)

* Add Tests

* Add list/accept/reject apis

* initial ui changes

* localisation

* show suggestion for empty description

* ui feedbacks

* Fix permission check for entities without owner

* Fix entityLink and add tests

* Add update suggestion WIP

* Fix test

* Fix PUT and Pagination

* Fix styling

* update test

* Update status

* add OM server connection in apps

* add permissions check

* Fix CI

* Remove TODO

* Fix feedResourceTest

* fix unit tests

* add private configs for apps

* add private configs for apps

* fix update application icons

* minor center align icon

* add private configs for apps

* Format

* Fix pydantic gen

* Remove token

* Update name

* Rework private conf

* Fix apps

* Fix apps

* Format

* Format

* show metapilot only if its installed

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: kwgdaig <18678754+kwgdaig@users.noreply.github.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: Abhishek Porwal <80886271+Abhishek332@users.noreply.github.com>
Co-authored-by: Carlo Q <carlo@machina.bio>
Co-authored-by: karanh37 <karanh37@gmail.com>
2024-01-31 18:51:09 -08:00
Ayush Shah
0d66c0e514
Minor: Add extended glossary (#14918)
* Add Nested glossary to extended sample data

* Fix lint

* Add Stress test config

* Fix Lint
2024-01-31 07:13:04 -08:00
Onkar Ravgan
4c537f2d68
MINOR: Fix E2E failures (#14940)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2024-01-30 14:05:51 +05:30