284 Commits

Author SHA1 Message Date
Sriharsha Chintalapani
706cebd97a
Opensearch connector (#19698)
* Fix #19667: OpenSearch Connector

* Fix #19667: OpenSearch Connector

* do not ingest any system level indexes

* fix pyformat

* Add AWS auth

* Use common schema and fix ssl config in client

* Add openseach connector docs and update schema

* Remove api key auth type and complete docs checklist

* Remove unnecessary httpx dependency and pyformat

* Add compatible version of httpx for elasticsearch

* Fix pylint fails and py-tests validation error

---------

Co-authored-by: Mohit Tilala <tilalamohit123@gmail.com>
Co-authored-by: Mohit Tilala <63147650+mohittilala@users.noreply.github.com>
2025-03-18 18:45:25 +05:30
harshsoni2024
826279608f
issue-19892: parse powerbi table source (#20141) 2025-03-12 12:59:29 +05:30
harshsoni2024
40a9c67875
Day 1 - Dashboard service lineage without db_service_name (#19911) 2025-03-07 11:16:58 +05:30
Suman Maharana
137f86c00d
Day1 - Pipeline Usage (#19960) 2025-03-06 18:19:42 +05:30
Suman Maharana
0cd35015d7
#1049 Day 1 Dashboard Usage - Tableau, Looker (#19796) 2025-03-06 18:15:28 +05:30
mgorsk1
0e260ce36a
feat: use dataset sql for superset lineage (#19617)
* 🎉 Init

* add fixes around row data type and backport fixes around array type

* refactor

* narrow down column lineage

* narrow down column lineage

* fix insert into

* review remarks + linter

* fix tests

* fix style check

* bring back compatibility with older python versions

* add unit tests

* fix tests

* linting

* expand tests

* make compatible with python 3.8

* checkstyle fix

* fix schema alias

* refactor - move get_dashboard_data_model_column_fqn to superset.utils as it's superset specific and won't work with other dashboard systems integrations
2025-02-12 07:14:31 -08:00
Suman Maharana
0d73ee7e36
Fixes #19171 - Looker parse derived view references query (#19687)
* Looker parse derived view references query

* Addressed Comments

* Linting

* Added Tests
2025-02-10 02:50:51 +05:30
Akash Verma
837e6dd418
Metabase: Ingest orphan charts inside a dashbaord named Default (#19419) 2025-01-28 16:53:19 +05:30
harshsoni2024
b1d481f2f1
issue-16744: salesforce column description with toggle api (#19527) 2025-01-27 16:54:35 +05:30
olof-nn
fd2575d244
ISSUE-19454: Fixes broken looker lineage (#19456)
* ISSUE-19454: Fixes the broken lineage in looker when backticks enclosed table refs

* refactor

* use isort

* Update ingestion/src/metadata/ingestion/source/dashboard/looker/metadata.py

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

---------

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2025-01-22 17:56:01 +05:30
Keshav Mohta
7bea4f957f
Feature: Docker Host Retry (#19127) 2025-01-14 19:48:10 +05:30
agriev
dcebc41e3f
Adds percona server for postgresql support (#19322)
* percona server for postgresql support

The only meaningful difference is version string in percona server for postgresql. So commit propose universal and safe way to detect server version by integer string, not complicated parsing of unformatted string.

* updated tests with get_server_version_num

commented outdated tests

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2025-01-13 17:51:40 -08:00
Akash Verma
b2898f7007
Cockroach enhancement (#19108) 2025-01-07 18:51:59 +05:30
Akash Verma
39dcb5baef
Feature : Cockroach db connector (#18961) 2025-01-02 13:07:55 +05:30
Akash Verma
69557e8716
fixes: #15742 Oracle stored package feature (#18852) 2024-12-16 19:35:20 +05:30
Keshav Mohta
cde3a7dd1e
Feature: Cassandra Connector (#18943) 2024-12-12 15:12:55 +05:30
Ayush Shah
8664c8df75
Fixes GEN-2199: Allow Fivetran filtering of pipelines using name instead of id (#18929) 2024-12-05 10:55:11 +05:30
harshsoni2024
cb33f274fc
Connector: rename microstrategy connector (#18604) 2024-11-28 18:50:42 +05:30
Suman Maharana
9a21e77e15
Added dbt cloud multi projects and jobs filter (#18801)
* Added dbt cloud multi project and jobs filter

* added tests

* change to array type

* updated yaml config

* added migrations
2024-11-28 16:10:34 +05:30
Imri Paran
ee7d043035
[GEN-2109] feat(mongo): added ssl support (#18731)
* feat(mongo): added ssl support

Added SSL support for MongoDB using the SSL manager.

Attached a video demo.

- [Example repository for setting up mongodb with SSL](https://github.com/sushi30/mongodb-docker-ssl-example)
- [MongoDB TLS documentation](https://www.mongodb.com/docs/manual/tutorial/configure-ssl/)

* fixed test_doris.py
2024-11-22 08:54:13 -08:00
Sriharsha Chintalapani
88c8fb48f3
Add Edit glossary terms, Edit Tier , Edit Tags as separate permissions (#18331)
* Add EditGlossaryTerms Permission

* Fix #18330: Add EDIT_GLOSSARY_TERM permission and enforce EDIT_TIER permisson

* add edit glossary term permission check in UI

* revert EDIT_GLOSSARY_TERMS operation

* Add EDIT_GLOSSARY_TERMS to common operations

* Add EDIT_TIER to common operations

* add default empty array for tags field, as patch calls can run into issues

* Fix tests

* Fix tests

* added glossary terms

* fix conflicts

* fix permission check for data model

* Add EditGlossaryTerms to DataConsumerPolicy

* Add EditGlossaryTerms,EditTier to DataConsumerPolicy

* fix tests

* Fix migrations for EditTier,EditGlossaryTerms

* add edit tier permission to data consumer

* Fix tests

* fix pytests

* missing test_dbt.py

---------

Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2024-11-15 10:50:15 -08:00
IceS2
dccba20101
Return s3 endpoint as str() instead of Url (#18521) 2024-11-05 17:39:50 +00:00
Katarzyna Kałek
47c75fe6a7
Enhanced Glue ingestion with external table features (#18511)
* added fileFormat, locationPath and external table lineage to Glue ingestion

* Improve Lineage Label

---------

Co-authored-by: Katarzyna Kałek <kkalek@olx.pl>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2024-11-05 21:48:20 +05:30
Mayur Singal
9d91325af8
Lineage-1: Move view lineage processing to lineage workflow (#18220)
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2024-10-28 18:18:22 +05:30
Suman Maharana
dd08bc9ffd
GEN-895: Added Glue Pipeline Lineage (#18063) 2024-10-14 13:08:17 +05:30
Sachin Chaurasiya
457f3d919a
GEN-1322: API Entity - Remove Beta (#17967)
* GEN-1322: API Entity - Remove Beta

* minor: add doc for the metadata pipeline

* api service refactor

* api service refactor backend changes

* add apiconnection in test service connection

* pytest fix

* fix java file formatting

* Fix casing of REST in ApiServiceRest.spec.ts

* Refactor REST to Rest in API classes

* minor change

* minor change

* minor change

* fix cashing for API to Api

* add playwright test for api service ingestion

* fix: playwright test

---------

Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
2024-10-08 14:39:55 +05:30
Ethan
49fceb4674
Fixes #18104 : change parse_obj and assertEquals which was deprecated (#18105)
* change deprecationwarning

* fix format python

* fix replace module

* change : java function name
2024-10-07 09:02:41 +02:00
sam-mccarty-mavenclinic
0dd3e97170
Fix 17911: Looker parsing improvements for liquid templating and view/model aliasing (#17912)
* Looker parsing improvements for liquid templating and view/model aliasing

* add python-liquid dependency to looker plugin requirements

* move to static method with 'openmetadata' context and add rendering tests

* remove backtick stripping

---------

Co-authored-by: Imri Paran <imri.paran@gmail.com>
2024-09-27 13:55:15 +02:00
Suman Maharana
37b6dc8290
Add Sigma Dashboard Connector (#17855)
* Add Sigma Dashboard Connector

* changed to id instead of name in dashboard entity

* Address Comments

* addressed comments

* Added Docs

* yaml file changes

* fix ui changes
2024-09-26 16:29:35 +05:30
harshsoni2024
4f57f4e710
GEN-309: OpenAPI Connector (#17754)
* configure api service metadata

* add rest api service

* fix test con. pyformat changes

* add models, fix test con.

* improve test con.

* add docs, side doc

* fix model data parse, url error fix

* add tests

* fix pytest errors

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-09-11 13:36:53 +05:30
nicor88
235337dbcd
fixes 17648: add tags and description for tableau published data source (#17678)
* tableau data-source tags and description

* tweaks

* PR review

* change test description

* black formatting
2024-09-03 11:25:27 +02:00
Pere Miquel Brull
bcb29b46da
MINOR - Implement SAP Hana Lineage (#17615)
* MINOR - SAP Hana Lineage

* skeleton

* parser

* lineage

* manage formulas

* add cvs

* add cvs

* better typing

* enum

* handle cvs

* saphana docs
2024-08-30 07:42:43 +02:00
Do Manh Ha
a868596db7
Fixes #17461: Unquote and interpret escaped characters in BigQuery dataset description (#17462)
* fix(bigquery): unquote and convert any escaped characters to their actual representations

* test: bigquery description with multiple line

---------

Co-authored-by: Imri Paran <imri.paran@gmail.com>
2024-08-20 17:56:19 +02:00
Imri Paran
31c2ec8c57
MINOR: fix qlikcloud test connection (#17459)
* fix: qlikcloud test connection

* patch test_connection for qlik cloud unit tests
2024-08-19 23:14:09 -07:00
Ayush Shah
af14267e09
Fixes #17319: ArrayDataType issue resolved, Fix Queries + Add DB Name to the queries (#17379)
* fixes arrayDataType must be not null, adding db name to queries as it fails

* Fix Pydantic Issue

* Partial: Add Unity Catalog Topology Test

* Fix lint

* Fix Tests, Fix UnityCatalog Array Column issue

* Fix Tests

* Address comments, add logger to the exception
2024-08-12 09:59:03 +02:00
harshsoni2024
0548342239
Fix #16958: column parser data type fix (#17154) 2024-08-11 00:24:46 +05:30
sam-mccarty-mavenclinic
11a49ef08d
Fixes 16010: Support Looker ingestion from Gitlab (#17284)
* introduce gitlab option to lookml ingestion

* fix reader and disable test

* fix copy paste in test case

* fix file read and keyset pagination for tree

* fix credentials to include gitlab credentials

* uncomment arguments for unused credentials to fix validation error

* fix credentials test

* fix credentials test

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2024-08-07 06:53:44 +02:00
Onkar Ravgan
fe7922c13c
MINOR: [SAP ERP Connector] Added column dtype displaynames and precision/scale values (#17240) 2024-08-01 12:49:34 +05:30
Mayur Singal
840a102887
Fix #17195: Support automated unstructured files ingestion & tags (#17196) 2024-07-31 00:05:58 +05:30
Sriharsha Chintalapani
fe107aa3cb
Issue #17012: Multi User/Team Ownership (#17013)
* Add multiple owners

* Multi Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 1

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 2

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 3

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 4

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 5

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 6

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 7

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 8

* Add Migrations for Owner Thread

* update ingestion for multi owner

* fix pytests

* fixed checkstyle

* Add Alert Name to Publishers (#17108)

* Add Alert Name to Publishers

* Fix Test

* Add Bound to Setuptools (#17105)

* Minor: fixed testSummaryGraph issue (#17115)

* feat: updated multi pipeline ui as per new mock (#17106)

* feat: updated multi pipeline ui as per new mock

* translation sync

* fixed failing unit test

* fixed playwright test

* fixed viewService click issue

* sorted pipeline based on test case length

* Added domo federated dataset support (#17061)

* fix usernames (#17122)

* Doc: Updated Doris & Redshift Docs (#17123)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Fix #12677: Added Synapse Connector - docs and side docs (#17041)

* Fix #17098: Fixed case sensitive partition column name in Bigquery (#17104)

* Fixed case sensitive partiion col name bigquery

* update test

* #13876: change placement of comment and close button in task approval workflow (#17044)

* change placment of comment and close button in task approval workflow

* minor change

* playwright test for the close and comment function

* supported ref in activityFeedEditor

* fix playwright test

* added playwright test for data steward

* fix the test for the data streward user

* fix the close button not showing if task has no suggestions and icon fixes

* fix sonar issue

* change glossary and add suggestion button to dropdown button

* fix the glossary failure due to button change

* icon change for add tag and description

* fix glossary cypress failure due to button chnages

* changes as per comments

* MINOR: docs links fix (#17125)

* alation link fix

* dbt yaml config source link fix

* bigquery doc fix

* Explore tree feedbacks (#17078)

* fix explore design

* update switcher icon

* show menu when search query exists

* fix selection of active service

* fix type error

* fix tests

* fix tests

* fix tests

* MINOR: Databricks view TableType fix (#17124)

* Minor: fixed AUT test (#17128)

* Fix #16692: Override Lineage Support for View & Dashboard Lineage (#17064)

* #17065: fix the tags not rendering in selector after selection in edit tags task (#17107)

* fix the tags not rendering in selector after selection in edit tags taks

* added playwright test

* minor changes

* minor fix

* fix the tags not updating in edit and accept tag

* fix explore type changes for collate (#17131)

* MINOR: changed log level to debug (#17126)

* changed log level to debug

* fixed type

* changed type to optional

* Get feed and count data of soft deleted user (#17135)

* Doc: Adding OIDC Docs (#17139)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* Doc: Updating Profiler Workflow Docs URL (#17140)

Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>

* fix playwright and cypress (#17138)

* Minor: fixed edit modal issue for sql test case (#17132)

* Minor: fixed edit modal issue for sql test case

* fixed test

* Minor: Added whats new content for 1.4.6 release (#17148)

* MINOR [GEN-799]: add option to disable manual trigger using scheduleType (#17031)

* fix: raise for triggering system app

* added scheduleType ScheduledOrManual

* minor: remove "service" field from required properties in createAPIEndpoint schema (#17147)

* initial commit multi ownership

* update glossary and other entities

* update owners

* fix version pages

* fix tests

* Update entity_extension to move owner to array (#17200)

* fix tests

* fix api page errors

* fix owner label design

* locales

* fix owners in elastic search source

* fix types

* fix tests

* fix tests

* Updated CustomMetric owner to entityReferenceList. (#17211)

* Fix owners field in search mappings

* fix search aggregates

* fix inherited label

* Issue #17012: Multi User/Team Ownership - Fix Tests - Part 9

* Fix QUeries

* Fix Mysql Queries

* Typo

* fix tests

* fix tests

* fix tests

* fix advanced search constants

* fix service ingestion tests

* fix tests

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: k.nakagaki <141020064+nakaken-churadata@users.noreply.github.com>
Co-authored-by: Prajwal214 <167504578+Prajwal214@users.noreply.github.com>
Co-authored-by: Prajwal Pandit <prajwalpandit@Prajwals-MacBook-Air.local>
Co-authored-by: Suman Maharana <sumanmaharana786@gmail.com>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
Co-authored-by: harshsoni2024 <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
Co-authored-by: Imri Paran <imri.paran@gmail.com>
Co-authored-by: sonika-shah <58761340+sonika-shah@users.noreply.github.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: karanh37 <karanh37@gmail.com>
Co-authored-by: Siddhant <86899184+Siddhanttimeline@users.noreply.github.com>
2024-07-30 08:06:39 +02:00
Ayush Shah
34c572334e
Fixes #14945: FiveTran Improvements (#17169) 2024-07-30 10:16:47 +05:30
Onkar Ravgan
b6745d7cf1
MINOR: Implemented SAPERP Connector feedback (#17137)
* Implemented saperp feedback

* Fixed pytest
2024-07-29 10:31:20 +05:30
Antoine Balliet
e67ba6b14c
feat: BigQuery ingestion allow specify projectId and fix primary key constrains retrieval (#16956) 2024-07-17 11:21:17 +05:30
harshsoni2024
52dc5b551e
MINOR: fix column name in databricks ingestion fail (#17019)
* add backticks in column_name

* add test for column name with .
2024-07-17 10:25:12 +05:30
Matt Chamberlin
d757aa9d77
Fixes 16652: add GCS storage service (#16917)
* FEAT-16652: add GCS storage service

* reformat

* update connection tests

* fix tests

* relax google-cloud-storage version constraint

* fix GCP config in tests

---------

Co-authored-by: Matthew Chamberlin <mchamberlin@ginkgobioworks.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-07-10 14:03:28 +02:00
Onkar Ravgan
80efc7075f
Fix #15163: Added SAP ERP Connector 2024-07-04 10:57:46 +05:30
harshsoni2024
da37626794
superset lineage fix (#16820)
* fix column list for lineage in db connection

* fix for column yield in datamodel

* superset add test for datamodel of db
2024-07-03 10:55:47 +05:30
Suman Maharana
2477bbc9ca
Fixes #16574 : Fix Databricks pipeline task status ingestion (#16802)
* Fixed Databricks pipeline task issue

* updated test cases

* addressed review
2024-06-27 12:07:04 +05:30
harshsoni2024
3f5bc1948d
Fix #14676: Athena S3 Lineage (#16426)
* get table ddl for athena tables

* changes in method to get all table ddls

* external table/container lineage for athena

* column lineage for external table lineage

* unittest for athena

* pyformat changes

* add external table lineage unit test

* fix unittest with pydantic v2 changes

* fix unittest formating

* fix code smell
2024-06-26 19:53:36 +05:30
harshsoni2024
afec7703cc
Fix #11888: Apache flink connector (#16601) 2024-06-26 19:21:16 +05:30