743 Commits

Author SHA1 Message Date
Mayur Singal
e9992a52a8
Fix #1604: Add Spline Pipeline Connector (#11562)
* Fix #1604: Add Spline Connector

* Add tests & grammer validation

* Spline UI Changes & Docs

* fix pipeline workflow doc

* chore: use common field for dbService name

* chore: use const for beta services

* chore: add service icon

* Update ingestion/src/metadata/ingestion/source/pipeline/spline/metadata.py

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-05-19 14:46:32 +05:30
Pere Miquel Brull
50ad38ea0f
Fix #11548 - Secrets Managers comms with OMeta (#11602)
* Remove secretsManagerCredentials from backend

* Remove secretsManagerCredentials from backend

* Add secrets manager loader

* Load SM in the ometa client

* Fix tests

* Fix tests

* Fix Lint

* Mock AWS region

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-05-19 09:43:11 +02:00
Chirag Madlani
763b96b8fe
bump(ui): rjsf to v5 (#11549)
* bump(ui): rjsf to v5
It will fix validation and extra parameter issues

* address comments
fix code smells
fix cypress failure

* revert schema files

* bigquery fix
sonar cloud fix

* fix big query

* fix superset

* fix cypress

* fix cypress for big query

* fix failures

* fix java issue

* revert connectionBasicType.json
create custom SupersetConnection.json

* fix Tableau service issues #11637
2023-05-18 08:03:54 -07:00
Mayur Singal
e4997c3749
Fix #11571: Support custom database name for glue (#11631) 2023-05-18 14:16:56 +05:30
Onkar Ravgan
e6a7ffc57c
Fixes: 11540 Added api_version and domain fields to Salesforce (#11618)
* Added sales new fields

* fixed image after merge

* Fix Failing Test

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-05-18 10:53:44 +05:30
Akash Jain
92d5bfa94e
fix: versions in main branch (#11478)
* fix: versions in main branch

* Prepare main branch for next release 1.1.0

* prepare main for latest release 1.0.1
2023-05-15 10:42:29 +05:30
Suresh Srinivas
e72125c3a1
11579 task - Change schema regex to allow parenthesis in the entity name (#11580) 2023-05-14 06:58:38 -07:00
Mohit Yadav
d12d32bf65
Add settings to Db for Custom Logo (#11583)
* Add settings to Db for Custom Logo

* Add test for Settings
2023-05-12 23:34:02 +05:30
Mayur Singal
ef7b02529d
Remove sqlalchemy-ibmi from db2 dependency (#11553)
* Remove sqlalchemy-ibmi from db2 dependency

* remove from json schema

* add migration

* update what's new
2023-05-11 15:03:26 +02:00
Suresh Srinivas
14b622efd0
11287 Task - Add retention period and customer properties to Database, Schema, and Tables (#11511) 2023-05-10 16:08:21 -07:00
Louie
5bce6447f6
Fixed typo SMPTS to SMTPS (#11487)
* Fixed typo SMPTS to SMTPS

* sonar lint fixes

---------

Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-05-09 11:22:20 +05:30
Nahuel
b2f2320145
Fix#10584: Update Lineage API to support lineage from table to dashboard data model (#11489)
* Fix: Update Lineage API to support lineage from table to dashboard data model

* Minor change
2023-05-08 22:02:11 -07:00
Pere Miquel Brull
80af2b20bb
Centralize SSL Schemas & Add SSL for Airflow REST (#11363) 2023-05-08 21:05:10 +02:00
Mohit Yadav
498873c011
Data insight reporting + extras (#11317)
* Refactor Data Insight Chart for Reusability

* Data Insight Reporting

* Add Schedule Info

* Add Error Handling

* Add Profile Field

* Remove Endpoint as required field

* Missing Language

* Reformat

* Remove Task and Test Result Settings

* Fix test

* fix: typescript errors

* fix: send alertType for subscriptions request

* feat: add alert data insights report page

* Fix test

* fix: alert type typescript error

* fix: loading flicker on alerts page

* chore: add placeholders for data insight report alert page

* fix: alert icon styling

* fix: confirmation is not showing up for delete operation for subscription.

* fix: delete operation is not working from alert details page

* fix Activity Feed Errors on update

* fix: test

* chore: change the description text area to the markdown editor

* Data Insight Reporting should be created on system startup

* Restrict DataInsight Report to only one

* Review Comments

* feat: show the default DataInsightReport data

* sonar fixes

* sonar fixes

* deprecate ConfirmationModal

* trigger should be updated

* feta: add edit flow for data insight report

* remove custom option for schedule info

* chore: update data insight report icon

* test: add unit test for add data insight alert form

* fix: update cypress tests

* update local message

* test: add unit tests

* move sql changes to a different file

* spacing issues

* add tag provider in createRequest

* chore: send provider in create and update alert request.

* fix: code smells

* fix: comment typo

* fix: permission issue for event subscription

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-05-08 16:54:45 +05:30
Mayur Singal
e20f75fa54
Fix #11468: Fix salesforce scheme issue (#11474) 2023-05-08 14:02:17 +05:30
Nahuel
b22192575d
Fix#11216: Use Enum for SASL Mechanism for Kafka and Redpanda connections (#11385)
* Fix#11216: Use ENUM for SASL Mechanism for Kafka and Redpanda connections

* Minor change

* Address PR comments

* Fix postgres migration
2023-05-08 09:35:49 +02:00
Mohit Yadav
1b7fb192d0
[Reindexing] Fixes and Addition for Reindexing (#11447)
* Remove Entries on Reindexing

* Fix Reindexing Count Issues

* Reindexing Cursor Print in case of error

* Add afterCursor to Request

* add comment for function
2023-05-05 14:19:51 +05:30
Nahuel
1ec6e5e285
Fix#11311: Add IBM dependency for i Series in DB2 connector (#11381) 2023-05-02 15:50:39 +02:00
Suresh Srinivas
fd243dc5cb
FeedDAO cleanup (#11369)
* Cleanup FeedDAO implmentation

* Use FeedFilter to avoid repeated code for forward/reverse pagination
2023-04-30 16:50:29 -07:00
Mayur Singal
fd5f63fb58
Fix MSSQL connection with pyodbc scheme (#11304)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-04-27 07:25:10 +02:00
Mohit Yadav
4dd395e90a
Add Email Field + Subscription to Profile to Teams (#11280)
* Add Email Field to Teams

* Add Subscription Config to Profile

* Added Test Cases for Profile and Team Email
2023-04-26 20:26:32 +05:30
Pere Miquel Brull
d3d523e96d
Ingestion md docs review (#11219)
* Update workflow docs

* Remove duplicate key

* Update Custom connector docs

* Update Domo connector docs

* Dashboard docs updates

* Some databases docs updates

* Finish db docs updates

* Remove Pulsar

* Messaging docs

* Metadata docs

* ML docs

* S3 docs

* Fix rendering

* Update title and description of the databaseSchema

* Pipeline Service docs

* remove pulsar from tests

* Format

* Fix test

* Remove pulsar

* Remove pulsar
2023-04-23 18:43:46 +02:00
Mayur Singal
d0781dcc07
Remove partition fields from Bigquery (#11195)
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-23 06:38:17 +02:00
Mayur Singal
dc7841cf40
Make Database Required for AzureSQL (#11199) 2023-04-22 20:31:32 +02:00
Mayur Singal
da2f03ca50
Salesforce docs & remove unnecessary fields (#11207) 2023-04-22 18:32:32 +02:00
Nahuel
93c0605012
Doc: Add email UI helper (#11206)
* UI: Update docs for email configuration

* UI: Update docs for email configuration

* UI: Update docs for email configuration
2023-04-21 20:41:24 +00:00
Mayur Singal
e494cdfb65
Ingestion Docs: Superset, Snowflake, Bigquery, Azure SQL (#11180)
* Snowflake, Bigquery & Azuresql docs with superset migrations

* Superset Docs

* Auto PII is not a part of ingestion

* bigquery docs

* Azure SQL Docs

* Snowflake Docs

* Superset Docs

* Url to URL

* replace hr with ---
2023-04-21 18:23:45 +05:30
Pere Miquel Brull
a4956a52cb
Vertica, Custom Connectors and Delta Docs + Custom Storage Connector (#11166)
* Deltalake docs

* Deltalake docs

* Add Vertica docs

* Docs & Custom Storage Connector
2023-04-21 12:24:35 +02:00
Teddy
3cff3b7ef9
doc: add missing type in connection schema (#11186) 2023-04-21 10:32:01 +02:00
Keith Sirmons
97b58c65f5
Impalaconnection (#11151)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala

* added cast to decimal for column when running average in mean.py

* fixed lint error

* fixed ui ordering of precision and scale.
Precision should be ordred in front of scale since the precision is set first in decimal data types

* first pass for impala connector

* updated default auth_mechanism to be one of the enum values.

* updated UI documentation to match fields for the impalaconneciton.

refined impalaConnection to bring use_ssl to a boolean instead or relying on an extra connection option being manually added.

Removed reference to hive for type mapping

added impala to the pip setup

* py_format updates

* removed print statement

* Lints and fixes

* Updated database documentation to follow new style

* Flag as BETA

* Remove tests

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-21 09:57:13 +02:00
Nahuel
8184516f80
Fix: Align includeOwner option with the rest of options to include entities (#11160) 2023-04-20 16:01:54 +02:00
Pere Miquel Brull
91cd1491ee
Add Athena, Lineage and Usage docs & Fix Athena UI Lineage and Usage workflows (#11148)
* Athena docs

* Lineage and Usage docs

* Missing section close

* Fix Athena Model
2023-04-20 06:31:53 +02:00
Suresh Srinivas
d3db1cb95c
Fixes #11507 - Non-admin user can't join open teams (#11131) 2023-04-19 07:11:04 -07:00
NiharDoshi99
1862ba2ba4
Changing behaviour for owners same as description for dashboards (#11118)
* changing behaviour for owners same as description

* fix typo
2023-04-19 12:31:56 +05:30
Hawkins
4e63d6809f
ISSUE-11035: add ZH ElasticSearch index mapping (#11042)
Co-authored-by: wanghj <wanghaojie12345>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-04-17 09:16:09 -07:00
Pere Miquel Brull
5152db488d
Add partition columns details (#11062) 2023-04-14 13:06:56 +02:00
Onkar Ravgan
bc6ce22a2b
Added oneof selection for tableau auth types (#11049)
* Added tableau oneof fields

* Fixed pytests

* fixed field in test

* Handle tableau auth converter

* Fixed java tests and imports

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-14 13:49:36 +05:30
Mohit Yadav
05ae862f15
Fix for issue leading to vulnerability flags + extras (#11044)
* Rest passwrod link update + Fix the socket connection error issue RFC

* prevent logging of stackTrace

* add default created ingestion pipelines

* provider added

* provider added
2023-04-14 11:31:36 +05:30
Pere Miquel Brull
4da9aa4984
Add location migration queries (#11030)
* Add location migration queries

* add missing mode

* Added impala auth modes

---------

Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
2023-04-13 18:05:44 +02:00
Pere Miquel Brull
dcbb77f46c
Looker - Update GitHub creds for UI form (#11019)
* Update GitHub creds for UI form

* Add filters
2023-04-13 06:05:02 +02:00
Ayush Shah
92f0f0df23
fix 11021: remove repeating item title (#11022) 2023-04-13 00:48:08 +05:30
Nahuel
673573a512
Fix#10584: Add filters for data models (#11008)
* Fix#10584: Add filters for data models

* Minor UI error

* fix unit test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-04-12 17:36:01 +05:30
Onkar Ravgan
1b930fa6f7
Added enum for ssl and auth (#11016) 2023-04-12 10:57:31 +00:00
Pere Miquel Brull
b5cb1d464a
Deprecate location and old storage service (#11004)
* Deprecate location and old storage service

* Format

* Fix test

* Refactor

* Clean location

* Rename object store to storage

* Rename object store to storage

* Rename object store to storage

* Format

* Format

* Refactor object store for storage

* Refactor object store for storage

* Rename object store to storage

* Fix test

* Fix test

* Format

* chore(ui): change Objectstore to  Storage

* Fixes

* Fix test

* Remove storage service from Glue cypress

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-04-12 11:44:46 +02:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning (#10603)
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Deepa Rao
147b7c551e
Fixes #10854 - Create/Update Chart Not working as Expected (#10991) 2023-04-11 06:28:01 -07:00
Pere Miquel Brull
09b283818d
Rel to #10927 - Looker DataModel (#10945)
* Organise calls

* Prepare skeleton

* Add looker model handling

* Parse files as sql

* Handle labels

* Linting

* Format

* Fix version

* Also check the API for explore lineage
2023-04-11 08:44:00 +02:00
Mohit Yadav
ee4f1b4e51
Add Stop Functionality For Reindex (#10971)
* Add stop Reindexing Job

* Languages

* Add Status

* Add Status

* Add messages
2023-04-10 13:15:44 +05:30
Ashish Gupta
c997d8c80b
Feat #10584 : Support Activity tab for DataModel Entity (#10941)
* Supported DataModel in Dashboard Page

* url fqn changes

* Support Activity tab for DataModel Entity

* Remove dataModel resource descriptor

* Fix tasks for dashboard data models

* changes as per comments

* minor fix

* changes as per comments and minor improvements

---------

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-04-06 23:05:40 -07:00
Pere Miquel Brull
cdd473c3b7
Add ingestionIpInfoEnabled configuration (#10870)
* Add ingestionIpInfoEnabled configuration

* Format

* hide ip address if api response if not 200

* removing dependancy on ip-modal and checking default state

* updated status code for ip to 204 to check default state

* fixed failing cypress

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2023-04-06 16:27:07 +02:00