1614 Commits

Author SHA1 Message Date
Pere Miquel Brull
47d814d059
Fix typo (#9969) 2023-01-28 18:39:59 +01:00
Nahuel
da1f911581
Doc: Add missing source config types in connectors config examples (#9955) 2023-01-27 15:30:48 +01:00
Noe Alejandro Perez Dominguez
d8674bd7f8
Deduplicate stl_querytext rows, fixes #8867 (#9870)
* Deduplicate stl_querytext rows, fixes #8867

* Update queries.py

Filter down result-set by joining with queries

* Reduce line character length
2023-01-27 10:36:49 +01:00
Pere Miquel Brull
4d474ab063
Add vertica e2e and fix checks (#9947)
* Add vertica e2e and fix checks

* Add missing task_id param
2023-01-27 10:22:12 +01:00
Nahuel
19f034eec4
Doc: Add Kafka troubleshooting section (#9941) 2023-01-26 18:52:06 +01:00
Mayur Singal
3f9364d8a4
Fix #9923: Fix usage for tables with same name (#9925)
* Fix #9923: Fix usage for tables with same name

* Fix #9923: Fix usage for tables with same name

* Update ingestion/src/metadata/ingestion/bulksink/metadata_usage.py

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-01-26 15:53:41 +00:00
Pere Miquel Brull
4650a453e1
Prepare Vertica Lineage and Usage (#9906)
* Prepare Vertica Lineage and Usage

* Simplify db usage

* Linting

* Revert postgres changes

* Revert postgres changes

* Add vertica flags
2023-01-26 14:34:33 +01:00
Pere Miquel Brull
c368116697
Add dbt cloud URL as an input parameter (#9911)
* Add dbt cloud URL as an input parameter

* support dbt cloud url from UI

* Flag dbtCloudUrl as required

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-01-26 09:36:03 +01:00
Milan Bariya
49d48e0546
Update: Trino, Presto Logic and Doc (#9859)
* Update: Trino, Presto Logic and Doc

* Update: Trino, Presto Logic and Doc

* Update permissions in Doc

* Update openmetadata-docs/content/connectors/database/trino/index.md

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>

Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-01-25 19:20:06 +05:30
Mayur Singal
dd82fad5c4
Fix sample data re-ingestion for upgrade test (#9905) 2023-01-25 17:44:18 +05:30
Milan Bariya
176111aa27
Added databricks in lineage model (#9887) 2023-01-25 16:06:48 +05:30
Mayur Singal
74d2132883
Supserset Connection Changes Doc (#9893) 2023-01-25 11:06:00 +05:30
Mayur Singal
b3f9e3ba97
Update snowflake Requirement Docs (#9900) 2023-01-25 11:05:14 +05:30
Pere Miquel Brull
bce8574f98
Dashboards and charts use the ID as name (#9810)
* Update dashboard names

* Fix: metabase lineage

Co-authored-by: Nihar Doshi <nihardoshi16@gmail.com>
2023-01-24 21:50:35 +05:30
NiharDoshi99
33090c149e
Fix: superset dummy credentials update (#9883) 2023-01-24 07:55:16 +00:00
Mayur Singal
d0191da8ec
Fix domo pipeline test connection (#9863) 2023-01-24 12:23:47 +05:30
Ayush Shah
d15d96ef1a
Fix Mapper Parser issue (#9865) 2023-01-24 11:21:50 +05:30
Pere Miquel Brull
f2fb0521c2
Update airflow loggers and rename ometa loggers (#9868)
* Update airflow loggers and rename ometa loggers

* ANSI print to logger

* Remove colored logging from tests

* Merge ometa_logger into the one used in loggers class

* linting

* linting

Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-01-23 16:28:17 +01:00
Teddy
871cd0414e
Fix Partition for tables with profile sample + added mssql e2e tests (#9862)
* fix: partition for sampled data

* fix: added MSSQL E2E cli test

* fix: added secerets varibale to action file

* fix: python checkstyle
2023-01-23 14:16:44 +01:00
Mayur Singal
5145dced25
Datalake parquet files fix (#9860) 2023-01-23 17:44:34 +05:30
Pere Miquel Brull
16a1b2c8be
Vertica comments, dbs, profiler and docs (#9845)
* Vertica comments, dbs, profiler and docs

* Revert metabase changes

* Format

* Fix median
2023-01-20 20:36:03 +01:00
Teddy
7d0eeeb802
Fix datetime fn mysql (#9825)
* fix: mysql datetime function

* empty commit to re-run tests
2023-01-20 14:33:55 +00:00
Nahuel
254ee9a186
Fix#9460: Avoid reuse inspector to get view definition (#9821)
* Avoid reuse inspector to get view definition

* Update openmetadata-sqllineage version
2023-01-20 13:54:41 +00:00
Mayur Singal
939adf887b
Fix #8973: Superset fetch metadata from DB (#9645)
* Docs #8973: Superset with SSO docs

* Superset support for mysql & postgres db

* remove unwanted field from yaml

* Fix pylint

* Fix superset tests

* Fix sample data connection

* ci fix & review comments

* cypress & defualt provider fix
2023-01-19 16:39:43 +01:00
Nahuel
ddff6e2875
Fix: Replace sqllineage with openmetadata-sqllineage (#9800)
* Replace sqllineage with openmetadata-sqllineage

* Fix checkstyle and failing test

* Move logic to retrieve dialect of a service type into a class

* Improve py-check message when it fails

* Updated mapper

* Update code after merge
2023-01-19 14:56:29 +01:00
Teddy
fd4d39ac05
fix: max value for column between (#9820) 2023-01-19 12:13:35 +00:00
Ayush Shah
14100d089f
Modify Charts (#9814) 2023-01-19 12:03:21 +00:00
Pere Miquel Brull
c6cf9ae764
Fix region name (#9811) 2023-01-19 12:49:18 +01:00
Nahuel
04dae8bc75
Revert "Fix #8884 & #9460 - View lineage happens in the lineage workflow (#9796)" (#9817)
This reverts commit e88fe2f559989dbb1c34662abcdc35f235adc8c4.
2023-01-19 11:56:21 +01:00
Pere Miquel Brull
b0058bdb65
Fix #9707 - Install instantclient and enable thick mode (#9761)
* Install instantclient

* Prep instant client

* Update docs

* Amend comments
2023-01-19 08:08:56 +01:00
Mayur Singal
f58765b7cc
Fix #6577: Optimize metadata reads (#9759) 2023-01-19 06:29:14 +00:00
Pere Miquel Brull
e88fe2f559
Fix #8884 & #9460 - View lineage happens in the lineage workflow (#9796)
* View lineage happens in the lineage workflow

* Format

* Remove table views from context

* lint
2023-01-18 18:32:40 +01:00
Ayush Shah
02147b6502
Fix: Quicksight Dashboard and chart Metadata Ingestion (#9786)
* Fix Quicksight json schema and errors

* Fix Quicksight lineage

* add pagination

* Generalize Pagination

* Fix Pylints

* update max results

* fix tests

* Add Pydantic Models for QuickSight resp

* Add fqn builder

* Fix Py lInt

* Fix Tests
2023-01-18 20:37:41 +05:30
NiharDoshi99
f2649041f2
Fix: atlas test connection (#9791)
* Fix: atlas test connection

* Fix: docs changes

* Fix: docs changes

* Fix: python checkstyle

* Fix: python test
2023-01-18 20:16:07 +05:30
Milan Bariya
7f111584b8
Deprecate OMetaDatabaseAndTable (#9760)
* Deprecate OMetaDatabaseAndTable

* Change Based On Comments

* Change Based On Comments

* Fix Pytest

* Fix Pytest
2023-01-18 14:29:32 +01:00
Onkar Ravgan
043f1e0601
Dbt e2e test (#9767)
* Added base files

* dbt e2e tests

* Added tag condition based on UI feedback

* Added owner and lineage tests

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-18 14:29:16 +01:00
Abhishek Pandey
6d7acdc140
Issue 9542 - Replace the default name for MySQL db (#9687)
* database-field-added-for-custom-database-name

* database-field-added-for-custom-database-name

* database field description-added

* databaseName-field-updated-in-connection-json

* database-name-field-updated-in-salesforce

* databaseName-description-updated

* database-name-logic-updated-in-common-db-source
2023-01-18 14:28:40 +01:00
Onkar Ravgan
08a4d33c5f
Powerbi pagination fix (#9784)
* Fixed powerbi pagination logic

* Added yaml changes
2023-01-18 17:17:13 +05:30
Suresh Srinivas
ac157a8f0b
Fixes #9618 Allow apostrophe in entity names (#9642)
* Fixes #9618 Allow apostrophe in entity names

* Add support for space in entity names (with exception) and column names

* Add regex checks to entity names, table column names to allow only required characters

* fix: allowed characters in entityName

* Add all the allowed special characters in the entity name in tests

* Allow '&', '/', ':' only for the table

* Override basic entityName by defining entity specific entityName for Python code gen

* fix: dagster migration + json schema path

* fix: python style check

* Fix test failure

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-01-17 11:34:29 -08:00
NiharDoshi99
2954f17832
Fix: Domo dashboard issue (#9743)
* Fix: Domo dashboard issue

* Fix: added execption handling

* Fix: changed as per comment
2023-01-17 21:27:15 +05:30
Ayush Shah
f11cdbfab1
Profile sample rows + Randomization of chunks for datalake (#9595) 2023-01-16 22:17:46 +05:30
Teddy
45dde7a4bb
fix (profiler/dq): partitioning logic (#9731) 2023-01-14 22:35:51 +01:00
Mohit Yadav
b0ace9446d
remove computePercentile from table usage summary (#9681)
* remove computePercentile from table usage summary

* Fix: removed the compute percentile and added at the end

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Nihar Doshi <nihardoshi16@gmail.com>
2023-01-13 17:16:15 +01:00
NiharDoshi99
029dbe892e
Fix: added test case for atlas (#9678)
* Fix: added test case for atlas

* Fix: resolved conflict

* Fix: changing back neo4j to old version

* Fix: changing back neo4j to old version

* Fix: changes as per comment

* Fix: changes as per comment

* Fix: python checkstyle
2023-01-13 16:07:29 +05:30
Corey Zuares
c3197beaf4
Extra headers for metadata API (#9321)
* Configurable Extra Authorization Header

- Adds two new header that can be set via env-var
  (`OMETA_HEADER_EXTRA_AUTH_NAME` and `OMETA_HEADER_EXTRA_AUTH_VALUE`)
- By default, it copies the Authorization header value to the extra auth header
- If a value is specified in the corresponding env-var, that value is
  used instead of the existing token

The primary use case is for Google Cloud's Identity Aware Proxy (IAP) which allows secure access to private services but removes the Authorization header in the process. The `Proxy-Authorization` header gets forwarded along to the service as the `Authorization` header for the internal service.

* Use openMetadataConnection schema instead of env vars

* Disable too-many-locals for _request function

* Fix dict merge
2023-01-13 09:59:44 +01:00
Teddy
6fb47061f8
fix (profiler): column compilation (#9705) 2023-01-13 08:12:04 +01:00
Milan Bariya
bb41fec08f
Add: Function for Postgres Tags (#9683)
* Add: Function for Postgres Tags

* Update: Query for generate fqn
2023-01-12 09:46:08 +01:00
Mayur Singal
c4a7965ffc
Fix #7156: Add Sample Data for Kafka and Redpanda (#9472)
Co-authored-by: Ayush Shah <ayush@getcollate.io>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-12 09:43:25 +01:00
Mohit Yadav
f2e54fe7b9
Fix Topic index to map properly (#9688)
* Fix Topic index to map properly

* fix python side
2023-01-11 18:59:05 +00:00
Teddy
41ce2406a4
fixes #9673 - Oracle casting error (#9682)
* fix (profiler): oracle profiler datatype error

* fix (profiler): linting

* fix: reorder connection type to default to db schema
2023-01-11 19:46:10 +01:00