2916 Commits

Author SHA1 Message Date
NiharDoshi99
d1996d4260
added docs for sqlite (#11232) 2023-04-24 11:07:15 +00:00
Teddy
017fbc6a32
fix: logic for number of profiled tables (#11222)
* fix: logic for number of profiled tables

* fix: python linting
2023-04-24 08:00:25 +02:00
Pere Miquel Brull
d3d523e96d
Ingestion md docs review (#11219)
* Update workflow docs

* Remove duplicate key

* Update Custom connector docs

* Update Domo connector docs

* Dashboard docs updates

* Some databases docs updates

* Finish db docs updates

* Remove Pulsar

* Messaging docs

* Metadata docs

* ML docs

* S3 docs

* Fix rendering

* Update title and description of the databaseSchema

* Pipeline Service docs

* remove pulsar from tests

* Format

* Fix test

* Remove pulsar

* Remove pulsar
2023-04-23 18:43:46 +02:00
Sriharsha Chintalapani
123758e21e
Fix #10964: update retentionSize based on retention.size in the topic config (#11217)
* Fix #10964: update retentionSize based on retention.size in the topic config

* Fix #10964: update retentionSize based on retention.size in the topic config
2023-04-23 08:36:58 +02:00
Sriharsha Chintalapani
9e259be44e
Fix #11214: Ingestion based elastic search index missing serviceType for MLModel and Container (#11215) 2023-04-23 07:30:37 +02:00
Mayur Singal
da2f03ca50
Salesforce docs & remove unnecessary fields (#11207) 2023-04-22 18:32:32 +02:00
Mayur Singal
cb5ee34a1b
Fix Lineage Via Table Entity Error (#11209) 2023-04-22 18:31:30 +02:00
Nahuel
ed1388827e
Doc: Add ElasticsearchReindex and Data Insight docs in UI (#11201) 2023-04-21 11:34:55 -07:00
Teddy
6e129c1e65
Issue 10805 Added Hive e2e (#11197)
* tests: Added E2E test for Hive + fix minor bug

* tests: ran python linting
2023-04-21 15:45:12 +00:00
Onkar Ravgan
4c3b20b910
Req Markdown docs: dbt, sagemaker, mode, powerbi, db2, dynamo, kinesis, fivetran (#11173)
* Added markdown req docs

* Added v1 docs

* Update openmetadata-docs-v1/content/v1.0.0/connectors/database/db2/index.md

fixed typo in db2 grant

Co-authored-by: Teddy <teddy.crepineau@gmail.com>

* typo fix v1 docs

---------

Co-authored-by: Teddy <teddy.crepineau@gmail.com>
2023-04-21 16:44:41 +02:00
Ayush Shah
a50c31539b
Fix HexByteString Issue, revert datatype change (#11145)
* Fix HexByteString Issue, revert datatype change

* Add E2E MSSQL Bit type
2023-04-21 10:08:27 +02:00
Keith Sirmons
97b58c65f5
Impalaconnection (#11151)
* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* updated metadata to work with the impala query engine.
Uses the describe function to grab column names, data types, and comments.

* added the ordinalPosition data point into the Column constructor.

* renamed variable to better describe its usage.

* updated profile errors.
Hive connections now comment columns by default.

* removed print statements

* Cleaned up code by pulling check into its own function

* Updated median function to return null when it is being used for first and third quartiles.

* removed print statements and ran make py_format

* updated to fix some pylint errors.
imported Dialects to remove string compare to "impala" engine

* moved huge comment into function docstring.
This comment shows us the sql to get quartiles in Impala

* added cast to decimal for column when running average in mean.py

* fixed lint error

* fixed ui ordering of precision and scale.
Precision should be ordred in front of scale since the precision is set first in decimal data types

* first pass for impala connector

* updated default auth_mechanism to be one of the enum values.

* updated UI documentation to match fields for the impalaconneciton.

refined impalaConnection to bring use_ssl to a boolean instead or relying on an extra connection option being manually added.

Removed reference to hive for type mapping

added impala to the pip setup

* py_format updates

* removed print statement

* Lints and fixes

* Updated database documentation to follow new style

* Flag as BETA

* Remove tests

---------

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-21 09:57:13 +02:00
Teddy
6f5d88ff63
tests: Added E2E tests for redshift (#11169) 2023-04-21 08:55:54 +02:00
Nahuel
8184516f80
Fix: Align includeOwner option with the rest of options to include entities (#11160) 2023-04-20 16:01:54 +02:00
Mayur Singal
7c1fe5202d
Fix Metabase E2E Tests (#11156) 2023-04-20 13:22:11 +00:00
Pere Miquel Brull
e453593760
Rel to #10805 - Ingestion docs glue & Kerberos Dep (#11157)
* Glue

* Add kerberos system dep
2023-04-20 10:56:46 +02:00
Pere Miquel Brull
91cd1491ee
Add Athena, Lineage and Usage docs & Fix Athena UI Lineage and Usage workflows (#11148)
* Athena docs

* Lineage and Usage docs

* Missing section close

* Fix Athena Model
2023-04-20 06:31:53 +02:00
Mayur Singal
dd754d586e
Metabase E2E Test & docs (#11126) 2023-04-20 00:50:23 +05:30
Onkar Ravgan
09fb69b68d
Fixed objectstore import (#11144) 2023-04-19 21:39:29 +05:30
Teddy
0f7d9699ad
Fix metrics filtering (#11149)
* fix: get column not filtering for metric types when profilerConfig with include columns is set

* fix: run python linting
2023-04-19 14:09:13 +00:00
Milan Bariya
66b25d2f30
Fix: Databricks usage issue (#11143) 2023-04-19 19:25:17 +05:30
Ayush Shah
ca861bc06e
Fixes #11137: Mssql Syntax Error + Arithemetic Error (#11138) 2023-04-19 15:08:12 +05:30
NiharDoshi99
1862ba2ba4
Changing behaviour for owners same as description for dashboards (#11118)
* changing behaviour for owners same as description

* fix typo
2023-04-19 12:31:56 +05:30
Onkar Ravgan
932ffa5fa0
Added Powerbi E2E and docs (#11130) 2023-04-19 11:49:20 +05:30
Pere Miquel Brull
a78a3b4734
Azure datalake metadata ingestion fixes (#11125)
* Add ADLS permissions

* Fix Azure DL ingestion

* Format

* enable decode for json

* fix gcs decode error

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-04-19 07:28:41 +02:00
Pere Miquel Brull
463f242d6b
Add S3 Storage docs & CI validation (#11120)
* Prep CI

* Update Athena

* Update docs for S3 Storage

* Add manifest information
2023-04-19 06:31:55 +02:00
Milan Bariya
7cbe48971d
fix: Redash lineage issue (#11098)
* fix: Redash lineage issue

* change based on comments

* change based on comments

* change based on comments
2023-04-18 21:22:17 +05:30
Teddy
07b74ce8e8
fix: remove count of results processed (#11115) 2023-04-18 14:00:14 +00:00
Teddy
97ff34967a
Fix histogram bin creation (#11105)
* fix: bin creation + pass full table name for mysql median computation

* fix: ran linting for python

---------

Co-authored-by: Nahuel <nahuel@getcollate.io>
2023-04-18 14:49:21 +02:00
Nahuel
c3bfd1310a
Fix: bump 'openmetadata-sqllineage' version to 1.0.4 (#11109) 2023-04-18 13:58:39 +02:00
Teddy
c415f04eb7
Issue 10805 profiler e2e mssql (#11096)
* feat: extracted builder logic into factory and cleaned up comments

* feat: added E3E logic for time based profiler partition + logic for testing detailed profiler results
2023-04-18 11:56:16 +02:00
Nahuel
22ce62e13b
Fix: Add Redash E2E test (#11091) 2023-04-18 12:52:38 +05:30
Mayur Singal
857ddeab1e
Ingestion: Metabase Unit Tests (#11080) 2023-04-18 09:08:17 +05:30
Mayur Singal
e7013b481a
Improve Redshift Query (#11082) 2023-04-18 00:51:06 +05:30
Teddy
b04f7225f8
fix: column retrieval for SNOWFLAKE (#11090) 2023-04-17 14:36:58 +00:00
NiharDoshi99
7e4b63997b
Changes for no columns and added owner for table (#11086)
* changes for no columns and added owner for table

* added pydantic model for owners
2023-04-17 18:42:30 +05:30
Onkar Ravgan
b82abd5047
Fixed tableau url (#11071)
* Fixed tableau url

* review comments and tests

* changes to remove host-port addition from the UI for dashboard and chart urls

---------

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
2023-04-17 11:20:11 +02:00
Mayur Singal
199fe8753a
Fix Top Level Imports (#11075) 2023-04-14 17:18:38 +00:00
Pere Miquel Brull
ae984d1808
Handle impala auth mechanism (#11074) 2023-04-14 18:04:42 +02:00
Teddy
a7d98dddda
Fixes #9632 - Add Profiler Support for BQ Arrays of Structs (#11059) 2023-04-14 19:29:26 +05:30
Nahuel
ea70580aff
Fix#10584: Tableau E2E and docs (#11054) 2023-04-14 19:28:31 +05:30
Nahuel
c596ad2006
Doc: Update Kafka troubleshooting section (#11067) 2023-04-14 13:35:58 +02:00
Pere Miquel Brull
5152db488d
Add partition columns details (#11062) 2023-04-14 13:06:56 +02:00
Pere Miquel Brull
8f3cf712ff
Catch ConnectionError (#11065) 2023-04-14 13:06:47 +02:00
NiharDoshi99
6b8b9a179b
airflow support for higher versions (#11001) 2023-04-14 16:04:42 +05:30
Onkar Ravgan
bc6ce22a2b
Added oneof selection for tableau auth types (#11049)
* Added tableau oneof fields

* Fixed pytests

* fixed field in test

* Handle tableau auth converter

* Fixed java tests and imports

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-04-14 13:49:36 +05:30
Teddy
77b94f9ebb
fix: rename tests endpoint to dataQuality/<specificity> (#10970) 2023-04-14 00:14:49 -07:00
Pere Miquel Brull
4c9690b0c2
Add Looker Sample Data (#11047) 2023-04-13 18:05:06 +02:00
Milan Bariya
dda5963e34
Data insights with metadata command-line (#11045)
* Data insights with metadata command-line

* Python Checkstyle

* Python Checkstyle
2023-04-13 13:42:07 +00:00
Milan Bariya
42de096f24
Fix E2E for Mysql (#11040) 2023-04-13 09:04:01 +00:00