3385 Commits

Author SHA1 Message Date
Mayur Singal
2f7a2193e0
Fix #12302: Add ElasticSearch Ingestion Source (#12892) 2023-08-18 15:10:31 +05:30
Onkar Ravgan
5b47fd4acf
Added source url to entities (#12901)
* Added source url to entites

* added support to create and update sourceUrl

* fixed pytests

---------

Co-authored-by: 07Himank <himank07mehta@gmail.com>
2023-08-18 10:17:38 +02:00
Ayush Shah
f9e3e1801f
Add External Table type support bq (#12889) 2023-08-18 13:45:14 +05:30
Onkar Ravgan
795294c87f
Added dbt multiple project support for s3, azure, gcs datalake sources (#12856)
* Added dbt multiple proj support

* added reader

* common method to group by dir

* added return type
2023-08-17 11:19:20 +02:00
Teddy
751c96e437
Fixes Issue #12297 - Added upsert logic for DI (#12466)
* fix: enforce DB upsert on DI ingestion

* fix: ran python linting

* feat: fix conflict and added BindFQN while removing hash

* feat: update aggregation from day to minute

* feat: ran python linting
2023-08-17 08:07:44 +02:00
vanshika18
d6e4169faf
Fixed #11519 Support Usage queries contain \n characters (#12676) 2023-08-17 11:21:12 +05:30
Onkar Ravgan
3114ef9b5c
Add support for JWT auth in Trino (#12823)
* Added jwt field in trino

* fixed postgres migration

* fixed pytests
2023-08-17 07:22:58 +02:00
Supan Shah
73bb5da6bf
ISSUE-12820: Sagemaket Metadata Success (#12843) 2023-08-16 13:02:55 -07:00
Pere Miquel Brull
a183fc67e2
Fix ADLS parquet reads (#12840)
* Fix ADLS parquet reads

* Generalize service methods

* Fix tests
2023-08-14 19:57:06 -07:00
Ayush Shah
1e2d93a604
Fix #12479: MariaDB Mysql Time Col profiler issues + fix e2e tests (#12868) 2023-08-14 17:28:09 +05:30
Teddy
8e4388c35e
Issue 11345 (#12859)
* feat: added serviceName dimension to entity report

* feat: fix python test
2023-08-14 08:05:14 +02:00
Ayush Shah
5ea4de646c
Fix Postgres Profiler for incompatible versions (#12795) 2023-08-11 15:32:43 +05:30
Mayur Singal
a17ac7bcbb
Fix: pyodbc type not supported for -150 (#12851) 2023-08-11 15:32:19 +05:30
Ayush Shah
8776069940
Fix Qliksense import (#12849) 2023-08-11 12:28:05 +05:30
Sriharsha Chintalapani
7aaf654f01
Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch (#12782)
* Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch

* Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch

* Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch

* Rebase fixes with main

* Add Sample Data

* lint fix

* remove unused import

* Fix service count test

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-08-10 16:47:37 -07:00
Pere Miquel Brull
d8e288626c
Storage connection naming, ADLS service and namespace packaging (#12810)
* Update storage connection naming

* Update storage services and remove typing of generic service

* Remove inits for namespace package

* Add test connection

* Revert "Remove inits for namespace package"

This reverts commit 89f1ae9ea8a633593776f51caf2586d2110d4636.

* Add enum

* Remove init to allow namespace packaging in some modules

* Remove init to allow namespace packaging in some modules

* Move examples and fix test

* Format

* Linting
2023-08-10 07:32:58 +02:00
Pere Miquel Brull
97c982b12a
Fix #12798 - Update Snowflake GetTables test query (#12804) 2023-08-10 07:30:34 +02:00
Pere Miquel Brull
e97d4befb1
Fix #12770 - Cleanup DL structure & Readers & Python 3.8 (#12776) 2023-08-09 16:07:16 +05:30
Sriharsha Chintalapani
d4ed941371
Fix #12783: Metadata Ingestion is dropping constraints just to add them again afterwards (#12784) 2023-08-08 13:51:10 +05:30
Pere Miquel Brull
a82c2f001c
Fix #9704 - Add inline TNS string (#12711)
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-08-07 11:04:13 +02:00
Pere Miquel Brull
6f89ca089d
Log the query in restore if there are any exceptions (#12761) 2023-08-07 11:03:47 +02:00
Mayur Singal
ed7f74c7b2
Improve exception handling for superset (#12753)
* Improve exception handling for supersert

* Update ingestion/src/metadata/ingestion/source/dashboard/superset/api_source.py
2023-08-04 11:31:40 +02:00
Mayur Singal
2f3b965f1c
Fix snowflake connection when no database passed (#12746) 2023-08-04 11:31:19 +02:00
Onkar Ravgan
146d8a5166
avro record dtype fix (#12750) 2023-08-04 14:27:29 +05:30
Akash Jain
fd0b1b9fed
fix: Align Docker Base Image to use airflow 2.5.3 (#12737) 2023-08-04 12:08:58 +05:30
Pere Miquel Brull
10f2567fe9
Fixes #12555 - Fix DL test suite (#12727)
* Fix DL test suite

* Fix linting
2023-08-03 11:48:22 +02:00
Matias Puerta
e203ece20c
[CHORE] Reduce docker image size by removing pip cache (#12708)
* [CHORE] Reduce docker image size by removing pip cache

* [CHORE] Reduce image size for ingestion/operators as well

* [CHORE] Reduce image size for CI
2023-08-02 14:36:27 +02:00
Mayur Singal
819bb27ee0
Fix airflow not ingesting pipelines (#12683) 2023-08-01 13:00:02 +05:30
Mayur Singal
5f07c6281a
Clickhouse: Make https protocol as permanent field (#12303) 2023-07-31 19:26:03 +05:30
Mayur Singal
8f6e5eed31
Automated Lineage by Query for Python SDK & CLI (#12672) 2023-07-31 19:25:41 +05:30
Ayush Shah
2e259a9899
Python Dependency Conflict fixes (#12667)
* Fixes Dependency conflict

* add fix to dep
2023-07-31 14:44:25 +02:00
Onkar Ravgan
795498c54a
Fixed athena import and lineage/usage validation (#12663)
* Athena fixes

* fixed to lower version
2023-07-31 12:37:48 +02:00
Mayur Singal
dacd13f6d0
Fix #11778: Add support for hive metastore ingestion (#12599) 2023-07-31 15:56:02 +05:30
Ayush Shah
e39918ffc3
Add fix for snowflake transient tables (#12662) 2023-07-31 15:07:05 +05:30
Onkar Ravgan
bf44c7996a
fixed nested records avro (#12633)
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-31 11:54:29 +05:30
Ayush Shah
b1308b76d5
oid ambiguitious error or not exists fixed (#12619) 2023-07-31 10:47:45 +05:30
vanshika18
a0dd821bd1
Support SingleStore metadata ingestion (JSON type fields) (#12623)
* Support SingleStore metadata ingestion (JSON type fields)

* Removing Collate Keyword all datatypes

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-29 12:27:05 -07:00
Onkar Ravgan
bad506b5d9
Backup CLI case sensitivity fix for postgres sql (#12588) 2023-07-28 11:27:19 +05:30
vanshika18
02ec8b0161
Support Azure Datalake Test Connection fixes (#12579)
* Support Azure Datalake Test Connection fixes

* Rename function name and remove unwanted argument

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-07-27 23:25:25 +05:30
vanshika18
3111f91a06
Support delete temporary credentials files of Google Cloud (#12590)
Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
2023-07-27 23:24:55 +05:30
Mayur Singal
c6de61f6bc
Add Lineage Parsing timeout and filtering condition (#12560) 2023-07-27 12:28:53 +05:30
Anuj359
1417621ddc
security-scan fix (#12585) 2023-07-27 11:41:25 +05:30
Pere Miquel Brull
6773541d15
[1.1.1] - Bump size for FQN (#12092)
* Bump size for FQN

* Bump table entityName size

* Bump table entityName size

* Fix table resource tests

* Remove pattern from fqn

* Remove pattern from fqn

* Remove pattern from fqn

* Generalize get_by_name in ometa client

* Generalize get_by_name in ometa client

* Format

* Fix test suite

* Remove limit from FQN max size

* Remove limit from FQN max size

* Add sample data

* Update lint names

* Add more sample data

* Bump column name size

* 1024 max FQN length

* 1024 max FQN length

* 1024 max FQN length

* Bump FQN
2023-07-26 12:36:42 -07:00
Sriharsha Chintalapani
67b68a70c9
Fix #7712: Align AuthProvider values in the JSON schema definition (#11508)
* Fix #7712: Align AuthProvider values in the JSON schema definition

* Fix ingestion bot initialization, create jwt token if there is no authMechanism pre-configured

* Fix ingestion bot initialization, create jwt token if there is no authMechanism pre-configured

* Handle new providers

* Lint

* Fix #7712: Align AuthProvider values in the JSON schema definition

* Handle new providers

* Fix ingestion bot initialization, create jwt token if there is no authMechanism pre-configured

* Fix ingestion bot initialization, create jwt token if there is no authMechanism pre-configured

* Lint

* fix rebase

* Fix test

* Fix test

* Fix Usage of Auth Provider in User Resource

* remove unused function

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com>
2023-07-26 11:34:58 -07:00
Onkar Ravgan
0f8ebf4afe
Fixed Tableau-Bigquery lineage issue (#12570) 2023-07-26 16:07:48 +05:30
Onkar Ravgan
30cdc840ab
Added excp msg (#12561) 2023-07-26 16:07:34 +05:30
Ayush Shah
307dcbf629
Add DateTimeRange support in postgres (#12583) 2023-07-26 14:31:48 +05:30
vanshika18
72e4d0070f
Support MSSLQ handle multiple databases with pyodbc connection (#12526) 2023-07-26 10:43:24 +05:30
Shailesh Parmar
19c3c016a1
ui: Display createdDateTime and sizeInByte profile metrics #12196 (#12568)
* ui: Display createdDateTime and sizeInByte profile metrics #12196

* Modify Sample Data

* updated profile matrix as per date object

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-07-25 23:02:19 +05:30
Ayush Shah
6c21da0d7b
Fixes 12224: Can't ingest binary sample data (#12564) 2023-07-25 12:33:26 +05:30