229 Commits

Author SHA1 Message Date
Pere Miquel Brull
2c3ff8dc08
Prep v1.1.7 migrations to address test cases & suites (#13345)
* Prep v1.1.7 migrations to address test cases

* get or create executable suite

* Format

* Fix tests

* Add postgres
2023-09-27 11:49:21 +02:00
Sriharsha Chintalapani
c5a4c5957e
Add 1.1.6 migrations dir (#13305) 2023-09-22 09:45:00 +02:00
Pere Miquel Brull
18a4513ccc
Fix #13237 - Rename to instanceDomain and test DomoDashboard charts (#13247)
* Rename sandboxDomain to instanceDomain

* Test Get Charts in DomoDashboard

* Fix schemas

* Fix test

* Fix test

* Rename to Auto Tag PII

* Fix query test

* Fix query test

* Fix query test
2023-09-19 14:14:04 +02:00
Sriharsha Chintalapani
c2ed4f422f
Fix LifeCycle inconsistencies in Schema, make it common entity field (#13252)
* Fix LifeCycle inconsistencies in Schema; Add DELETE api

* set autocommit to true for non transactional

* make lifecycle common field for entities

* Add LifeCycle as common entity field

* Fix python life cycle code

* Fix search indexes

* remove unnecessary constant

* Add test back to entity resource test

* Fix lint

* Fix lint

* Fix lint

* Fix lint

* Add missing schema

---------

Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-09-19 14:03:57 +02:00
Pere Miquel Brull
22b0f44e38
Part of #12998 - Query Service & Lineage filter processed queries (#13215)
* Update mapping

* Prep

* prep

* Prep query migration

* Add query index deletion fix

* Docs and Maven CI fix

* Fix tests

* Add service filter

* Add query entity FQN col migration

* Fix lint

* supported serviceFQN in query api

* Prep repo

* Prep ES query search

* Do not recompute lineage

* Format

* Fix test

---------

Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2023-09-19 07:37:47 +02:00
Sriharsha Chintalapani
047ab980cc
only add collation to hash columns (#13201) 2023-09-15 12:49:11 +05:30
Teddy
4012f25d64
fix: add test case result extension for ts migration (#13195) 2023-09-14 16:40:11 +02:00
Sriharsha Chintalapani
1af4941797
Minor fix: Do not re-run migration for properly formed native test su… (#13124)
* Minor fix: Do not re-run migration for properly formed native test suites

* rename 1.1.3 to 1.1.5

* rename 1.1.3 to 1.1.5 and add 1.1.4

* Fix table -> native testsuite migration
2023-09-12 11:43:10 -07:00
Pere Miquel Brull
f0995cbddc
Part of #12998 - Prep Stored Procedures Skeleton for Snowflake (#13121)
* Prep Stored Procedures Skeleton for Snowflake

* Update pylint and add migrations

* Fix test

* Reuse source url computation
2023-09-12 14:25:42 +02:00
Mohit Yadav
5f5aeef7f0
Transaction for Delete @UnitOfWork (#13115)
* Fix #12494: Jdbi Transactions opens a new connection per DAO causing recursive delete to not to rollback

* Fix #12494: Jdbi Transactions opens a new connection per DAO causing recursive delete to not to rollback

* rebase with main

* Fix styling

* Add jdbiUnitOfWork

* Improvements for Change Event Repository and missing Annotations

* Add connection commit/rollback

* push schemaChange fix

* Improvements for Change Event Repository and missing Annotations

* Pass wrapped collectionDAO everywhere

* Pass wrapped collectionDAO everywhere

* Separate event handlers transactions

* Fix checkstyle

* Wrap PUT , POST, and DELETE in transactions

* Add Patch for transactions

* Add primary key to tag.id

* Proxy internal @CreateSQLObjects

* Fix exception handling in ManagedHandleInvocationHandler

* Java typo

* Update schema files

* Checkstyle fix and conflicts resolve issue

* Remove @JdbiUnitOfWOrk from Feed Repository

* remove unnecessary @JdbiUnitOfWork annotation

* Test Failures fix

* Test Failures fix

* Increase wait time , changeEventDAO takes time due to handle.commit()

* commit change

* No need to use commit in ChangeEventHandler

* Add a lookup for avoiding transaction that are not started and should not be rolled back

* remove JdbiUnitOfWork.java

---------

Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-09-08 23:19:44 +05:30
Onkar Ravgan
d8e78a32d8
Added Life Cycle fields, API and ingestion integration (#13014)
* Added initial code

* temp commit

* Added snowflake changes

* Added postgres migration

* removed non req changes

* review comments one

* Added ometa tests

* lint fix

* fix lint naming

* fixed merge changes
2023-09-06 12:37:55 +02:00
Teddy
ab3042e8ee
Issue 12297 bis -- Delete Insert logic in the DI workflow (#13058)
* fix: updated ingestion to delete existing data for the ingestion day

* fix: added delete endpoint for the report data

* fix: added migration to delete duplicate data + added tables creation & data deltion logic in a transaction

* fix: made deletion SQL engine aware

* fix: added ES deletion logic back while we work on DI event publisher
2023-09-01 18:01:20 +02:00
Teddy
867e1c4946
Issue-12914 -- Split profiler, data quality and data insight table from entity_time_series_extension (#12991)
* fix: write testCase result testSuite to compute summary

* fix: empty commit to trigger CI

* fix: extracted report data out of entity_extension_time_series and into report_data_time_series

* fix: split profiler data  from entity_extension_time_series

* fix: split data quality data time series table

* fix: updated migration file
2023-08-28 08:14:15 +02:00
Teddy
101cd0ebac
Issue 8930 - Update profiler timestamp from seconds to milliseconds (#12948) 2023-08-25 08:47:16 +02:00
Sriharsha Chintalapani
19b5c946a7
Fix #12167: Support for Stored Procedures as another entity under Database Schema (#12999)
* Add Stored Procedure Entity

* Stored Procedure repository

* Stored Procedure repository

* Fix #12998: Support for Stored Procedures as another entity under Database Schema

* Fix #12998: Support for Stored Procedures as another entity under Database Schema
2023-08-25 08:14:30 +02:00
Pere Miquel Brull
10b1578eed
Fix airflow config retries postgres migration (#12988) 2023-08-24 14:42:43 +02:00
Pere Miquel Brull
4862b0df28
Fix #10344 - Use ingestionPipeline retries when deploying (#12926) 2023-08-23 14:52:54 +05:30
Pere Miquel Brull
d140883747
Prepare 1.1.3 migrations based on 1.1.2 (#12958) 2023-08-22 09:19:16 +02:00
Pere Miquel Brull
ccad601a4f
Fix postgres migration files (#12923) 2023-08-18 14:54:43 +02:00
Sriharsha Chintalapani
524e611edc
Deprecate Flyway and use Native migrations (#12865)
* Add ascii_bin for fqnhash, Add Constraints on timeseries

* Attemp of moving migrations to files

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Deprecate Flyway and use Native migrations

* Move upstream migrations to new path
2023-08-17 14:39:05 -07:00
Onkar Ravgan
3114ef9b5c
Add support for JWT auth in Trino (#12823)
* Added jwt field in trino

* fixed postgres migration

* fixed pytests
2023-08-17 07:22:58 +02:00
Sriharsha Chintalapani
7aaf654f01
Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch (#12782)
* Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch

* Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch

* Fix #12779: Add support for SearchIndexes for ElasticSearch and OpenSearch

* Rebase fixes with main

* Add Sample Data

* lint fix

* remove unused import

* Fix service count test

---------

Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2023-08-10 16:47:37 -07:00
Pere Miquel Brull
71df43fdab
Fix impala migration query (#12809) 2023-08-09 09:41:41 +02:00
Pere Miquel Brull
73b58011d4
Add v015 for 1.2.0 release (#12716) 2023-08-02 19:27:49 +05:30
Pere Miquel Brull
a68e12c7e1
Clean old test connections bootstrap (#12695) 2023-08-01 16:15:56 +02:00
Mayur Singal
5f07c6281a
Clickhouse: Make https protocol as permanent field (#12303) 2023-07-31 19:26:03 +05:30
Suresh Srinivas
8830d1707d
Initialize EntityResoruceTest support flags automatically (#12513) 2023-07-27 20:42:10 -07:00
mohitdeuex
8ab4072504 Change ignoreFileChksum to force 2023-07-26 16:40:52 +05:30
vanshika18
72e4d0070f
Support MSSLQ handle multiple databases with pyodbc connection (#12526) 2023-07-26 10:43:24 +05:30
vanshika18
bbe58b1b5f
Handle Table type for Snowflake Transient Tables (#12274)
* Handle Table type for Snowflake Transient Tables

* Clean the code by removing comments

* Optimize the code

* Optimize the code

* Clean the code

* Rename includeTempTables with includeTransientTables

* Rename includeTempTables to includeTransientTables

* Update Documentation and resolved migartion query error

* Resolved migartion query for postgres connection

---------

Co-authored-by: Vanshika Kabra <vanshikakabra@Vanshikas-MacBook-Pro.local>
Co-authored-by: Mayur Singal <39544459+ulixius9@users.noreply.github.com>
2023-07-13 19:25:46 +02:00
Onkar Ravgan
e200a42926
Removed datamodels references from dashboard and datamodels (#12371) 2023-07-13 14:26:28 +05:30
Pere Miquel Brull
1b7c5e3233
Fix #12321 - Deprecate impala scheme from Hive (#12327)
* Deprecate impala scheme from Hive

* Add migrations
2023-07-11 17:38:32 +02:00
Pere Miquel Brull
5ccddc3275
Add query_entity migration fix (#12340)
* Add query_entity migration fix

* Do not touch pipelines without tasks

* Move query entity queries

* Format
2023-07-10 16:08:32 +02:00
Suresh Srinivas
97140e13fd
12142.task2 - Add domain and data product properties to entities (#12255) 2023-07-03 15:42:35 -07:00
Pere Miquel Brull
a8396092d8
Fix DL migrations (#12268) 2023-07-03 22:25:12 +02:00
mohitdeuex
74ad065796 Add ignoreFileChecksum to script 2023-07-03 19:42:12 +05:30
Ayush Shah
93c405365b
Add Airflow Migrations for IAM RDS connection (#12259)
* Add Migrations

* Add not null

* Resolve comments
2023-07-03 15:16:30 +05:30
Suresh Srinivas
4d4993effc
12142.task1 - Add support for Domains (#12143) 2023-07-02 11:32:52 -07:00
Teddy
37a7cf99b6
fix: test defintion migration for parameter definition changes (#12207) 2023-06-28 16:02:49 +00:00
Ayush Shah
0894d339ac
Fix Migrations (#12146) 2023-06-26 09:36:45 +00:00
Sriharsha Chintalapani
33c752d2f5
interfaces for db migrations (#12057)
* interfaces for db migrations

* Add Data Migration for entities to 1.1

* Migration for FQN hash

* Added Tag Usage And Fixed some transaction issue in testing

* Added SERVER_CHANGE_LOG validations

* Update Path for files

* Added Postgres Migration

* Test Suite Migration [WIP]

* remove ingestion pipeline

* moved Migration to Table Initializer

* init Data source config in TableInitializer

* make checksum ignore configurable

---------

Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com>
2023-06-26 13:03:44 +05:30
Pere Miquel Brull
35cca0e178
Add automations workflow truncate migration (#12007) 2023-06-21 08:35:50 +02:00
Mayur Singal
c3cec54be9
Fix for #11807 Part 2: Add SourceUrl for table entity (#12013)
* Fix #11807 Part 2: Add SourceUrl for table entity

* address review comments: centralize sourceurl

* remove qlick

* pytest fix

* fix typo
2023-06-20 11:46:45 +02:00
Sriharsha Chintalapani
9dbaabad44
Adding MD5 hash to fullyQualifiedName and names to simplify DB indexes for lookups and increasing the size of FQN (#11960)
* Fix fqn hash

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* Add name & fqnHash and remove generated columns from top level

* merge commits

* merge commits

* merge commits

* Fix glossary import/export

* Fix BotResource Tests

* Fix BotResource Tests

* Fix Glossary Term tests

* Fix Glossary Term tests

* Fix Glossary Import/Export tests

* All backend tests are fixed.

* merge with main

* Fix tests and revert local changes

* Fix tests and revert local changes

* Fix NullPointerException for Glossary and Query

* Fix Tests

---------

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2023-06-19 15:43:05 +05:30
Ayush Shah
f80eaf3a26
Fixes 11068: mysql & postgres iam auth (#11937) 2023-06-16 13:18:12 +05:30
Mayur Singal
82a0222257
SourceUrl changes for dashboard, pipeline & chart entities (#11991) 2023-06-15 14:44:48 +05:30
Pere Miquel Brull
1aae06840a
Align bootstrap after 1.0.4 patch (#11932)
* Align bootstrap after 1.0.4 patch

* Align v012 script
2023-06-10 18:38:05 -07:00
Pere Miquel Brull
b68d063694
Update Test Connection Definition FQN (#11861)
* Update Test Connection Definition FQN

* chore: use object literal for string concatenation

* Fix test

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Update Test Connection Definition FQN

* Fix test

---------

Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2023-06-06 15:22:32 +02:00
Ayush Shah
65f370e4aa
Rename GCS to GCP (#11812) 2023-06-06 11:57:00 +05:30
Pere Miquel Brull
fdeea71671
Fix Looker explore git link & Add BitBucket reader (#11837)
* Add looker test connection step

* Add looker test connection step

* Update Credentials

* Fix explore link and add bitbucket reader

* Format

* Fix test

* Fix spline linting

* Fix import
2023-06-02 07:19:32 +02:00