8 Commits

Author SHA1 Message Date
Mayur Singal
ef67474310
Data Insights Refactor (#16532)
---------

Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
Co-authored-by: Pablo Takara <pjt1991@gmail.com>
Co-authored-by: Chira Madlani <chirag@getcollate.io>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2024-07-30 17:38:43 +05:30
Pere Miquel Brull
c742835766
Auto Tagger Application - Preparing the Ingestion Framework (#13862)
* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Handle app runner

* Prepare the skeleton for generic app registration

* Prepare the skeleton for generic app registration

* Allow deployment

* Fix PII APP

* Fix lint

* Fix PII APP

* Fix PII APP

* Prepare config-based external apps

* Prepare config-based external apps

* Fix lint

* Prepare config-based external apps

* Fix DI errors

* Amend comments
2023-11-13 08:58:38 +01:00
Ashish Gupta
7c181a5a89
Dbt worflow (#9302) 2022-12-16 17:00:28 +05:30
Pere Miquel Brull
f33003485e
ES reindex Airflow workflow (#8723)
* Added support for 'Metadata Service' connectors

* Fixed failing unit tests

* Removed delete button for 'Metadata Service' connector of type OpenMetadataServer

* Code optimization for SelectServiceType component

* Fixed errors arrived due to backend changes related to OpenMetadata Services

* init reindex

* Added additional step for adding Metadata to ES configs while creating ingestion pipeline for Metadata Service

* Add ES reindex

* Add ES reindex

* Format

* Format

* Rename service

* Pick service name from IngestionPipeline

* fix ui conflicts

Co-authored-by: Aniket Katkar <aniketkatkar97@gmail.com>
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-14 18:59:56 +01:00
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Shailesh Parmar
3b67cc824d
Data qaulity add test's workflow (#6957)
* initial setup for data quality form

* added stepper and form component

* added select/add test suite step form

* added form for table test and cron step

* added data quality table test form flow

* bug fix for profiler

* added column test form

* render right panel information dynamically

* updated test as per new changes

* updated data test id

* Fixed ingestionPipeline

* Fixed pytest + python format

* miner fix, added sql editor

* removed filter for duplicate test check

* miner fix

Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2022-08-29 17:26:59 +05:30
Pere Miquel Brull
99f0c96df4
Fix #6278 - Lineage workflow + Usage workflow (#6568)
* Fixing conflicts

* Fixing Conflicts

* Fix snowflake

* Prepare snowflake and redshift

* snowflake use a single query

* Add bigquery

* Add clickhouse

* format

* Add query log lineage source

* Add filters to core

* Prepare mssql usage

* Update schemas

* Add example workflows

* Prepare lineage deployment

* Remove lineage

* Use filters

* Prep lineage DAG

* Prepare Lineage Ingestion

* Test lineage workflow

* format

* Comments

* fixed tests

* Revert "fixed tests"

This reverts commit 0e846c7c4806098e4adf7c071d133d1df68aedf5.

* fixed get connection method

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-08-12 13:24:58 +05:30
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362)
2022-07-28 14:46:25 +02:00