2916 Commits

Author SHA1 Message Date
Teddy
dcd0bbb566
Fixes 8470 -- Implements ES aggregation for web analytic data (#8566)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Added dailyActiveUsers aggregation

* Added page views entities aggregation and active users aggregation
2022-11-08 17:25:31 +01:00
Milan Bariya
4d16be2608
Fix: Hive connection issue (#8583)
* Fix: Hive connection issue

* Change based on comments
2022-11-08 16:22:37 +01:00
Teddy
2ffb88f6aa
Added bit -> binary column type conversion for mssql (#8581) 2022-11-08 16:21:55 +01:00
michizhou
da06d183db
feat: add QuickSight connector (#8442) 2022-11-08 19:54:49 +05:30
Mayur Singal
710e3d785f
Fix #8495: Update pipeline sources for pipeline as edge lineage (#8556) 2022-11-08 14:13:49 +05:30
Abhishek Pandey
73b370b5e2
schema-filter-added-in-datalake-for-bucket (#8516)
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
2022-11-08 10:57:16 +05:30
Mayur Singal
83102d6697
Fix #8455: fix dbt test errors when run not passed (#8545) 2022-11-08 10:50:27 +05:30
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events (#8528)
* Moved webanalytics type in its own folder

* Added data insight chart api endpoint

* Jave formatting

* Added resource descriptor

* Added metadata entity endpoint

* Added aggregation endpoint for dataInsight

* Fix tag name

* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight

* added domo to test subpackage

* cleaned up branch by removing commit from issue-8353 that were not merged in main

* Added web analytics data refinement

* Added get_status function

* Added from __futur__ for typing

* Fixed typos brought up during reviews
2022-11-07 17:08:20 +01:00
NiharDoshi99
1c1f8660f2
Fixed: dagster to connect to cloud (#8387)
* Fixed: dagster to connect to cloud

* Fixed: changed test_cases

* Fixed: lineage between the assets and jobs

* Changes as per comments

* Fix python checkstyle

* Fix as per comments

* Fix added migration scripts

* minor fixes
2022-11-03 18:19:50 +01:00
Tushar Mittal
6f2c93089c
feat: add SageMaker connector (#8435)
* feat: add sagemaker connector

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* fix: fix linting errors and update imports

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

* test: add unit tests for sagemake source

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>

Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-03 18:19:20 +01:00
Milan Bariya
6a7704a746
Fix: Filter Databricks history by date (#8520)
* Fix: Filter Databricks history by date

* Fix: Filter Databricks history by date

* Change based on comment
2022-11-03 22:08:56 +05:30
Shailesh Parmar
0f0489a7d4
Feat: Added pipeline as edge from ui (#8503)
* Feat: Added pipeline as edge from ui

* fixed failing unit test

* removed pipeline

* added unit test for util function
2022-11-03 17:46:49 +05:30
Pere Miquel Brull
c326cc1370
Move logging to DEBUG (#8519) 2022-11-03 13:04:33 +01:00
Mayur Singal
1386b43607
Fix #6141: Ingestion Pipeline Status Updates (#8216)
* Fix #6141: Ingestion Pipeline Status Updates

* List Pipeline Status API & Improvements

* Rename State field to PipelineState in UI

* Convert Pipeline Status array to single object

* fix braking UI

* Rebase Fixes

* Profiler, TestStuite & DataInsigts Pipeline

* py_format

* fix logs page not loading
add pipelineStatus endpoint

* fix recent run changes

* Fix Tests

* address review comments for ui

* fix failing checks

* fix unit tests

* fix cypress test

* remove loader test as not using it

* wait for API when we click on the tab

* fix cypress waitFor api

* fix failed cypress tests

Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2022-11-03 14:37:26 +05:30
Onkar Ravgan
8273197b48
Fix #7767: Added new PowerBi APIs and lineage (#8485)
* Added new APIs and lineage

* Addressed review comments

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-03 08:31:31 +01:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector (#8452)
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-02 16:12:45 +05:30
Pere Miquel Brull
119763afc4
Bump datamodel-code-generator (#8492) 2022-11-02 10:31:44 +01:00
Mayur Singal
fa68a1f18f
Fix #6754: Added User Details for Usage (#8449) 2022-11-02 14:07:41 +05:30
Milan Bariya
fbe9e59dcd
Fix: Athena connector fails to ingest complex types (#8453)
* Fix: Athena connector fails to ingest complex types

* Fix: Athena connector fails to ingest complex types

* Fix: Athena connector fails to ingest complex types

* Fix: Change based on comments
2022-11-02 07:06:39 +01:00
Teddy
989f2911c2
Fixes #7810 - Allow to only pass min or max (#8474)
* ISSUE-7810 Added default values for min and max
For all data validations on columns:-
min_bound is set to float("-inf"), if there is no next value
max_bound is set to float("inf"), if there is no next value

* Fixed PR errors by removing tuple + added tests

Co-authored-by: demi <deepak1212365@gmail.com>
2022-11-01 13:26:51 +01:00
Onkar Ravgan
5819c45937
Databricks + DBT issue (#8391)
* Databricks dbt issue fixed

* fixed typo

* Added optimizations

* skipping flaky dashboard test

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-11-01 13:41:41 +05:30
Onkar Ravgan
a6d6cff0d5
Fixed abstract init issue (#8446)
* abstract init issue fixed

* abstract init issue fixed messaging service

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-01 11:40:08 +05:30
Onkar Ravgan
ddc66c8392
vertica query syntax issue fixed (#8463) 2022-11-01 00:16:45 +05:30
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse (#8182) 2022-10-31 18:12:26 +05:30
Pere Miquel Brull
f4c20ded32
Ingestion Image - Remove expat version release and CI (#8447)
* Remove expat version

* Remove expat version
2022-10-31 10:01:05 +01:00
Pere Miquel Brull
bc0ce0cec1
Remove expat version (#8445) 2022-10-31 13:55:44 +05:30
Nahuel
3ed6ad8456
Update GID used in ingestion docker images (#8434) 2022-10-29 13:03:37 -07:00
Pere Miquel Brull
c8f4d496a5
Airflow exception management (#8410)
* Airflow exception management

* Airflow exception management
2022-10-28 16:11:23 +02:00
Pere Miquel Brull
f834b28198
patch ingestion image (#8385) 2022-10-27 15:10:04 +02:00
Onkar Ravgan
c156c344e6
Fixed incorrect const import (#8380)
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-10-27 08:54:00 +02:00
Pere Miquel Brull
6d404ccdc1
Domo Pipeline - Use pipeline ID with display name (#8374)
* Use pipeline ID with display name

* handle id

* lint

* update test for ID
2022-10-26 21:22:26 +02:00
amymareemc
12bc9df0b2
Issue 4886: Add support for Azure Blob (#8334)
* Issue 4686: Add support for Azure Blob

* ISSUE-4868: make changes as suggested in PR review

* ISSUE-4868: run py_format

* ISSUE-4868: Make changes to enum and formatting

* ISSUE-4868: fix linting issues and update setup.py
2022-10-26 16:14:51 +02:00
Onkar Ravgan
6f1d02ea56
Fixed dbt test case issue (#8308)
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-10-26 05:52:23 -07:00
Nahuel
0e66d3c437
Add missing calls to 'clean_raw_query' when using LineageRunner (#8363) 2022-10-26 11:34:50 +02:00
Teddy
83b048c07c
Fixes #8206 -- Implement data insight refinement for entity report data (#8351)
* added data insight module to refine and ingest data

* Added json schema for data insight workflow

* Added mixin for data insight

* Added tests for data insight workflow and methods

* Fixed python style

* Fixed python linting

* Fixed unit test

* Added data insight worflow + workflow logging

* Fix python style

* Renamed analytic -> data_insight + added test to make + fixed style

* Fixed java style

* Fixed code smells + python style

* Ignore ES linting for now

* Fixed comments from review

* fix python formatting

* Fixe PipelineType attribute for data insight
2022-10-26 11:18:08 +02:00
Pere Miquel Brull
28401ac9c8
Domo exception management (#8360) 2022-10-26 10:33:28 +02:00
Nahuel
1583adee29
Fix lineage to support 'merge into' queries starting with comments (#8355) 2022-10-25 10:29:59 -07:00
Teddy
468721fa8a
change sampling method for snflk (#8336) 2022-10-24 18:26:00 +02:00
Nahuel
36b12bd6f1
Fix lineage issues with merge_into and copy grants queries (#8335) 2022-10-24 17:22:22 +02:00
Pere Miquel Brull
60694bd60f
Disable timeout for windows (#8331) 2022-10-24 08:47:00 +02:00
Sriharsha Chintalapani
d5a9cd4bc1
Issue-8315: Parameter 'input' on the metadata restore CLI not working (#8317) 2022-10-21 21:49:01 -07:00
Suresh Srinivas
229b56e7b3
Fixes #8236 - Initialize bots from JSON data files (#8304) 2022-10-21 08:38:52 -07:00
Onkar Ravgan
0ba3036382
Added DBT col level lineage using queries (#8176)
* Added DBT col level lineage using queries

* Fixed pylint

* removed too many local vars

* Fixed pylint after rebase

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-10-21 13:57:52 +02:00
Pere Miquel Brull
d576540cb6
Bump Snowflake version (#8300) 2022-10-21 09:41:43 +02:00
Onkar Ravgan
2a85ea4747
Used ES instead of get_by_name (#8288)
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-10-20 18:16:39 +05:30
Nahuel
8d8ee3cfdd
Improve error messages and update dashboard YAML example files (#8286) 2022-10-20 14:19:58 +02:00
Teddy
030d576df7
Fix test suite bug for redefined quotes (#8248)
* Fixed bug when running test for same table name across different DB

* Fix python check style

* Fixed linting
2022-10-19 16:30:46 +02:00
Onkar Ravgan
d9b50b5be9
Added project id to DBT cloud config (#8062)
* Added project id to DBT cloud config

* added field in cloud config source Dbt config

* fix unit test issue

* Fixed pylint

Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
Co-authored-by: Ashish Gupta <ashish@getcollate.io>
2022-10-19 18:59:33 +05:30
NiharDoshi99
e086fb7c25
Add Domo Database Connector (#8221)
* added domo database connector

* added domo database connector

* changes based on comment
2022-10-19 12:50:58 +02:00
Mayur Singal
0b6e3741b3
Fix Datalake Json Error (#8246) 2022-10-19 14:12:23 +05:30