NiharDoshi99
012f96a7a2
Domo issue ( #10080 )
...
* Domo issue
* added pydantic models and changes as per comment
* Fix: added ownership details
* fixed pytest
2023-02-07 17:16:26 +05:30
Pere Miquel Brull
fb15c896b3
Handle XLets in groups for AirflowLineageRunner ( #10114 )
...
* Handle XLets in groups
* Linting
* Linting
2023-02-07 06:49:46 +01:00
Pere Miquel Brull
f0f3f0be6a
Add looker unit tests ( #9691 )
...
* Add looker tests
* Empty-Commit
* Install GE for tests
* Fix usage details python name
* Add missing test requirement
2023-02-01 09:20:26 +00:00
Teddy
ba08302ea1
Issue #7291 - Implements Table Rows Inserted to be Between test ( #9813 )
...
* staging commit
* staging commit
* refactor: partitioning logic
* refactor (tests): move to parametrized tests for test validations
* refactor: local variables into global
* (feat): Added logic for table row inserted test
* (feat): fix python checkstyle
* feature: extracted get_query_filter logic into its own function
2023-01-31 15:57:51 +01:00
Teddy
0b3996d2fa
Remove profiler support for dynamodb ( #10014 )
...
* fix(DynamoDB): remove profiler support and cleaned up documentation
* fix(DynamoDB): remove supportProfiler filed in tests
2023-01-31 08:29:35 +01:00
Abhishek Pandey
933e195ef6
salesforce-test-added ( #9890 )
...
* salesforce-test-added
* license-headers-added
2023-01-25 21:24:21 +05:30
Pere Miquel Brull
bce8574f98
Dashboards and charts use the ID as name ( #9810 )
...
* Update dashboard names
* Fix: metabase lineage
Co-authored-by: Nihar Doshi <nihardoshi16@gmail.com>
2023-01-24 21:50:35 +05:30
Pere Miquel Brull
f2fb0521c2
Update airflow loggers and rename ometa loggers ( #9868 )
...
* Update airflow loggers and rename ometa loggers
* ANSI print to logger
* Remove colored logging from tests
* Merge ometa_logger into the one used in loggers class
* linting
* linting
Co-authored-by: Nahuel Verdugo Revigliono <nahuel@getcollate.io>
2023-01-23 16:28:17 +01:00
Nahuel
ddff6e2875
Fix: Replace sqllineage with openmetadata-sqllineage ( #9800 )
...
* Replace sqllineage with openmetadata-sqllineage
* Fix checkstyle and failing test
* Move logic to retrieve dialect of a service type into a class
* Improve py-check message when it fails
* Updated mapper
* Update code after merge
2023-01-19 14:56:29 +01:00
Ayush Shah
14100d089f
Modify Charts ( #9814 )
2023-01-19 12:03:21 +00:00
Ayush Shah
02147b6502
Fix: Quicksight Dashboard and chart Metadata Ingestion ( #9786 )
...
* Fix Quicksight json schema and errors
* Fix Quicksight lineage
* add pagination
* Generalize Pagination
* Fix Pylints
* update max results
* fix tests
* Add Pydantic Models for QuickSight resp
* Add fqn builder
* Fix Py lInt
* Fix Tests
2023-01-18 20:37:41 +05:30
NiharDoshi99
f2649041f2
Fix: atlas test connection ( #9791 )
...
* Fix: atlas test connection
* Fix: docs changes
* Fix: docs changes
* Fix: python checkstyle
* Fix: python test
2023-01-18 20:16:07 +05:30
Milan Bariya
7f111584b8
Deprecate OMetaDatabaseAndTable ( #9760 )
...
* Deprecate OMetaDatabaseAndTable
* Change Based On Comments
* Change Based On Comments
* Fix Pytest
* Fix Pytest
2023-01-18 14:29:32 +01:00
Abhishek Pandey
86a57293ef
databricks-test-added ( #9749 )
...
* databricks-test-added
* databricks-test-added
2023-01-18 08:45:46 +01:00
Suresh Srinivas
ac157a8f0b
Fixes #9618 Allow apostrophe in entity names ( #9642 )
...
* Fixes #9618 Allow apostrophe in entity names
* Add support for space in entity names (with exception) and column names
* Add regex checks to entity names, table column names to allow only required characters
* fix: allowed characters in entityName
* Add all the allowed special characters in the entity name in tests
* Allow '&', '/', ':' only for the table
* Override basic entityName by defining entity specific entityName for Python code gen
* fix: dagster migration + json schema path
* fix: python style check
* Fix test failure
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-01-17 11:34:29 -08:00
NiharDoshi99
2954f17832
Fix: Domo dashboard issue ( #9743 )
...
* Fix: Domo dashboard issue
* Fix: added execption handling
* Fix: changed as per comment
2023-01-17 21:27:15 +05:30
Ayush Shah
f11cdbfab1
Profile sample rows + Randomization of chunks for datalake ( #9595 )
2023-01-16 22:17:46 +05:30
NiharDoshi99
029dbe892e
Fix: added test case for atlas ( #9678 )
...
* Fix: added test case for atlas
* Fix: resolved conflict
* Fix: changing back neo4j to old version
* Fix: changing back neo4j to old version
* Fix: changes as per comment
* Fix: changes as per comment
* Fix: python checkstyle
2023-01-13 16:07:29 +05:30
Pere Miquel Brull
bf753a4dee
Fix #7768 - Update and organize versions ( #9664 )
...
Fix #7768 - Update and organize versions (#9664 )
2023-01-11 07:05:12 +01:00
Pere Miquel Brull
84348d4748
Fix #8866 - bump datamodel-codegen ( #9623 )
...
* Fix #8866 - bump datamodel-codegen
* Update connection options and arguments structure
* Add builders test
* Format
* Allow Any values in componentConfig
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
2023-01-09 13:20:32 +01:00
NiharDoshi99
27c68fb208
test case for DomoDatabase ( #9630 )
...
* test case for domo-database
* test case for domo-database
* Fix: pytest
2023-01-06 12:02:01 +01:00
Mayur Singal
9ef32eef99
Fix #6829 : Improve usage filtering ( #9613 )
2023-01-05 15:07:03 +00:00
Steve Russo
3a3f248edd
Issue 5592: Add Databricks Pipeline Connector ( #9554 )
...
* feat: update databricks_client
* feat: create pipelineconnection & connection
* test: add unit test
* feat: add examples file
* fix: remove unused imports
* style: fix python style
* fix: code review adjustments
* fix: remove unneeded schema ref
* fix: format requirements
* fix: change context calls
* test: add pipeline_status test
* format: pylint format change
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2023-01-05 14:14:12 +01:00
Pere Miquel Brull
1f49ea46b5
Fix #9149 - Postgres foreign tables & reorganize queries for sources ( #9588 )
...
Fix #9149 - Postgres foreign tables & reorganize queries for sources (#9588 )
2023-01-04 08:50:53 +01:00
8vicat
7e7528e13b
Improve SQL lineage parsing ( #9537 )
...
* CTEs support
* Supports inline comments
* Expands `*` in sources
2023-01-03 09:27:45 +01:00
NiharDoshi99
8f69386076
Fix: refactor get_column_type ( #9559 )
...
* Fix: refactor get_column_type
* Fix: added changes as per comments
* Fix: pylint
* Fix: minor changes
* Fix: minor changes
2023-01-03 10:28:38 +05:30
Pere Miquel Brull
7f21a7bced
Fix #8088 - Restructure source connections & clients ( #9545 )
2023-01-02 13:52:27 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package ( #9521 )
...
Fix #6562 - Sources have their own package (#9521 )
2022-12-27 15:00:22 +01:00
Suresh Srinivas
758c976cba
Fixes #9259 Change Tags APIs to conform with rest of the APIs ( #9260 )
2022-12-26 12:32:17 -08:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers ( #9513 )
...
Organise dynamic importers (#9513 )
2022-12-23 12:17:06 +01:00
Teddy
9e01fe0636
feat(CLI): added migration logic cmd ( #9437 )
2022-12-21 07:55:18 +00:00
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number ( #9104 )
2022-12-20 14:55:11 +05:30
Milan Bariya
a3495c67f0
Databricks improve code and doc update ( #9394 )
...
* Databricks improve code and doc update
* Change base on comments
* Fix pytest fail
2022-12-19 18:53:10 +05:30
Teddy
3856d63f3a
Fixes #5522 by implementing timeout for the profiler ( #9340 )
...
* feat(Profiler): Added timeout logic in the profiler workflow
* feat(Profiler): Added additional input in workflow UI
* fix(Profiler): timeout text blob in UI
* fix(Profiler): Return profile_results outside the context manager
* Fix(Profiler): Added localizer for timeout label and message
* Update ingestion/src/metadata/interfaces/sqalchemy/sqa_profiler_interface.py
Co-authored-by: Nahuel <nahuel@getcollate.io>
Co-authored-by: Nahuel <nahuel@getcollate.io>
2022-12-16 17:01:12 +01:00
NiharDoshi99
3190ac2516
Fix: making token as optional field ( #9314 )
2022-12-16 14:17:47 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update ( #9322 )
...
* DB2 connection config and ingestion update
* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Onkar Ravgan
b539b299ee
Integrated schema parsers ( #9305 )
...
* Integrated schema parsers
* Addressed review comments
* fixed pytests
2022-12-15 16:54:55 +05:30
Ayush Shah
a6ae9fd11a
Add Test Suite Implementation for Datalake ( #9235 )
2022-12-14 21:14:51 +05:30
Nahuel
9a4e3a7a46
Fix#8424: Remove brackets from tables and schemas on lineage ( #9257 )
...
* Refactor LineageRunner use
* Address PR comments
* Address pylint errors
* Fix failing test
* Remove brackets from tables and schemas on lineage
2022-12-13 06:40:37 +01:00
Ayush Shah
231b28fc87
Fix 7278: add test connection ( #9196 )
...
* Add Test Connection to Profiler
* remove Code Lint
* Fix AttributeError
* Fix Pytest
* Fix Bigquery Partition tests
* Fix Lint
2022-12-10 17:16:28 +01:00
Teddy
ac77f33b08
Fixes #7447 -- Add freshness metrics to profiler ( #9159 )
...
* refactor(profiler): integrated getter func.
Removed metric getter function from their own file.
Added metric getter to their own interface classs.
created dispatch by value methdo to dispatch metric getter func.
* feature(profiler): added systemProfiler schema
* feat(profiler): workflow fresh. & snflk impl.
* feat(profiler): freshness endpoint for put and get
* feat(profiler): added system met. for redshift
* feat(profiler): freshness met. for bigquery
* fix(profiler): keyword not found in func
* feat(profiler): Added sample data for freshness
* fix(profiler): fetch previous day for BQ
* fix(profiler): sonar + data fetching logic
* fix: typo in SystemMetric Class
* fix: linting
* fix: extracted out EntityList class into models.py
2022-12-07 14:33:30 +01:00
Nahuel
00375341f7
Fix#7690: Use ES to build Tables FQN when processing Lineage ( #9079 )
...
* Use ES to build Tables FQN when processing Lineage
* Fix pylent check
* Fix test failing
* Remove warning when calling ES with empty query
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-01 09:56:42 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use ( #9073 )
...
* Refactor LineageRunner use
* Address PR comments
* Address pylint errors
* Fix failing test
2022-11-30 16:02:21 +01:00
Teddy
3cad959e44
Fixes #6760 -- Implements REGEX for regex test ( #9033 )
...
feat(testCase): impelemented regex logic for test suite
2022-11-29 13:00:28 +01:00
Milan Bariya
27997b2f77
Fix: Revert Athena tests and types change ( #8909 )
2022-11-21 12:35:23 +01:00
Nahuel
b14a3bc6ca
Review sensitive data not marked as password in the JSON schemas ( #8818 )
2022-11-18 11:35:08 +01:00
Pere Miquel Brull
b203a92565
Add config file env var loading tests ( #8799 )
...
* Add env var load test
* Add env var load test
2022-11-17 16:52:42 +00:00
Pere Miquel Brull
93381a7343
Add Pylint print checker and py-checkstyle as required ( #8849 )
...
* Always run the checkstyle
* Add print checker lint plugin
* Typos
* Add print checker
* Fix e2e
2022-11-17 16:44:26 +00:00
NiharDoshi99
03d4011a17
Fix: Changes in bigquey for project-id ( #8708 )
2022-11-17 14:26:37 +05:30
Teddy
4ccdce50e0
Fix data insight bugs ( #8761 )
...
* - made ES ingestion of data inisght idempotent for the daily execution
- added web analytic timedelta to 1 day to fecth data from the previous day
* Added time utils
* fixed timezone
2022-11-15 16:10:30 +01:00
