Ayush Shah
1f3667b46f
Fix Python test ( #4095 )
2022-04-13 21:03:33 +05:30
Mayur Singal
726da97dd8
Fix #3654 : Fixed TypeError in Deltalake ( #3710 )
...
* Fix #3654 : fixed minor bugs
* Fixed table ingestion issues
* Added sample script for deltalake
* Added Hive Metastore support
* Added support for local metastore
* removed code smell
* Resolved review comments
* fixed display data type logic
* Fixed Code Smell and Resolved conflicts
* Fixed Delta lake source
* removed set spark
* Updated prepare method
* id fix
* Metatore exception added
2022-04-11 10:45:59 +05:30
dvecerdea
23d68b24d9
Replace sqlalchemy-trino with trino[sqlalchemy] ( #3800 )
2022-04-01 12:12:03 -07:00
Pere Miquel Brull
3b7b65d46c
Upper bound click req ( #3724 )
2022-03-29 11:28:41 +05:30
Pere Miquel Brull
1c21abebdd
Fix #3645 - Revisit lineage for Airflow 1.10.15 ( #3696 )
...
Fix #3645 - Revisit lineage for Airflow 1.10.15 (#3696 )
2022-03-28 08:33:02 +02:00
Mayur Singal
3aa8755a7f
Fix #3465 : Added trino version ( #3466 )
2022-03-16 09:00:08 -07:00
Ayush Shah
28a1918ed8
Fixed Looker - Dashboard and Dashboard Elements ( #3441 )
...
* Looker fix
* Update looker.py
2022-03-16 11:52:29 +05:30
Onkar Ravgan
ad04136847
Fix #3095 : Added support for private key ( #3408 )
...
* ISSUE-3095: Add support for private key in snowflake
rebase main branch
* Fix #3095 : Added support for private key
* Fix #3095 : Added support for private key and review changes
* Fix #3095 : Added support for private key and review changes
* changed direct assignment of dictionary
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-03-15 18:19:02 +05:30
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln ( #3370 )
...
Fix #3104 - Remove unused imports with pycln (#3370 )
2022-03-14 06:59:15 +01:00
Ayush Shah
7c84f063e6
Minor Fixes - Ingestion - Datetime format, Looker fix ( #3365 )
...
* Minor Fixes - UTC timezone for usage, tag method modification, looker fix
* Modified core version
* Snowflake Usage Result limit and Makefile ignore env fix
2022-03-11 13:31:42 +05:30
Akash Jain
211a8a55e8
feat: Prepare 0.9.1-SNAPSHOT ( #3297 )
...
* prepare snapshot version for 0.9.1
* prepare dev version for ingestion 0.9.1
* prepare dev version for ingestion-core 0.9.1
* prepare Docker CI workflows for 0.9.1
2022-03-09 12:08:42 +05:30
Pere Miquel Brull
a9290bf1a0
Fix #3181 - Align ingestion requirements
...
Fix ES and bump down redash (#3182 )
2022-03-06 09:00:58 +01:00
codingwithabhi
ad341e8259
fix ISSUE-3137: updated-redash-library-and-function ( #3157 )
2022-03-04 12:02:08 -08:00
Ayush Shah
c3cc685663
Excluded db2 from all plugins as it was creating issues with docker image creation ( #3097 )
2022-03-03 14:55:40 +05:30
Pere Miquel Brull
71207de362
Fix #2875 - Profiler API Sink ( #3011 )
...
Fix #2875 - Profiler API Sink
2022-03-02 16:46:28 +01:00
Alberto Miorin
fe5618c8f1
Fix #3037 : metadata --version doesn't work ( #3038 )
2022-03-01 12:19:36 +01:00
codingwithabhi
06d9329ae3
Atlas connector ( #2855 )
...
* Fix #1161 : Add Atlas Connector
* Atlast Decimal type handled
* Atlas col_data_type handled
* Atlas columns exception handling implemented
* atlas-services-added-into-single-file
* import-updated
* authentication-fix
* atlas-connector-services-added
* code-smell-removed
* code-smell-removed
* code-smell-removed
* file-exist-check-added
* updated-mapping-json-logic
* pr-comment-changes-completed
* atlas-linegae-added
* atlas-linegae-added
* pr-changes-done
* code-formatted
* bug-resolved
* topic-bug-resolved
Co-authored-by: Sriharsha Chintalapani <harsha@getcollate.io>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-03-01 11:50:14 +05:30
Ayush Shah
568845b65c
Power BI connector ( #3019 )
...
* Power BI Dashboard and Tiles added
* Added Powerbi dependency
* Modified Powerbi debug logs
* Bump up _version.py
* Modified Power BI - added failure status, resolved comments
2022-03-01 10:09:35 +05:30
Mayur Singal
c49af971a7
Fix #2984 : added azure sso auth ( #3002 )
2022-02-28 11:44:28 -08:00
Pere Miquel Brull
00552f6d36
Fix #3012 - Dependency breaking Airflow Container ( #3015 )
...
Fix #3012 - Dependency breaking Airflow Container
2022-02-28 17:11:18 +01:00
Ayush Shah
412d61a875
Fix Okta Authentication and Validation - Ingestion ( #2955 )
2022-02-26 08:19:36 -08:00
Mayur Singal
e79be68bea
Fix #2937 : added clickhouse usage connector ( #2987 )
2022-02-26 08:18:49 -08:00
Mayur Singal
df98931ed4
fix #2809 : Auto populate View Lineage to Tables ( #2977 )
2022-02-24 08:28:40 -08:00
Mayur Singal
6a2180e975
Fix #2539 : redshift lineage mapping ( #2828 )
...
* py_format
* Fixed code smells
* used util method to ingest lineage
* removed sample sql query
2022-02-24 13:20:09 +05:30
codingwithabhi
2ef2edbf17
Bigquery lineage mapping ( #2849 )
...
* lineage-boilerplate-code-added
* bigquery-lineage-completed
* lineage-code-optimised
* code-cleaned
* code-smell-fixed
* code-refined
* added-database-name-in-fqdn
* code-smell-removed
* exception-handled
* removed-print-statement
* formatted-setup.py
* added-sqllineage-version
* included-valid-type-none
* debug-added-in-logger
* lineage-logic-updated
* updated-lineage-params
* global-config-variables-removed
* return-type-updated-in-entity
* updated-conditionin-from-entity
* imported-sqllineage-inside-method
* code-smell-removed
* code-formatted
2022-02-24 11:59:00 +05:30
Ayush Shah
959d441661
Glossary Entity Added - Ingestion fix ( #2794 )
...
* Added Glossary
* Modified Glossary: Mixin and suffix
* Glossary Mixin Methods
* Sample Data and Mixin for Glossary
* Modified comments
* Glossary Mixin Updated
* Added Glossary import
* Import issues fixed, removed upper limit for jinja2 as it conflicted
* Removed Glossary Import
2022-02-18 12:50:55 +05:30
Pere Miquel Brull
f304d290b4
Fix #2751 - Init ORM Profiler ( #2831 )
...
* ORM Profiler skeleton
* Fix table name within service
* Add license
* Prepare custom types
* Fix converter
* Compute stddev only on numeric
* Prepare smart registries
* Update tests
* Update results retrieval
* Fix composed metrics result
* Format
* Add missing type
* Add _label decorator
* clean readme
* clean readme
* Filter types when profiler runs not allowed metric types
* Fix null ratio
* Add proper type
* RuleMetric skeleton
* Prepare table metrics
* Update simple profiler
* Format
* Define test expression grammar and node visiting
* Unify metric registry
* Prepare validation core
* Add grammar lib
* Add safe get
* Format
* Allow decimals in grammar
* Test validation conversion
* Fix validation conversion and test
* Rename to row_number
* Update READMEs
* Format
* Row number naming
* Fix rename
2022-02-18 07:48:38 +01:00
Mayur Singal
419448a28f
Issue 2727: Added Databricks Connector ( #2745 )
...
* Issue #2727 : Added Databricks Connector
* optional db and licence info
* make core_bump_version_dev
* host_port fix
* fixed service_type
* change package to sqlalchemy-databricks
* Update databaseService.json
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 11:22:13 +05:30
codingwithabhi
088dba3a1f
azuresql-config-added ( #2711 )
...
* azuresql-config-added
* rebased-with-main
* tables-ingestion-added
* duplicate-files-removed
* Update azuresql.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 09:23:07 +05:30
Mayur Singal
5e4cd516bf
issue #2712 : Added SingleStore Connector ( #2747 )
2022-02-15 21:11:46 -08:00
Mayur Singal
88afb17ee8
issue #2611 : added clickhouse connector ( #2679 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-10 08:32:01 -08:00
Ayush Shah
4f5f63b226
Issue-2672: Added support for personal access token name and secret and upgraded tableau version ( #2682 )
2022-02-08 23:02:56 +05:30
codingwithabhi
66c3f43bbb
Fix #2537 : Ingestion Connector - DynamoDB
2022-02-07 10:52:11 -08:00
Mayur Singal
2d25da149d
Fix #2626 : add db2 connector ( #2661 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-07 10:49:59 -08:00
Pere Miquel Brull
08a46814e6
Fix #2625 : Use pytest 7.0.0 ( #2640 )
2022-02-06 14:41:56 -08:00
Ayush Shah
21491559de
Fix #2587 : Bigquery ADC and Policy Tag implementation
2022-02-06 10:36:10 -08:00
Pere Miquel Brull
c2ad7f6373
Fix #2317 : Override Pipeline tasks from client
2022-02-06 10:17:08 -08:00
Pere Miquel Brull
6ea6f0c43d
Fix #2383 - Multiple validation workflows ( #2500 )
...
* Clean duplicated validation
* Move black dep to dev
* Fix black version for ingestion core
* Validate core generated in the same workflow
2022-01-30 12:26:41 -08:00
Pere Miquel Brull
d10639958c
Fix black version 21.12b0 ( #2499 )
2022-01-30 06:54:11 -08:00
Ayush Shah
02f3678f2a
Fixed Mysql profiler + removed additional dependency for profiler ( #2447 )
...
* Fixed Mysql profiler + removed additional dependency for profiler
* Update sql_source.py
* Update sql_source.py
* Update setup.py
2022-01-26 16:45:28 +05:30
Akash Jain
20309cde64
fix:cherry pick from 0.8.0 and miscellaneous fixes ( #2445 )
...
* updating ingestion-core version to 0.8.0
* updating ingestion version to 0.8.0
* fixing file copy path
* update docker images to point to 0.8.0
* set up new release version
* reset ingestion version to 0.9.0.dev0
* reset ingestion-core version to 0.9.0
2022-01-26 14:00:27 +05:30
Ayush Shah
b4d63bd951
Fix: Typo in setting up pyarrow dependency ( #2398 )
2022-01-25 11:58:10 +05:30
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency ( #2385 )
...
* Added pyarrow dependency
* Fix #2378
* Fix Looker
* Fix Looker, secretstr to str
2022-01-24 12:29:54 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI ( #2349 )
...
* Prepare cli module
* Check call
* Prepare backup
* Run pytest on schema changes
* Remove traces
* Add backup plugin
* run generate during CI
* Install after generate
* Address logger hotspot
2022-01-22 12:08:14 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration ( #2235 )
...
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements ( #1895 )
...
* revamped docker commands
* update metadata docker commands in comments
* Added Docker Compose Check
* code refactored to docker.py
* fix py-checkstyle lints
* Adding clean functionality; refactor code
* pylint checkstyle fixes
* version string revised
2022-01-13 20:39:46 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation ( #2016 )
...
* Reflection Cache for Bigquery and Redshift
* Overrided few sqlalchemy packages
* Added Geography Support
* Reformatted files
* DBT models error handling implemented
* Geography type added as a custom sqlalchemy datatype
* GEOGRAPHY and VARIANT added as custom sql types
* Implemented file formatting using black
* Implemented file formatting using black
2022-01-11 14:58:03 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel ( #2119 )
...
* Move staticmethods to utils
* Use functions from utils
* Convert sklearn to MlModel
* merge main
2022-01-10 09:36:08 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud ( #2085 )
...
* Update path
* Prepare sonar properties
* Prepare coverage recipes
* Add coverage
* Simplify pytest
* Organise integration tests
* Update path
* Use setup instead of reqs
* Update recipes
* Fix PR event to target
* Update event_name
* Prepare sonar
* Run tests & sonar
* Use sonarcloud host
* Fix compose
* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
c081eabb7e
Fix #1868 : Add openmetadata-ingestion-core as dependency ( #1869 )
2021-12-21 17:41:20 +01:00
