Ayush Shah
412d61a875
Fix Okta Authentication and Validation - Ingestion ( #2955 )
2022-02-26 08:19:36 -08:00
Mayur Singal
e79be68bea
Fix #2937 : added clickhouse usage connector ( #2987 )
2022-02-26 08:18:49 -08:00
Mayur Singal
df98931ed4
fix #2809 : Auto populate View Lineage to Tables ( #2977 )
2022-02-24 08:28:40 -08:00
Mayur Singal
6a2180e975
Fix #2539 : redshift lineage mapping ( #2828 )
...
* py_format
* Fixed code smells
* used util method to ingest lineage
* removed sample sql query
2022-02-24 13:20:09 +05:30
codingwithabhi
2ef2edbf17
Bigquery lineage mapping ( #2849 )
...
* lineage-boilerplate-code-added
* bigquery-lineage-completed
* lineage-code-optimised
* code-cleaned
* code-smell-fixed
* code-refined
* added-database-name-in-fqdn
* code-smell-removed
* exception-handled
* removed-print-statement
* formatted-setup.py
* added-sqllineage-version
* included-valid-type-none
* debug-added-in-logger
* lineage-logic-updated
* updated-lineage-params
* global-config-variables-removed
* return-type-updated-in-entity
* updated-conditionin-from-entity
* imported-sqllineage-inside-method
* code-smell-removed
* code-formatted
2022-02-24 11:59:00 +05:30
Ayush Shah
959d441661
Glossary Entity Added - Ingestion fix ( #2794 )
...
* Added Glossary
* Modified Glossary: Mixin and suffix
* Glossary Mixin Methods
* Sample Data and Mixin for Glossary
* Modified comments
* Glossary Mixin Updated
* Added Glossary import
* Import issues fixed, removed upper limit for jinja2 as it conflicted
* Removed Glossary Import
2022-02-18 12:50:55 +05:30
Pere Miquel Brull
f304d290b4
Fix #2751 - Init ORM Profiler ( #2831 )
...
* ORM Profiler skeleton
* Fix table name within service
* Add license
* Prepare custom types
* Fix converter
* Compute stddev only on numeric
* Prepare smart registries
* Update tests
* Update results retrieval
* Fix composed metrics result
* Format
* Add missing type
* Add _label decorator
* clean readme
* clean readme
* Filter types when profiler runs not allowed metric types
* Fix null ratio
* Add proper type
* RuleMetric skeleton
* Prepare table metrics
* Update simple profiler
* Format
* Define test expression grammar and node visiting
* Unify metric registry
* Prepare validation core
* Add grammar lib
* Add safe get
* Format
* Allow decimals in grammar
* Test validation conversion
* Fix validation conversion and test
* Rename to row_number
* Update READMEs
* Format
* Row number naming
* Fix rename
2022-02-18 07:48:38 +01:00
Mayur Singal
419448a28f
Issue 2727: Added Databricks Connector ( #2745 )
...
* Issue #2727 : Added Databricks Connector
* optional db and licence info
* make core_bump_version_dev
* host_port fix
* fixed service_type
* change package to sqlalchemy-databricks
* Update databaseService.json
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 11:22:13 +05:30
codingwithabhi
088dba3a1f
azuresql-config-added ( #2711 )
...
* azuresql-config-added
* rebased-with-main
* tables-ingestion-added
* duplicate-files-removed
* Update azuresql.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-02-17 09:23:07 +05:30
Mayur Singal
5e4cd516bf
issue #2712 : Added SingleStore Connector ( #2747 )
2022-02-15 21:11:46 -08:00
Mayur Singal
88afb17ee8
issue #2611 : added clickhouse connector ( #2679 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-10 08:32:01 -08:00
Ayush Shah
4f5f63b226
Issue-2672: Added support for personal access token name and secret and upgraded tableau version ( #2682 )
2022-02-08 23:02:56 +05:30
codingwithabhi
66c3f43bbb
Fix #2537 : Ingestion Connector - DynamoDB
2022-02-07 10:52:11 -08:00
Mayur Singal
2d25da149d
Fix #2626 : add db2 connector ( #2661 )
...
Co-authored-by: Mayur SIngal <mayursingal@Mayurs-MacBook-Pro.local>
2022-02-07 10:49:59 -08:00
Pere Miquel Brull
08a46814e6
Fix #2625 : Use pytest 7.0.0 ( #2640 )
2022-02-06 14:41:56 -08:00
Ayush Shah
21491559de
Fix #2587 : Bigquery ADC and Policy Tag implementation
2022-02-06 10:36:10 -08:00
Pere Miquel Brull
c2ad7f6373
Fix #2317 : Override Pipeline tasks from client
2022-02-06 10:17:08 -08:00
Pere Miquel Brull
6ea6f0c43d
Fix #2383 - Multiple validation workflows ( #2500 )
...
* Clean duplicated validation
* Move black dep to dev
* Fix black version for ingestion core
* Validate core generated in the same workflow
2022-01-30 12:26:41 -08:00
Pere Miquel Brull
d10639958c
Fix black version 21.12b0 ( #2499 )
2022-01-30 06:54:11 -08:00
Ayush Shah
02f3678f2a
Fixed Mysql profiler + removed additional dependency for profiler ( #2447 )
...
* Fixed Mysql profiler + removed additional dependency for profiler
* Update sql_source.py
* Update sql_source.py
* Update setup.py
2022-01-26 16:45:28 +05:30
Akash Jain
20309cde64
fix:cherry pick from 0.8.0 and miscellaneous fixes ( #2445 )
...
* updating ingestion-core version to 0.8.0
* updating ingestion version to 0.8.0
* fixing file copy path
* update docker images to point to 0.8.0
* set up new release version
* reset ingestion version to 0.9.0.dev0
* reset ingestion-core version to 0.9.0
2022-01-26 14:00:27 +05:30
Ayush Shah
b4d63bd951
Fix: Typo in setting up pyarrow dependency ( #2398 )
2022-01-25 11:58:10 +05:30
Ayush Shah
b689c2d632
Multiple fixes - Looker, Ingestion fix, pyarrow dependency ( #2385 )
...
* Added pyarrow dependency
* Fix #2378
* Fix Looker
* Fix Looker, secretstr to str
2022-01-24 12:29:54 -08:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI ( #2349 )
...
* Prepare cli module
* Check call
* Prepare backup
* Run pytest on schema changes
* Remove traces
* Add backup plugin
* run generate during CI
* Install after generate
* Address logger hotspot
2022-01-22 12:08:14 -08:00
Sriharsha Chintalapani
0d3ded0742
Data Profiler Integration ( #2235 )
...
* Fix 2234: Data profiler integration
2022-01-18 20:25:43 -08:00
Akash Jain
105d8c1b67
Feat: docker cmd improvements ( #1895 )
...
* revamped docker commands
* update metadata docker commands in comments
* Added Docker Compose Check
* code refactored to docker.py
* fix py-checkstyle lints
* Adding clean functionality; refactor code
* pylint checkstyle fixes
* version string revised
2022-01-13 20:39:46 -08:00
Ayush Shah
f379b35279
Reflection Cache Implementation ( #2016 )
...
* Reflection Cache for Bigquery and Redshift
* Overrided few sqlalchemy packages
* Added Geography Support
* Reformatted files
* DBT models error handling implemented
* Geography type added as a custom sqlalchemy datatype
* GEOGRAPHY and VARIANT added as custom sql types
* Implemented file formatting using black
* Implemented file formatting using black
2022-01-11 14:58:03 +05:30
Pere Miquel Brull
d3b6c7cf27
[issue-1973] - Python API from Sklearn to MlModel ( #2119 )
...
* Move staticmethods to utils
* Use functions from utils
* Convert sklearn to MlModel
* merge main
2022-01-10 09:36:08 +01:00
Pere Miquel Brull
48ebcffbd0
[issue-1976] - Ingestion SonarCloud ( #2085 )
...
* Update path
* Prepare sonar properties
* Prepare coverage recipes
* Add coverage
* Simplify pytest
* Organise integration tests
* Update path
* Use setup instead of reqs
* Update recipes
* Fix PR event to target
* Update event_name
* Prepare sonar
* Run tests & sonar
* Use sonarcloud host
* Fix compose
* Use ingestion token
2022-01-07 10:28:38 +01:00
Alberto Miorin
c081eabb7e
Fix #1868 : Add openmetadata-ingestion-core as dependency ( #1869 )
2021-12-21 17:41:20 +01:00
Sriharsha Chintalapani
7111267669
Update version to 0.8 and ingestion to 0.6 ( #1859 )
2021-12-20 22:02:59 -08:00
Pere Miquel Brull
ab620e95af
[issue-1750] - Expandvars transforming values ( #1830 )
...
* Prepare tests for basic config cases
* Add test cases
* Use os expandvars
* Add missing commas
* Add missing commas
* Remove dataclasses backport
2021-12-18 15:35:12 -08:00
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Sriharsha Chintalapani
5a8c70bbe9
Fix #1747 : Ingestion: parse connection config method ( #1748 )
...
* Fix #1747 : Ingestion: parse connection config method
* Fix #1747 : Ingestion: parse connection config method
* Fix #1747 : Ingestion: parse connection config method
2021-12-15 00:58:57 +05:30
Ayush Shah
14982501c6
Snowflake Version Upgrade and Connection string change ( #1707 )
2021-12-12 08:26:15 -08:00
Pere Miquel Brull
a7c8056bf5
[issue-1079] - MlFlow Connector ( #1677 )
...
* Prepare infra
* Store experiment information in MySQL & MinIO
* Use CreateMlModelEntityRequest instead of MlModel for PUT operations
* Update MlFlow infra
* Prepare MlFlow source
* Prepare Mlflow workflow
* Simplify test and prepare README
* Revert compose
* Fix compose
* Prepare warnings and fix features
* Use non-default port for integration test
* Use mlflow-skinny for the client application
2021-12-11 18:40:19 -08:00
Ayush Shah
65fb71f819
Update setup.py - removed upper click version ( #1638 )
...
fix #1636
Removed click upper limit version as upgraded click package doesn't conflict if installed with openmetadata
2021-12-09 07:58:09 -08:00
Ayush Shah
67433f559c
Trino JDBC updated ( #1614 )
2021-12-08 13:14:38 +05:30
Ayush Shah
85f132738b
Remove faker from Sample-data ( #1536 )
...
* Removed Faker as a dependency
* Added ModuleNotFoundError
* Removed faker from requirements.txt
* Removed SampleData plugin from setup.py
2021-12-06 10:22:27 -08:00
Ayush Shah
341d2d5bf8
Update setup.py ( #1542 )
2021-12-03 08:56:29 -08:00
parthp2107
a9e70e6a89
License header update ( #1498 )
...
* updated liscense header
* updated license header
* updated license header
* updated license header
* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Sriharsha Chintalapani
7009a0bc12
Fix #1073 : Apache Druid Support ( #1424 )
...
* Fix #1073 : Apache Druid Support
2021-11-29 10:59:15 -08:00
Akash Jain
96c65e7ebd
fix: remove spaCy dependencies from setup.py ( #1362 )
...
* remove spaCy dependencies from setup.py
* Spacy, PII and Processor dependencies removed
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-11-24 12:23:17 -08:00
Ayush Shah
eb34d04285
Setup.py modified to support 3.10
2021-11-23 21:50:44 -08:00
Alberto Miorin
00fd7f4175
Fix #1291 : Python Inspect Code errors
2021-11-21 15:51:51 +01:00
Sriharsha Chintalapani
e4fa0247f5
Fix #1280 : Amundsen connector to import metadata ( #1281 )
...
* Fix #1280 : Amundsen connector to import metadata
2021-11-20 14:08:27 -08:00
Ayush Shah
219246b78e
Glue pagination added ( #1282 )
2021-11-20 12:46:18 -08:00
Sriharsha Chintalapani
f3054658f5
Fix #968 : Add DBT Connector ( #1200 )
2021-11-16 01:02:45 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added ( #1158 )
...
* Script modified - supports running from different locations
* Docker support from Python CLI
* Docker plugin setup.py
* Paths for latest and local dockers updated
* Resolved Comments - Docker CLI optimized, timestamp added
* help attribute added to options
* Docker clean code refactored
2021-11-12 10:30:28 -08:00
Ayush Shah
9839191242
Issue 483 - Glue Implementation ( #1124 )
...
* Glue Tables and Pipeline workflows implemented
* Glue Config Added
* Relative imports changed to absolute
* Resolving Comments - Changed Imports, serviceTypes
* Type fixed in setup.py
2021-11-10 07:28:13 -08:00
