Chirag Madlani
bf6fc5f93a
prepare(release) next release ( #9479 )
...
* prepare(release) next release
* airflow typo
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
2022-12-27 20:15:46 +05:30
Nahuel
2c43ebba6f
Fix#9448: Add ES volumes ( #9506 )
...
* Add ES volumes
* Fix run_local_docker script
* Fix error run_local_docker script
* Update Es volumes in docker-compose files
2022-12-23 17:33:30 +01:00
Ayush Shah
2bf5eb9051
fix 7995: profileSample % and row number ( #9104 )
2022-12-20 14:55:11 +05:30
Nahuel
a2b34dd0f4
Fix: Update Ingestion docker images and fix python libraries dependencies ( #9342 )
...
* Update Ingestion docker images and fix python libraries dependencies
* Install also apache-airlfow-providers-http
2022-12-16 14:46:25 +00:00
Nahuel
819001182f
Fix#9251: DB2 connection config and ingestion update ( #9322 )
...
* DB2 connection config and ingestion update
* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Update ingestion/src/metadata/ingestion/source/database/common_db_source.py
Co-authored-by: Ayush Shah <ayush@getcollate.io>
* Update bootstrap/sql/com.mysql.cj.jdbc.Driver/v007__create_db_connection_info.sql
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-12-16 07:43:18 +01:00
Ayush Shah
a633befe2a
Fix: Snyk Severity for Delta Spark ( #9323 )
2022-12-16 12:12:44 +05:30
Onkar Ravgan
b539b299ee
Integrated schema parsers ( #9305 )
...
* Integrated schema parsers
* Addressed review comments
* fixed pytests
2022-12-15 16:54:55 +05:30
Mayur Singal
099853ab10
Fix Clickhouse Comments ( #9295 )
2022-12-14 21:42:06 +05:30
Sriharsha Chintalapani
38074f763b
Fix #8509 : Add schema fields ( #9209 )
...
* Fix #8509 : Add schema fields
* Fix #4675 : Ingestion deployment from UI is broken
* Added sample data for topics
* Fixed pytests
* Address comments
* Refactored sampledata according to new schema
* Added return type
* Feat(ui) ✨ : Add support for Avro editor (#9224 )
* Feat(ui) ✨ : Add support for Avro editor
* chore : add mock data for nested fields
* style: add group and opacity css
* add schema fields component to topic details and unit test
* Add locale keys
* Add support for edit description.
* refactor: expandableConfig
* test: add unit test for util method
* chore : make changes according to the schema change
* test : fix unit test and add new mock data
* chore : rename files
* Add row key
* chore : add default value for tags
* chore : update util method
* chore : add support for editing field tags
* chore : rename util files
* test : add unit test
* add comments
* addressing comments
* Address comments
* Added avro requirments
* Added requirement versions
* fixed versions
* protobuf version fix
* chore : rename util test file
* Fixed Dataype
* test: add unit test for schema component
Co-authored-by: Onkar Ravgan <onkar.10r@gmail.com>
Co-authored-by: Sachin Chaurasiya <sachinchaurasiyachotey87@gmail.com>
2022-12-14 19:56:37 +05:30
NiharDoshi99
b959709275
Fix: added datalake gen2 ( #9245 )
...
* Fix: added datalake gen2
* Fix: pytest
* Fix: pytest
* Fix: made changes to incoporate azure in datalake
* Fix: adlfs version
* Fix: adlfs version
* Fix: add tsv file type
* Fix: refractor for reading tsv file
* Fix: refractor for reading tsv file
2022-12-14 17:05:59 +05:30
Milan Bariya
c6b98751af
Logic improved for BYTES column ( #9261 )
...
* Logic improved for BYTES column
* Try Except block added
* cahange based on comments
2022-12-13 20:10:45 +05:30
Pere Miquel Brull
c75ba751b7
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification ( #9241 )
...
Fix #9116 & #8284 - Clean tableau source, fix ownership, add description and SSL verification (#9241 )
2022-12-13 06:36:55 +01:00
Pere Miquel Brull
1b3ff505c2
Fix #8858 - Add chart description and add lineage flexibility ( #9124 )
...
Fix #8858 - Add chart description and add lineage flexibility (#9124 )
2022-12-02 16:22:09 +01:00
Nahuel
76773e69de
Fix#6203: Refactor LineageRunner use ( #9073 )
...
* Refactor LineageRunner use
* Address PR comments
* Address pylint errors
* Fix failing test
2022-11-30 16:02:21 +01:00
Ayush Shah
91536044e7
Fix Grpcio failure ( #8938 )
2022-11-22 13:54:53 +00:00
Pere Miquel Brull
fe16dea584
Fix #8794 - Separate DL requirements and lazy imports ( #8806 )
2022-11-17 09:11:54 +00:00
Ayush Shah
44613b1532
Fix Profiler issue ( #8796 )
2022-11-16 17:13:34 +05:30
Teddy
3dbaa69978
Data insight workflow ( #8729 )
2022-11-15 05:44:25 +01:00
Mayur Singal
01bc9f1cfe
Fix PyMSSQL Version ( #8696 )
2022-11-14 08:40:41 +01:00
Pere Miquel Brull
34ba9d95c5
Ingestion Pipeline deployed, Athena tests and pydantic extras ( #8682 )
...
* Always run python tests
* Fix athena tests and types
* Update deployed prop in IngestionPipeline
* Fix #8554
* Format
* Use true as default deployed migration
* Remove repeated req
* Pydantic wiggle room
2022-11-13 11:59:43 +01:00
Onkar Ravgan
eee3f9ffec
Fix:#8553 Parse Avro/Protobuf/Json schemas ( #8654 )
...
* Added topic parsers
* Fixed pylint
* Addressed review comments
Co-authored-by: Onkar Ravgan <onkarravgan@Onkars-MacBook-Pro.local>
2022-11-11 16:35:09 +05:30
Teddy
199b342288
Fixes #8135 - Implement partitioning config for profiler ( #8623 )
...
* Added logic to handle partitioning config in profiler
* extracted get_partition_details out of workflow classes
2022-11-10 10:54:31 +01:00
Teddy
b44972ef60
Fixes #8470 - Implements refinement functions for web analytics events ( #8528 )
...
* Moved webanalytics type in its own folder
* Added data insight chart api endpoint
* Jave formatting
* Added resource descriptor
* Added metadata entity endpoint
* Added aggregation endpoint for dataInsight
* Fix tag name
* Added logic to ingestion pipeline resource to add ES config info if pipeline type is dataInsight
* added domo to test subpackage
* cleaned up branch by removing commit from issue-8353 that were not merged in main
* Added web analytics data refinement
* Added get_status function
* Added from __futur__ for typing
* Fixed typos brought up during reviews
2022-11-07 17:08:20 +01:00
Tushar Mittal
6f2c93089c
feat: add SageMaker connector ( #8435 )
...
* feat: add sagemaker connector
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
* fix: fix linting errors and update imports
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
* test: add unit tests for sagemake source
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-03 18:19:20 +01:00
Tushar Mittal
2a65df5f36
feat: add Kinesis connector ( #8452 )
...
Signed-off-by: Tushar Mittal <chiragmittal.mittal@gmail.com>
2022-11-02 16:12:45 +05:30
Pere Miquel Brull
119763afc4
Bump datamodel-code-generator ( #8492 )
2022-11-02 10:31:44 +01:00
Ayush Shah
2d7d89754c
Remove Click and Add ArgParse ( #8182 )
2022-10-31 18:12:26 +05:30
amymareemc
12bc9df0b2
Issue 4886: Add support for Azure Blob ( #8334 )
...
* Issue 4686: Add support for Azure Blob
* ISSUE-4868: make changes as suggested in PR review
* ISSUE-4868: run py_format
* ISSUE-4868: Make changes to enum and formatting
* ISSUE-4868: fix linting issues and update setup.py
2022-10-26 16:14:51 +02:00
Nahuel
36b12bd6f1
Fix lineage issues with merge_into and copy grants queries ( #8335 )
2022-10-24 17:22:22 +02:00
Pere Miquel Brull
d576540cb6
Bump Snowflake version ( #8300 )
2022-10-21 09:41:43 +02:00
NiharDoshi99
0e44f7d35d
Fixed: Intergrated DOMO Connector APIs ( #7989 )
2022-10-13 10:55:48 +05:30
Nahuel
5c499d2a7a
Add CLI E2E tests for MySQL ( #8041 )
...
* Add CLI E2E tests for MySQL
* Fix setup.py and pylint
* Add missing doc and update code after pylint refactor
2022-10-10 11:36:20 +02:00
Abhishek Pandey
566602373f
datalake-gcs-performance-improved ( #7609 )
2022-10-04 09:52:10 +05:30
NiharDoshi99
8adcf2ee3d
Fixed: Dagster to work with its API ( #7579 )
2022-09-23 17:46:40 +05:30
Mayur Singal
a439161d8b
#7316 : Bump sqllineage version to 1.3.6 ( #7598 )
...
Co-authored-by: Mayur <mayur@Mayurs-MacBook-Pro.local>
2022-09-21 10:14:47 +05:30
Mayur Singal
947cb4e99e
Fix #7316 : Fix Hive View error for older version ( #7592 )
2022-09-21 07:50:17 +05:30
Nahuel
2a6c6134f4
Fix#7272: Improve logging when initializing workflow from CLI ( #7522 )
...
* Improve logging when initializing workflow from CLI
* Fix broken tests
2022-09-19 08:00:00 -07:00
Cristian Calugaru
a326d3bcc0
lib upgrades ( #7536 )
...
* lib upgrades
* black reformatting
* downgraded datamode-core-generator to 0.13.0 and pydantic to 1.9.0
2022-09-18 16:51:49 -07:00
Milan Bariya
591517a487
Fix: Align pre-commit isort ( #7380 )
2022-09-14 15:10:06 +02:00
Teddy
d171e2f100
Removed typing_extension dependency ( #7406 )
2022-09-13 15:10:11 +02:00
Akash Jain
f2fc45ea41
feat: prepare-for-next-release-0-13-0 ( #7348 )
2022-09-09 01:25:41 +05:30
Pere Miquel Brull
613a329f40
Fix #7089 - Add Redpanda connector ( #7092 )
...
Fix #7089 - Add Redpanda connector (#7092 )
2022-09-01 06:39:27 +02:00
Pere Miquel Brull
6cebf37b6c
Fix #6906 - Add Nifi connector ( #6956 )
...
* Add Nifi connector
* Mock token
* Format
* pass verify param
2022-08-26 21:50:01 +02:00
Milan Bariya
994b769d25
Databricks Connection - User & Password clean ( #6866 )
...
* Databricks Connection - User & Password clean
* Fix: Python checkstyle failed
* Fix: update the migration scripts
2022-08-23 14:36:58 +02:00
Abhishek Pandey
a6a7662517
dagster-connector-added ( #6513 )
...
* dagster-connector-added
* code-formatted
* dagster-comments-removed
* dagster-comments-removed
* dagster-unittest-added
* code-formatted-pr-comments-resolved
* improved-code-quality
* increased-py-test-time-out
* improved-dagster-tests
* improved-dagster-tests
* reverted-pytests-timeout
* dagster-pytest-improved
* added-schema-in-omdcore
* updated-schema-in-omd-core
* ui-schema-updated-omd-core
* updated-dagster-schema
* dagter-schema-test-updated
2022-08-13 15:18:14 +05:30
Milan Bariya
c38ab8c8aa
Support for AWS-delegated credentials for Elasticsearch ( #6644 )
...
* Support for AWS-delegated credentials for Elasticsearch
* FIX: Based on comments
* FIX: Based on comments
* FIX: Based on comments
2022-08-09 12:36:16 +05:30
Mayur Singal
5e1b124d57
Fix #6530 & Fix #4975 : Oracle Connection Enhancements ( #6579 )
...
* Fix #6530 & Fix #4975 : Oracle Connection Enhancements
* updated migrations & docs
2022-08-05 10:28:04 +05:30
Sriharsha Chintalapani
1a42428e42
Add time series extention ( #6416 )
...
Co-authored-by: Vivek Ratnavel Subramanian <vivekratnavel90@gmail.com>
Co-authored-by: Teddy <teddy.crepineau@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-08-04 07:22:47 -07:00
Pere Miquel Brull
b7eb41a560
Fix #5840 - Upgrade Airflow container to 2.3.3 ( #6362 )
...
Fix #5840 - Upgrade Airflow container to 2.3.3 (#6362 )
2022-07-28 14:46:25 +02:00
Pere Miquel Brull
7560635114
Fix #6280 - Bump DeltaLake version, tests and docs ( #6307 )
...
Fix #6280 - Bump DeltaLake version, tests and docs (#6307 )
2022-07-24 18:49:15 +02:00
