IceS2
c522f14178
MINOR: Refactor output_handlers to a WorkflowOutputHandler class ( #17149 )
...
* Refactor output_handlers to a WorkflowOutputHandler class
* Add old methods as deprecated to avoid breaking changes
* Extract WorkflowInitErrorHandler from workflow_output_handler
* Fix static checks
* Fix tests
* Fix tests
* Update code based on comments from PR
* Update comment
2024-07-29 09:20:34 +02:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 ( #16480 )
...
* pydantic v2
* pydanticv2
* fix parser
* fix annotated
* fix model dumping
* mysql ingestion
* clean root models
* clean root models
* bump airflow
* bump airflow
* bump airflow
* optionals
* optionals
* optionals
* jdk
* airflow migrate
* fab provider
* fab provider
* fab provider
* some more fixes
* fixing tests and imports
* model_dump and model_validate
* model_dump and model_validate
* model_dump and model_validate
* union
* pylint
* pylint
* integration tests
* fix CostAnalysisReportData
* integration tests
* tests
* missing defaults
* missing defaults
2024-06-05 21:18:37 +02:00
Pere Miquel Brull
f4369ded8d
MINOR - Allow dict-based xlets in Airflow ( #16262 )
2024-05-15 15:08:09 +02:00
Ayush Shah
0d0d820305
MINOR: Add Static Sample Data, Modify Lineage ( #14752 )
2024-01-17 20:07:49 +05:30
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern ( #14467 )
2024-01-11 14:23:33 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers ( #14358 )
...
* MINOR - Clean ingestion ES code and auth providers
* clean parser
* Clean security config for the client
* Clean security config for the client
* Improve class conversion exceptions
* Fix tests
* Clean up java client
* Clean up java client
* clean parser
* Fix test
* Fix test
* fix NO_AUTH error
* Fix test
* Format
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets ( #14191 )
...
* Add OMEntity model
* Test OMEntity
* Update repr
* Fix __str__
* Add entity ref map
* Test serializer for backend
* Fix tests
* Fix serializer
* Test runner
* Add runner tests
* Update docs
* Format
2023-12-01 06:29:44 +01:00
Pere Miquel Brull
f6a87ee02a
Fix #12082 - Bump PyAthena version ( #13464 )
2023-10-09 20:47:19 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
Pere Miquel Brull
095f15064c
Fix #11752 - Multiple inlets in AirflowLineageOperator ( #11945 )
...
* Fix multiple inlets in AirflowLineageOperator
* Update dag file
2023-06-12 07:01:19 +02:00
Pere Miquel Brull
9ebc0cd5e4
Fix sample usage DAG directory ( #11276 )
2023-04-26 06:53:10 +02:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning ( #10603 )
...
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Pere Miquel Brull
fb15c896b3
Handle XLets in groups for AirflowLineageRunner ( #10114 )
...
* Handle XLets in groups
* Linting
* Linting
2023-02-07 06:49:46 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package ( #9521 )
...
Fix #6562 - Sources have their own package (#9521 )
2022-12-27 15:00:22 +01:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers ( #9513 )
...
Organise dynamic importers (#9513 )
2022-12-23 12:17:06 +01:00
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test ( #9363 )
...
* Prep airflow operator integration tests
* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state ( #9206 )
2022-12-10 19:54:41 +01:00
Onkar Ravgan
23468fb868
Fixed Broken docs links ( #9125 )
2022-12-02 15:42:06 +01:00
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image ( #8065 )
...
Prepare Ingestion Base Docker image (#8065 )
2022-10-11 07:50:49 +02:00
mohitdeuex
cceff48f0d
Default token ( #7695 )
2022-09-23 20:03:54 +05:30
Akash Jain
fc8312c5fb
feat: Refactor openmetadata/ingestion docker image ( #7456 )
...
* centralize openmetadata/ingestion docker image
* update volume mappings; fix ingestion tag!
* fix PR comments
* fix cypress mysql tests!
2022-09-19 09:20:54 +05:30
Nahuel
ed58c27f76
Remove unused property extra_hosts from docker-compose files ( #7096 )
...
* Remove unused properties from docker-compose files
* Minor fix in run_local_docker script
2022-08-31 21:30:24 +02:00
Ashish Gupta
b59878e1b5
Fix ( #6984 ) doc links in OpenMetaData ( #6986 )
...
* Fix doc links in OpenMetaData
* Link updated
* remove unwanted file
* changes as per comments
* Changes as per comments
2022-08-29 19:38:54 +05:30
Nahuel
cf2cb6d531
Fix#6517: Add clusterName property to the application config yaml ( #6610 )
...
* Add cluster name in the app configuration and start using it to create secrets id
* Update secret manager client in openmetadata for using default auth provider
* Add missing property in test config file
2022-08-09 09:00:43 +02:00
Nahuel
0602e8c894
Fix#5924: Implement secret manager for Airflow REST Plugin ( #6128 )
...
Fix#5924: Implement secret manager for Airflow REST Plugin (#6128 )
2022-07-19 14:51:44 +02:00
Pere Miquel Brull
a6dd100be4
Fix metadata es DAG ( #5933 )
2022-07-07 12:15:09 -07:00
Sriharsha Chintalapani
8528c51b9c
Fix #5516 : Refactor elasticsearch indexes to be same as entity specifications ( #5573 )
...
* Fix #5516 : Refactor elasticsearch indexes to be same as entity specifications
Co-authored-by: darth-coder00 <aashit@getcollate.io>
Co-authored-by: Sachin-chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-06-27 01:19:27 -07:00
Pere Miquel Brull
bb4ac68ee3
Add backend connection for airflow dag extraction
2022-06-24 20:01:50 +02:00
Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices ( #5446 )
...
Fix #5367 - MlFlow connector & MLServices (#5446 )
2022-06-21 14:54:36 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector ( #5125 )
...
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125 )
2022-05-25 08:35:16 +02:00
Pere Miquel Brull
c6915ca510
Fix #4923 - Presto ingestion & SQLAlchemy test connection refactor ( #4946 )
...
Fix #4923 - Presto ingestion & SQLAlchemy test connection refactor (#4946 )
2022-05-14 06:44:43 +02:00
Ayush Shah
61591058fb
Added support openmetadata extraction - Ingestion ( #4195 )
2022-04-19 19:37:41 +05:30
Pere Miquel Brull
2aecebd595
Fix #4189 - Fix airflow lineage callback ( #4191 )
...
Fix #4189 - Fix airflow lineage callback (#4191 )
2022-04-18 15:13:26 +02:00
Ayush Shah
4f5c5a9e23
Ingestion - fix pytest workflow errors and modify airflow pipeline ( #4178 )
2022-04-18 11:36:21 +05:30
Sriharsha Chintalapani
06cc35d048
Fix #4082 : Do not start slack event publisher if there is no config ( #4083 )
2022-04-12 17:25:23 -07:00
Ayush Shah
15766e85a6
Sample Usage fixed ( #4065 )
2022-04-12 14:57:18 +05:30
Pere Miquel Brull
6a6507e754
Fix #3962 - Profiler uses DatabaseSchema & Sample Data fix ( #4056 )
2022-04-12 13:40:59 +05:30
Pere Miquel Brull
d6b49f070c
Fix #3951 - OpenMetadata Airflow REST ( #4034 )
...
Fix #3951 - OpenMetadata Airflow REST (#4034 )
2022-04-11 18:38:26 +02:00
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field ( #3909 )
...
* Fix sample data DAG
* Fix callback imports
* Use --set-default-enum-member in generate
* Format
* Add faulty merge hard_delete
* Fix airflow lineage, improve naming and fix lineage tests
* Add mysql url test
* Add mysql url test
* Update CI name
* Fix test ometa endpoint
* Format
* Fix metadata config
2022-04-07 18:22:50 +05:30
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service ( #3804 )
...
* Fix #3659 Refactor Service Connection String to be specific to per service
* Simplify and centralize Airflow Pipeline info for REST (#3740 )
* Remove code
* Modified Configs based on refactoring schema (#3816 )
* Clean WorkflowContext
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Ayush Shah
351db56687
Added fix for Sample_Usage not being ingested in Local docker ( #3650 )
2022-03-27 22:38:13 -07:00
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln ( #3370 )
...
Fix #3104 - Remove unused imports with pycln (#3370 )
2022-03-14 06:59:15 +01:00
Sriharsha Chintalapani
891ff465a0
Fix #2838 : Fix airflow example DAG to import airflow libs ( #2839 )
2022-02-18 10:47:07 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG ( #2337 )
...
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
2022-01-22 16:37:56 +05:30
Alberto Miorin
c8de136cca
First part of #2091 : Python code in branch main is always formatted correctly ( #2093 )
2022-01-07 06:17:48 +01:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink ( #1956 )
...
* Query Usage fixed
* Usage updated
* Reformatted change
* Usage files reformatted
2022-01-04 17:12:10 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed ( #1788 )
...
* Metabase and Snowflake Fixed
* Removed DBT
2021-12-16 11:57:43 +05:30
parthp2107
64386b035e
added new tests for tags page ( #1621 )
...
* added new tests for tags page
* addressing reviewdog findings
* Added Sample-usage service_name
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-12-08 16:54:48 +05:30
Ayush Shah
0e205d93dd
Added DBT to airflow, modified tables.json - added profile data, fixed user's displayName ( #1501 )
...
* User's Name fixed, added profile data, added dbt pipeline under airflow
* service name changed
2021-12-01 08:32:29 -08:00
