Mayur Singal
7760663b22
MINOR: Change ingestion licence header ( #20549 )
2025-04-03 10:39:47 +05:30
Pere Miquel Brull
7402feba6f
MINOR - Remove airflow_lineage_operator from final ingestion image ( #20551 )
2025-04-02 11:53:55 +02:00
IceS2
c522f14178
MINOR: Refactor output_handlers to a WorkflowOutputHandler class ( #17149 )
...
* Refactor output_handlers to a WorkflowOutputHandler class
* Add old methods as deprecated to avoid breaking changes
* Extract WorkflowInitErrorHandler from workflow_output_handler
* Fix static checks
* Fix tests
* Fix tests
* Update code based on comments from PR
* Update comment
2024-07-29 09:20:34 +02:00
Pere Miquel Brull
d8e2187980
#15243 - Pydantic V2 & Airflow 2.9 ( #16480 )
...
* pydantic v2
* pydanticv2
* fix parser
* fix annotated
* fix model dumping
* mysql ingestion
* clean root models
* clean root models
* bump airflow
* bump airflow
* bump airflow
* optionals
* optionals
* optionals
* jdk
* airflow migrate
* fab provider
* fab provider
* fab provider
* some more fixes
* fixing tests and imports
* model_dump and model_validate
* model_dump and model_validate
* model_dump and model_validate
* union
* pylint
* pylint
* integration tests
* fix CostAnalysisReportData
* integration tests
* tests
* missing defaults
* missing defaults
2024-06-05 21:18:37 +02:00
Pere Miquel Brull
f4369ded8d
MINOR - Allow dict-based xlets in Airflow ( #16262 )
2024-05-15 15:08:09 +02:00
Ayush Shah
0d0d820305
MINOR: Add Static Sample Data, Modify Lineage ( #14752 )
2024-01-17 20:07:49 +05:30
Ayush Shah
9c6d202555
Add Sample data, modify regex pattern ( #14467 )
2024-01-11 14:23:33 +05:30
Pere Miquel Brull
e16ad3c5dc
MINOR - Clean ingestion ES code and auth providers ( #14358 )
...
* MINOR - Clean ingestion ES code and auth providers
* clean parser
* Clean security config for the client
* Clean security config for the client
* Improve class conversion exceptions
* Fix tests
* Clean up java client
* Clean up java client
* clean parser
* Fix test
* Fix test
* fix NO_AUTH error
* Fix test
* Format
---------
Co-authored-by: Chirag Madlani <12962843+chirag-madlani@users.noreply.github.com>
2023-12-19 12:08:48 +01:00
Pere Miquel Brull
7fcdf08ca4
#11626 & #14131 - Lineage with other Entities & attr-based xlets ( #14191 )
...
* Add OMEntity model
* Test OMEntity
* Update repr
* Fix __str__
* Add entity ref map
* Test serializer for backend
* Fix tests
* Fix serializer
* Test runner
* Add runner tests
* Update docs
* Format
2023-12-01 06:29:44 +01:00
Pere Miquel Brull
f6a87ee02a
Fix #12082 - Bump PyAthena version ( #13464 )
2023-10-09 20:47:19 +02:00
Pere Miquel Brull
6c0e9f5061
Part of #7272 - Centralize Workflows, Status, and Exception Management ( #13029 )
...
* Prep changes
* Prep changes
* prep changes
* Update imports
* Format
* Prep delete
* Prep delete
* Fix sink
* Prep test
* Commit
* passing either
* passing either
* Prep Either
* Metadata source with Either
* Update status
* Merge remote-tracking branch 'upstream/main' into issue-7272
* Format
* Linting
* Linting
* Linting
* Linting
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Fix tests
* Comments
2023-08-30 15:49:42 +02:00
Pere Miquel Brull
095f15064c
Fix #11752 - Multiple inlets in AirflowLineageOperator ( #11945 )
...
* Fix multiple inlets in AirflowLineageOperator
* Update dag file
2023-06-12 07:01:19 +02:00
Pere Miquel Brull
9ebc0cd5e4
Fix sample usage DAG directory ( #11276 )
2023-04-26 06:53:10 +02:00
Ayush Shah
9d11029ec8
Fixes 10351: Fixes Metrics Computation, Samping, test suites and partioning ( #10603 )
...
Co-authored-by: Teddy Crepineau <teddy.crepineau@gmail.com>
2023-04-11 20:58:31 +05:30
Pere Miquel Brull
fb15c896b3
Handle XLets in groups for AirflowLineageRunner ( #10114 )
...
* Handle XLets in groups
* Linting
* Linting
2023-02-07 06:49:46 +01:00
Pere Miquel Brull
a4521fd664
Fix #6562 - Sources have their own package ( #9521 )
...
Fix #6562 - Sources have their own package (#9521 )
2022-12-27 15:00:22 +01:00
Pere Miquel Brull
ec6ebb3694
Organise dynamic importers ( #9513 )
...
Organise dynamic importers (#9513 )
2022-12-23 12:17:06 +01:00
Pere Miquel Brull
3b7ae73473
Airflow e2e integration test ( #9363 )
...
* Prep airflow operator integration tests
* Add integration test to Makefile
2022-12-16 19:52:12 -08:00
Pere Miquel Brull
0963eac48e
Fix #9182 - Airflow Lineage Operator & Airflow lineage state ( #9206 )
2022-12-10 19:54:41 +01:00
Onkar Ravgan
23468fb868
Fixed Broken docs links ( #9125 )
2022-12-02 15:42:06 +01:00
Pere Miquel Brull
339abc5bf3
Prepare Ingestion Base Docker image ( #8065 )
...
Prepare Ingestion Base Docker image (#8065 )
2022-10-11 07:50:49 +02:00
mohitdeuex
cceff48f0d
Default token ( #7695 )
2022-09-23 20:03:54 +05:30
Akash Jain
fc8312c5fb
feat: Refactor openmetadata/ingestion docker image ( #7456 )
...
* centralize openmetadata/ingestion docker image
* update volume mappings; fix ingestion tag!
* fix PR comments
* fix cypress mysql tests!
2022-09-19 09:20:54 +05:30
Nahuel
ed58c27f76
Remove unused property extra_hosts from docker-compose files ( #7096 )
...
* Remove unused properties from docker-compose files
* Minor fix in run_local_docker script
2022-08-31 21:30:24 +02:00
Ashish Gupta
b59878e1b5
Fix ( #6984 ) doc links in OpenMetaData ( #6986 )
...
* Fix doc links in OpenMetaData
* Link updated
* remove unwanted file
* changes as per comments
* Changes as per comments
2022-08-29 19:38:54 +05:30
Pere Miquel Brull
a6dd100be4
Fix metadata es DAG ( #5933 )
2022-07-07 12:15:09 -07:00
Sriharsha Chintalapani
8528c51b9c
Fix #5516 : Refactor elasticsearch indexes to be same as entity specifications ( #5573 )
...
* Fix #5516 : Refactor elasticsearch indexes to be same as entity specifications
Co-authored-by: darth-coder00 <aashit@getcollate.io>
Co-authored-by: Sachin-chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: Pere Miquel Brull <peremiquelbrull@gmail.com>
Co-authored-by: Shailesh Parmar <shailesh.parmar.webdev@gmail.com>
2022-06-27 01:19:27 -07:00
Pere Miquel Brull
bb4ac68ee3
Add backend connection for airflow dag extraction
2022-06-24 20:01:50 +02:00
Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices ( #5446 )
...
Fix #5367 - MlFlow connector & MLServices (#5446 )
2022-06-21 14:54:36 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector ( #5125 )
...
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125 )
2022-05-25 08:35:16 +02:00
Pere Miquel Brull
c6915ca510
Fix #4923 - Presto ingestion & SQLAlchemy test connection refactor ( #4946 )
...
Fix #4923 - Presto ingestion & SQLAlchemy test connection refactor (#4946 )
2022-05-14 06:44:43 +02:00
Ayush Shah
61591058fb
Added support openmetadata extraction - Ingestion ( #4195 )
2022-04-19 19:37:41 +05:30
Pere Miquel Brull
2aecebd595
Fix #4189 - Fix airflow lineage callback ( #4191 )
...
Fix #4189 - Fix airflow lineage callback (#4191 )
2022-04-18 15:13:26 +02:00
Ayush Shah
4f5c5a9e23
Ingestion - fix pytest workflow errors and modify airflow pipeline ( #4178 )
2022-04-18 11:36:21 +05:30
Sriharsha Chintalapani
06cc35d048
Fix #4082 : Do not start slack event publisher if there is no config ( #4083 )
2022-04-12 17:25:23 -07:00
Ayush Shah
15766e85a6
Sample Usage fixed ( #4065 )
2022-04-12 14:57:18 +05:30
Pere Miquel Brull
6a6507e754
Fix #3962 - Profiler uses DatabaseSchema & Sample Data fix ( #4056 )
2022-04-12 13:40:59 +05:30
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field ( #3909 )
...
* Fix sample data DAG
* Fix callback imports
* Use --set-default-enum-member in generate
* Format
* Add faulty merge hard_delete
* Fix airflow lineage, improve naming and fix lineage tests
* Add mysql url test
* Add mysql url test
* Update CI name
* Fix test ometa endpoint
* Format
* Fix metadata config
2022-04-07 18:22:50 +05:30
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service ( #3804 )
...
* Fix #3659 Refactor Service Connection String to be specific to per service
* Simplify and centralize Airflow Pipeline info for REST (#3740 )
* Remove code
* Modified Configs based on refactoring schema (#3816 )
* Clean WorkflowContext
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Ayush Shah
351db56687
Added fix for Sample_Usage not being ingested in Local docker ( #3650 )
2022-03-27 22:38:13 -07:00
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln ( #3370 )
...
Fix #3104 - Remove unused imports with pycln (#3370 )
2022-03-14 06:59:15 +01:00
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG ( #2337 )
...
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
2022-01-22 16:37:56 +05:30
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink ( #1956 )
...
* Query Usage fixed
* Usage updated
* Reformatted change
* Usage files reformatted
2022-01-04 17:12:10 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed ( #1788 )
...
* Metabase and Snowflake Fixed
* Removed DBT
2021-12-16 11:57:43 +05:30
parthp2107
64386b035e
added new tests for tags page ( #1621 )
...
* added new tests for tags page
* addressing reviewdog findings
* Added Sample-usage service_name
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-12-08 16:54:48 +05:30
Ayush Shah
0e205d93dd
Added DBT to airflow, modified tables.json - added profile data, fixed user's displayName ( #1501 )
...
* User's Name fixed, added profile data, added dbt pipeline under airflow
* service name changed
2021-12-01 08:32:29 -08:00
parthp2107
a9e70e6a89
License header update ( #1498 )
...
* updated liscense header
* updated license header
* updated license header
* updated license header
* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Ayush Shah
47a97e7be0
FQN changed from bigquery to bigquery_gcp
2021-11-29 16:57:39 +05:30
parthp2107
4a5f5e69b3
uncommenting add-user test ( #1097 )
...
* uncommenting add-user test
* quick-fix
* Ingestion script modified
* Airflow sample_usage modified, added sleep in curl commands
* adding chrome options for failing tests
* addressing reviewdog finding
* testing
* addressing reviewdog findings
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-11-14 20:16:39 -08:00
