Pere Miquel Brull
a14de6eee1
Fix #5367 - MlFlow connector & MLServices ( #5446 )
...
Fix #5367 - MlFlow connector & MLServices (#5446 )
2022-06-21 14:54:36 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector ( #5125 )
...
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125 )
2022-05-25 08:35:16 +02:00
Pere Miquel Brull
c6915ca510
Fix #4923 - Presto ingestion & SQLAlchemy test connection refactor ( #4946 )
...
Fix #4923 - Presto ingestion & SQLAlchemy test connection refactor (#4946 )
2022-05-14 06:44:43 +02:00
Ayush Shah
61591058fb
Added support openmetadata extraction - Ingestion ( #4195 )
2022-04-19 19:37:41 +05:30
Pere Miquel Brull
2aecebd595
Fix #4189 - Fix airflow lineage callback ( #4191 )
...
Fix #4189 - Fix airflow lineage callback (#4191 )
2022-04-18 15:13:26 +02:00
Ayush Shah
4f5c5a9e23
Ingestion - fix pytest workflow errors and modify airflow pipeline ( #4178 )
2022-04-18 11:36:21 +05:30
Sriharsha Chintalapani
06cc35d048
Fix #4082 : Do not start slack event publisher if there is no config ( #4083 )
2022-04-12 17:25:23 -07:00
Ayush Shah
15766e85a6
Sample Usage fixed ( #4065 )
2022-04-12 14:57:18 +05:30
Pere Miquel Brull
6a6507e754
Fix #3962 - Profiler uses DatabaseSchema & Sample Data fix ( #4056 )
2022-04-12 13:40:59 +05:30
Pere Miquel Brull
d6b49f070c
Fix #3951 - OpenMetadata Airflow REST ( #4034 )
...
Fix #3951 - OpenMetadata Airflow REST (#4034 )
2022-04-11 18:38:26 +02:00
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field ( #3909 )
...
* Fix sample data DAG
* Fix callback imports
* Use --set-default-enum-member in generate
* Format
* Add faulty merge hard_delete
* Fix airflow lineage, improve naming and fix lineage tests
* Add mysql url test
* Add mysql url test
* Update CI name
* Fix test ometa endpoint
* Format
* Fix metadata config
2022-04-07 18:22:50 +05:30
Sriharsha Chintalapani
7b3e459eb3
Fix #3659 Refactor Service Connection String to be specific to per service ( #3804 )
...
* Fix #3659 Refactor Service Connection String to be specific to per service
* Simplify and centralize Airflow Pipeline info for REST (#3740 )
* Remove code
* Modified Configs based on refactoring schema (#3816 )
* Clean WorkflowContext
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2022-04-04 12:46:09 -07:00
Ayush Shah
351db56687
Added fix for Sample_Usage not being ingested in Local docker ( #3650 )
2022-03-27 22:38:13 -07:00
Pere Miquel Brull
130bbb0c5c
Fix #3104 - Remove unused imports with pycln ( #3370 )
...
Fix #3104 - Remove unused imports with pycln (#3370 )
2022-03-14 06:59:15 +01:00
Sriharsha Chintalapani
891ff465a0
Fix #2838 : Fix airflow example DAG to import airflow libs ( #2839 )
2022-02-18 10:47:07 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG ( #2337 )
...
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
2022-01-22 16:37:56 +05:30
Alberto Miorin
c8de136cca
First part of #2091 : Python code in branch main is always formatted correctly ( #2093 )
2022-01-07 06:17:48 +01:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink ( #1956 )
...
* Query Usage fixed
* Usage updated
* Reformatted change
* Usage files reformatted
2022-01-04 17:12:10 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed ( #1788 )
...
* Metabase and Snowflake Fixed
* Removed DBT
2021-12-16 11:57:43 +05:30
parthp2107
64386b035e
added new tests for tags page ( #1621 )
...
* added new tests for tags page
* addressing reviewdog findings
* Added Sample-usage service_name
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-12-08 16:54:48 +05:30
Ayush Shah
0e205d93dd
Added DBT to airflow, modified tables.json - added profile data, fixed user's displayName ( #1501 )
...
* User's Name fixed, added profile data, added dbt pipeline under airflow
* service name changed
2021-12-01 08:32:29 -08:00
parthp2107
a9e70e6a89
License header update ( #1498 )
...
* updated liscense header
* updated license header
* updated license header
* updated license header
* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Ayush Shah
47a97e7be0
FQN changed from bigquery to bigquery_gcp
2021-11-29 16:57:39 +05:30
parthp2107
4a5f5e69b3
uncommenting add-user test ( #1097 )
...
* uncommenting add-user test
* quick-fix
* Ingestion script modified
* Airflow sample_usage modified, added sleep in curl commands
* adding chrome options for failing tests
* addressing reviewdog finding
* testing
* addressing reviewdog findings
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-11-14 20:16:39 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added ( #1158 )
...
* Script modified - supports running from different locations
* Docker support from Python CLI
* Docker plugin setup.py
* Paths for latest and local dockers updated
* Resolved Comments - Docker CLI optimized, timestamp added
* help attribute added to options
* Docker clean code refactored
2021-11-12 10:30:28 -08:00
Ayush Shah
c95f1542eb
Docker fix ( #1101 )
...
* Docker Fix, User Ingestion Fix
* Docker file optimized
* ES docker version rolledback
* Healthcheck removed - incompatible with 3.9
* Docker & Airflow Config files updated with - resolved comments
* Openmetadata managed Airflow apis support added
* Dockerfile updated
* Docker file Updated with release package
* Elasticsearch allocated memory increased
2021-11-10 21:22:32 -08:00
Akash Jain
06b2f70e3e
kubernetes docker image compatibility for #1026 ( #1042 )
2021-11-03 08:18:52 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes ( #983 )
...
* Docker fix - Architecture, MySQL
* Docker Airflow API Dagrun support
* Docker latest changes modified
2021-10-30 09:05:30 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas ( #935 )
...
* Sample Profile Data for Sample Tables (#815 )
* Sample Profile Data for Sample Tables
* Disabling Profile as Default
* Added Sample Profile Data to 3 additional sample tables
* Sample Tables fixed (#850 )
* Pydantic fix, Docker update (#860 )
* Setup.py Modified with openmetadata-airflow package, docker update
* Setup.py Modified
* Update setup.py
* Removed Pandas from Sample Data
* Sample Users added under sample data
* Sample User Standalone pipelines and modules removed
* Docker release package updated
* Dockerfile updated, removed redundant files
* Setup.py removed from ingestion src directory
* User Resource failing check resolved
* Modifying Usage Columns Datatype
2021-10-26 09:14:24 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs ( #773 )
...
* Removing Cron from all configs
* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Ayush Shah
85b6b72848
Airflow docker ( #762 )
...
* Airflow Docker implementation - Ingestion
* Dockerfiles modified
2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
bfec0bfbed
Ingestion: Airflow integration to ingest metadata about pipelines and tasks ( #609 )
...
* [WIP] Airlfow integration
* [WIP] Airlfow integration
* [WIP] airflow integration
* [WIP] Airflow
* [WIP] Airflow
* Fix #608 : Ingestion: Airflow integration to ingest metadata about pipelines and tasks
* Fix #608 : Ingestion: Airflow integration to ingest metadata about pipelines and tasks
* Update DashboardServiceResource.java
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-09-29 11:32:09 -07:00
Sriharsha Chintalapani
ed698dc40f
Fix #466 : Fix airflow example dag ( #467 )
2021-09-10 22:28:23 -07:00
Suresh Srinivas
8256de30e9
Ingestion: Refactor, docs, airflow example
2021-08-14 10:18:52 -07:00
