Sriharsha Chintalapani
891ff465a0
Fix #2838 : Fix airflow example DAG to import airflow libs ( #2839 )
2022-02-18 10:47:07 +05:30
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG ( #2337 )
...
* Simplify sample data DAG
* Remove mkdir
* Generate sources before running compose
* Generate sources to install models to ingestion image
* Add python-on-whales for docker --start
* Remove python-on-whales from base
* Install venv
* Setup python
2022-01-22 16:37:56 +05:30
Alberto Miorin
c8de136cca
First part of #2091 : Python code in branch main is always formatted correctly ( #2093 )
2022-01-07 06:17:48 +01:00
Ayush Shah
19e5f737ff
Usage fix - removed service_name from bulk_sink ( #1956 )
...
* Query Usage fixed
* Usage updated
* Reformatted change
* Usage files reformatted
2022-01-04 17:12:10 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup ( #1730 )
...
* Fix typo
* Clean setup
* Update ingestion local image to be barebone on connector dependencies
* Prepare ingestion connectors base image
* Add system dependencies
* Prepare docker CLI
* Add docker provider
* Prepare entrypoint for the image
* Remove DBT pipeline as per Issue 1658
* Add TODO for ingestion build
* Bind docker socket
* Update comment
* Update README
* Use DockerOperator in sample data
* Build images with latest tag
* Prepare symlink to pass the volume to the DockerOperator
* Update README
* Prepare Base image for CI
* COPY multiple files into dir
* COPY multiple files into dir
* Remove DBT source as is now part of table ingestion
* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Ayush Shah
06712fa425
Metabase and Snowflake Fixed ( #1788 )
...
* Metabase and Snowflake Fixed
* Removed DBT
2021-12-16 11:57:43 +05:30
parthp2107
64386b035e
added new tests for tags page ( #1621 )
...
* added new tests for tags page
* addressing reviewdog findings
* Added Sample-usage service_name
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-12-08 16:54:48 +05:30
Ayush Shah
0e205d93dd
Added DBT to airflow, modified tables.json - added profile data, fixed user's displayName ( #1501 )
...
* User's Name fixed, added profile data, added dbt pipeline under airflow
* service name changed
2021-12-01 08:32:29 -08:00
parthp2107
a9e70e6a89
License header update ( #1498 )
...
* updated liscense header
* updated license header
* updated license header
* updated license header
* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Ayush Shah
47a97e7be0
FQN changed from bigquery to bigquery_gcp
2021-11-29 16:57:39 +05:30
parthp2107
4a5f5e69b3
uncommenting add-user test ( #1097 )
...
* uncommenting add-user test
* quick-fix
* Ingestion script modified
* Airflow sample_usage modified, added sleep in curl commands
* adding chrome options for failing tests
* addressing reviewdog finding
* testing
* addressing reviewdog findings
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-11-14 20:16:39 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added ( #1158 )
...
* Script modified - supports running from different locations
* Docker support from Python CLI
* Docker plugin setup.py
* Paths for latest and local dockers updated
* Resolved Comments - Docker CLI optimized, timestamp added
* help attribute added to options
* Docker clean code refactored
2021-11-12 10:30:28 -08:00
Ayush Shah
c95f1542eb
Docker fix ( #1101 )
...
* Docker Fix, User Ingestion Fix
* Docker file optimized
* ES docker version rolledback
* Healthcheck removed - incompatible with 3.9
* Docker & Airflow Config files updated with - resolved comments
* Openmetadata managed Airflow apis support added
* Dockerfile updated
* Docker file Updated with release package
* Elasticsearch allocated memory increased
2021-11-10 21:22:32 -08:00
Akash Jain
06b2f70e3e
kubernetes docker image compatibility for #1026 ( #1042 )
2021-11-03 08:18:52 -07:00
Ayush Shah
93921814af
Docker fix - latest release changes ( #983 )
...
* Docker fix - Architecture, MySQL
* Docker Airflow API Dagrun support
* Docker latest changes modified
2021-10-30 09:05:30 -07:00
Ayush Shah
759574a8de
Ingestion Optimization - Sample Users, Dockerfiles, Removal of Pandas ( #935 )
...
* Sample Profile Data for Sample Tables (#815 )
* Sample Profile Data for Sample Tables
* Disabling Profile as Default
* Added Sample Profile Data to 3 additional sample tables
* Sample Tables fixed (#850 )
* Pydantic fix, Docker update (#860 )
* Setup.py Modified with openmetadata-airflow package, docker update
* Setup.py Modified
* Update setup.py
* Removed Pandas from Sample Data
* Sample Users added under sample data
* Sample User Standalone pipelines and modules removed
* Docker release package updated
* Dockerfile updated, removed redundant files
* Setup.py removed from ingestion src directory
* User Resource failing check resolved
* Modifying Usage Columns Datatype
2021-10-26 09:14:24 -07:00
Ayush Shah
72e4b8831f
Removing Cron from all configs ( #773 )
...
* Removing Cron from all configs
* Removing Cron from Documentation
2021-10-14 15:21:47 -07:00
Ayush Shah
85b6b72848
Airflow docker ( #762 )
...
* Airflow Docker implementation - Ingestion
* Dockerfiles modified
2021-10-14 07:46:24 -07:00
Sriharsha Chintalapani
bfec0bfbed
Ingestion: Airflow integration to ingest metadata about pipelines and tasks ( #609 )
...
* [WIP] Airlfow integration
* [WIP] Airlfow integration
* [WIP] airflow integration
* [WIP] Airflow
* [WIP] Airflow
* Fix #608 : Ingestion: Airflow integration to ingest metadata about pipelines and tasks
* Fix #608 : Ingestion: Airflow integration to ingest metadata about pipelines and tasks
* Update DashboardServiceResource.java
Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-09-29 11:32:09 -07:00
Sriharsha Chintalapani
ed698dc40f
Fix #466 : Fix airflow example dag ( #467 )
2021-09-10 22:28:23 -07:00
Suresh Srinivas
8256de30e9
Ingestion: Refactor, docs, airflow example
2021-08-14 10:18:52 -07:00
