22 Commits

Author SHA1 Message Date
Nahuel
ed58c27f76
Remove unused property extra_hosts from docker-compose files (#7096)
* Remove unused properties from docker-compose files

* Minor fix in run_local_docker script
2022-08-31 21:30:24 +02:00
Teddy
811f640a18
Fixes #6891 by implementing support for VARIANT DType (#7084) 2022-08-31 19:01:00 +02:00
Nahuel
30ff470922
Reduce the CI times for py-tests (#6352)
* Testing if we can reduce the CI times

* Update POM and run_local_docker script
2022-07-26 17:47:03 +02:00
Pere Miquel Brull
bbfb0dd7d2
Give more verbosity for CI (#6048)
Give more verbosity for CI (#6048)
2022-07-18 14:44:50 +02:00
Pere Miquel Brull
8e121fb306
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125)
Fix #4096 - PipelineService JSON refactor & Airflow metadata source connector (#5125)
2022-05-25 08:35:16 +02:00
Pere Miquel Brull
6a6507e754
Fix #3962 - Profiler uses DatabaseSchema & Sample Data fix (#4056) 2022-04-12 13:40:59 +05:30
Pere Miquel Brull
93525aea0d
Fix #3823 & Fix generate default Enum field (#3909)
* Fix sample data DAG

* Fix callback imports

* Use --set-default-enum-member in generate

* Format

* Add faulty merge hard_delete

* Fix airflow lineage, improve naming and fix lineage tests

* Add mysql url test

* Add mysql url test

* Update CI name

* Fix test ometa endpoint

* Format

* Fix metadata config
2022-04-07 18:22:50 +05:30
Sriharsha Chintalapani
b14c8dc2c4
Issue-3685: Variable based separator used for fullyQualifiedName instead of hardcoded . for Python and make the separator to : (#3778)
* Issue-3685: Variable based separator used for fullyQualifiedName instead of hardcoded . for Python and make the sepearator to :

* Fix failing test

* Use colon for run_local_docker validation

* Update tests FQDN

* Update tests FQDN

Co-authored-by: Sachin-chaurasiya <sachinchaurasiyachotey87@gmail.com>
Co-authored-by: pmbrull <peremiquelbrull@gmail.com>
2022-03-31 19:20:27 +05:30
Ayush Shah
351db56687
Added fix for Sample_Usage not being ingested in Local docker (#3650) 2022-03-27 22:38:13 -07:00
Ayush Shah
9a620c380d
Fixing Metadata indexing issue (#3580) 2022-03-22 14:37:41 +05:30
Ayush Shah
b865d85d85
Docker Calculation fix + Reset DB command added (#3459) 2022-03-21 10:28:58 -07:00
Pere Miquel Brull
1eb3f1ad41
[issue-2322] - Backup CLI (#2349)
* Prepare cli module

* Check call

* Prepare backup

* Run pytest on schema changes

* Remove traces

* Add backup plugin

* run generate during CI

* Install after generate

* Address logger hotspot
2022-01-22 12:08:14 -08:00
Pere Miquel Brull
6693b4ae40
[issue-2262] - Simplify sample data DAG (#2337)
* Simplify sample data DAG

* Remove mkdir

* Generate sources before running compose

* Generate sources to install models to ingestion image

* Add python-on-whales for docker --start

* Remove python-on-whales from base

* Install venv

* Setup python
2022-01-22 16:37:56 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Ayush Shah
07924f80c3
Added Metabase Connector (#1726)
* Added Metabase Connector

* Metabase Added support for lineage

* Modified Metabase as per comments

* Metabase filter patterns added

* Update databaseService.json
2021-12-15 09:53:55 -08:00
parthp2107
a9e70e6a89
License header update (#1498)
* updated liscense header

* updated license header

* updated license header

* updated license header

* addressing stylecheck findings
2021-12-01 12:46:28 +05:30
Ayush Shah
47a97e7be0 FQN changed from bigquery to bigquery_gcp 2021-11-29 16:57:39 +05:30
Ayush Shah
11e2229972
Added OMeta API, removed wget (#1376)
* Added OMeta API, removed wget

* Added curl instead of Wget in run_local_docker.sh
2021-11-24 11:57:33 -08:00
Ayush Shah
2111f6c938
Docker Commands Modified (#1199)
* Docker ingestion modified

* Command line commands updated
2021-11-16 00:25:40 -08:00
Ayush Shah
5dc3bb9297
Docker Support from Python added (#1158)
* Script modified - supports running from different locations

* Docker support from Python CLI

* Docker plugin setup.py

* Paths for latest and local dockers updated

* Resolved Comments - Docker CLI optimized, timestamp added

* help attribute added to options

* Docker clean code refactored
2021-11-12 10:30:28 -08:00
Pere Miquel Brull
1291bf6ec0
[Issue-892] - GET operations return service (#916)
* Force image build in local docker

* Use service as default field query param for entities with required service

* Return service in List

* Naming convention

* Update Repository setFields to include service

* Revert resource changes

* Revert resource changes

* Update tests to check we always return the service, even if not specified

* Return service for database
2021-10-24 10:26:28 -07:00
Ayush Shah
319f7a795c
Docker local (#723)
* added docker file for local build and ingestion

* Docker Modified for local setup

* Ingestion script optimized

* Added script to directly run Local Docker

Co-authored-by: Parth Panchal <parthpanchal@Parths-MacBook-Air.local>
2021-10-09 10:58:33 -07:00