14 Commits

Author SHA1 Message Date
Imri Paran
247e05a83d
MINOR: Update README.md python version (#15893) 2024-04-18 21:33:16 +05:30
IceS2
ce3f124a33
MINOR: Add Multithread Documentation (#15706)
* Add Multithread Documentation

* Add general considerations
2024-03-27 08:30:16 +01:00
IceS2
e7c9d6aa7f
FIXES 15215: Implement initial Multithreading approach for the Metadata Ingestion on Databases (#15130)
* Implement Initial MultiThread suggestion

* Update all the ingestion sources to use the new ContextManager

* Fix missing wraps on decorator

* Fix Unittests

* Fix linters

* Fix linters

* Fix BigQuery UnitTests

* Add UnitTests to the newly created code

* Fix unittest

* change the threads from table to schemas

* Update README.md

* Small change suggested by Sonar

* Slight change to test a different way to multithread over tables

* Debug changes

* More multithread tests

* Remove uneeded wait time

* Testing

* refactor code based on removal of time.sleep

* Fix wrong paste

* Improve ExecutionTimeContextManager

* Fix missing .get() and unit tests

* Fix conflicting changes

* Update Multithread logic with the incremental extraction

* Fix linters

* Fix unittest

* Remove commented code

* Fix Unittests

* Fix checkstyle

* Change default to threads = 1
2024-03-25 18:20:40 +01:00
Pere Miquel Brull
e3ae3a8a33
[WIP] - Add Scarf gateway (#9652)
* Update ingestion Readme with scarf

* Test scarf docker

* Update ingestion Readme with scarf

* Test scarf docker

* Add docker domain
2023-01-11 09:33:09 +01:00
Onkar Ravgan
23468fb868
Fixed Broken docs links (#9125) 2022-12-02 15:42:06 +01:00
Ashish Gupta
b59878e1b5
Fix (#6984) doc links in OpenMetaData (#6986)
* Fix doc links in OpenMetaData

* Link updated

* remove unwanted file

* changes as per comments

* Changes as per comments
2022-08-29 19:38:54 +05:30
Pere Miquel Brull
2ec32017bb
[issue-1698] - Airflow DockerOperator setup (#1730)
* Fix typo

* Clean setup

* Update ingestion local image to be barebone on connector dependencies

* Prepare ingestion connectors base image

* Add system dependencies

* Prepare docker CLI

* Add docker provider

* Prepare entrypoint for the image

* Remove DBT pipeline as per Issue 1658

* Add TODO for ingestion build

* Bind docker socket

* Update comment

* Update README

* Use DockerOperator in sample data

* Build images with latest tag

* Prepare symlink to pass the volume to the DockerOperator

* Update README

* Prepare Base image for CI

* COPY multiple files into dir

* COPY multiple files into dir

* Remove DBT source as is now part of table ingestion

* Build docker base in run_local_docker
2021-12-18 16:41:38 +01:00
Akash Jain
96c65e7ebd
fix: remove spaCy dependencies from setup.py (#1362)
* remove spaCy dependencies from setup.py

* Spacy, PII and Processor dependencies removed

Co-authored-by: Ayush Shah <ayush@getcollate.io>
2021-11-24 12:23:17 -08:00
Sriharsha Chintalapani
27c439a53d
Update README.md
update ingestion docs to point to docs.open-metadata.org
2021-11-24 09:06:56 -08:00
Sriharsha Chintalapani
d2458777c8
Update README.md 2021-11-24 07:18:33 -08:00
Pere Miquel Brull
67ac286f56
[Issue-1285] - Update ingestion package generation (#1295)
* Remove generated sources

* Ignore python generated sources

* Keep the generated dir

* Prepare publish recipe

* Simplify recipe

* Update README with generation

* Update docs with python local setup information

* Generate python sources in the CI
2021-11-21 08:02:31 -08:00
Pere Miquel Brull
f91bcc03f6
[Issue-877] - High Level API (#890)
* First approach

* Test model

* Validate endpoints

* Update schema codegen

* Prepare reqs dev

* Rename titles and codegen

* Update README

* Keep lineage title as addLineage

* Update pydantic classes

* Update API wrapper & tests

* Handle services methods

* Prepare tests
2021-10-21 14:51:38 -07:00
Ayush Shah
a3f4398284 README and Setup.py Modification 2021-08-13 02:03:39 +05:30
Ayush Shah
72a355bd5d Ingestion Setup.py modification 2021-08-13 01:40:56 +05:30