feat(docs): Adds docs on K8s scheduled ingestion (#5984)

This commit is contained in:
Pedro Silva 2022-09-20 10:34:42 +01:00 committed by GitHub
parent e24e5b61b9
commit 7e5f44a481
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 48 additions and 0 deletions

View File

@ -150,6 +150,7 @@ module.exports = {
"metadata-ingestion/schedule_docs/intro",
"metadata-ingestion/schedule_docs/cron",
"metadata-ingestion/schedule_docs/airflow",
"metadata-ingestion/schedule_docs/kubernetes",
],
},
// {

View File

@ -0,0 +1,47 @@
# Using Kubernetes
If you have deployed DataHub using our official [helm charts](https://github.com/acryldata/datahub-helm) you can use the
datahub ingestion cron subchart to schedule ingestions.
Here is an example of what that configuration would look like in your **values.yaml**:
```yaml
datahub-ingestion-cron:
enabled: true
crons:
mysql:
schedule: "0 * * * *" # Every hour
recipe:
configmapName: recipe-config
fileName: mysql_recipe.yml
```
This assumes the pre-existence of a Kubernetes ConfigMap which holds all recipes being scheduled in the same namespace as
where the cron jobs will be running.
An example could be:
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: recipe-config
data:
mysql_recipe.yml: |-
source:
type: mysql
config:
# Coordinates
host_port: <MYSQL HOST>:3306
database: dbname
# Credentials
username: root
password: example
sink:
type: datahub-rest
config:
server: http://<GMS_HOST>:8080
```
For more information, please see the [documentation](https://github.com/acryldata/datahub-helm/tree/master/charts/datahub/subcharts/datahub-ingestion-cron) of this sub-chart.