mirror of
https://github.com/datahub-project/datahub.git
synced 2025-11-10 16:32:26 +00:00
feat(docs): Adds docs on K8s scheduled ingestion (#5984)
This commit is contained in:
parent
e24e5b61b9
commit
7e5f44a481
@ -150,6 +150,7 @@ module.exports = {
|
|||||||
"metadata-ingestion/schedule_docs/intro",
|
"metadata-ingestion/schedule_docs/intro",
|
||||||
"metadata-ingestion/schedule_docs/cron",
|
"metadata-ingestion/schedule_docs/cron",
|
||||||
"metadata-ingestion/schedule_docs/airflow",
|
"metadata-ingestion/schedule_docs/airflow",
|
||||||
|
"metadata-ingestion/schedule_docs/kubernetes",
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
// {
|
// {
|
||||||
|
|||||||
47
metadata-ingestion/schedule_docs/kubernetes.md
Normal file
47
metadata-ingestion/schedule_docs/kubernetes.md
Normal file
@ -0,0 +1,47 @@
|
|||||||
|
# Using Kubernetes
|
||||||
|
|
||||||
|
If you have deployed DataHub using our official [helm charts](https://github.com/acryldata/datahub-helm) you can use the
|
||||||
|
datahub ingestion cron subchart to schedule ingestions.
|
||||||
|
|
||||||
|
Here is an example of what that configuration would look like in your **values.yaml**:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
datahub-ingestion-cron:
|
||||||
|
enabled: true
|
||||||
|
crons:
|
||||||
|
mysql:
|
||||||
|
schedule: "0 * * * *" # Every hour
|
||||||
|
recipe:
|
||||||
|
configmapName: recipe-config
|
||||||
|
fileName: mysql_recipe.yml
|
||||||
|
```
|
||||||
|
|
||||||
|
This assumes the pre-existence of a Kubernetes ConfigMap which holds all recipes being scheduled in the same namespace as
|
||||||
|
where the cron jobs will be running.
|
||||||
|
|
||||||
|
An example could be:
|
||||||
|
```yaml
|
||||||
|
apiVersion: v1
|
||||||
|
kind: ConfigMap
|
||||||
|
metadata:
|
||||||
|
name: recipe-config
|
||||||
|
data:
|
||||||
|
mysql_recipe.yml: |-
|
||||||
|
source:
|
||||||
|
type: mysql
|
||||||
|
config:
|
||||||
|
# Coordinates
|
||||||
|
host_port: <MYSQL HOST>:3306
|
||||||
|
database: dbname
|
||||||
|
|
||||||
|
# Credentials
|
||||||
|
username: root
|
||||||
|
password: example
|
||||||
|
|
||||||
|
sink:
|
||||||
|
type: datahub-rest
|
||||||
|
config:
|
||||||
|
server: http://<GMS_HOST>:8080
|
||||||
|
```
|
||||||
|
|
||||||
|
For more information, please see the [documentation](https://github.com/acryldata/datahub-helm/tree/master/charts/datahub/subcharts/datahub-ingestion-cron) of this sub-chart.
|
||||||
Loading…
x
Reference in New Issue
Block a user