mirror of
https://github.com/datahub-project/datahub.git
synced 2025-11-03 20:27:50 +00:00
feat(docs): Adds docs on K8s scheduled ingestion (#5984)
This commit is contained in:
parent
e24e5b61b9
commit
7e5f44a481
@ -150,6 +150,7 @@ module.exports = {
|
||||
"metadata-ingestion/schedule_docs/intro",
|
||||
"metadata-ingestion/schedule_docs/cron",
|
||||
"metadata-ingestion/schedule_docs/airflow",
|
||||
"metadata-ingestion/schedule_docs/kubernetes",
|
||||
],
|
||||
},
|
||||
// {
|
||||
|
||||
47
metadata-ingestion/schedule_docs/kubernetes.md
Normal file
47
metadata-ingestion/schedule_docs/kubernetes.md
Normal file
@ -0,0 +1,47 @@
|
||||
# Using Kubernetes
|
||||
|
||||
If you have deployed DataHub using our official [helm charts](https://github.com/acryldata/datahub-helm) you can use the
|
||||
datahub ingestion cron subchart to schedule ingestions.
|
||||
|
||||
Here is an example of what that configuration would look like in your **values.yaml**:
|
||||
|
||||
```yaml
|
||||
datahub-ingestion-cron:
|
||||
enabled: true
|
||||
crons:
|
||||
mysql:
|
||||
schedule: "0 * * * *" # Every hour
|
||||
recipe:
|
||||
configmapName: recipe-config
|
||||
fileName: mysql_recipe.yml
|
||||
```
|
||||
|
||||
This assumes the pre-existence of a Kubernetes ConfigMap which holds all recipes being scheduled in the same namespace as
|
||||
where the cron jobs will be running.
|
||||
|
||||
An example could be:
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: recipe-config
|
||||
data:
|
||||
mysql_recipe.yml: |-
|
||||
source:
|
||||
type: mysql
|
||||
config:
|
||||
# Coordinates
|
||||
host_port: <MYSQL HOST>:3306
|
||||
database: dbname
|
||||
|
||||
# Credentials
|
||||
username: root
|
||||
password: example
|
||||
|
||||
sink:
|
||||
type: datahub-rest
|
||||
config:
|
||||
server: http://<GMS_HOST>:8080
|
||||
```
|
||||
|
||||
For more information, please see the [documentation](https://github.com/acryldata/datahub-helm/tree/master/charts/datahub/subcharts/datahub-ingestion-cron) of this sub-chart.
|
||||
Loading…
x
Reference in New Issue
Block a user