yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-03 20:27:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(nocode): Add datahub-upgrade job to helm chart and set version to v0.8.1 (#2643)

Browse Source
This commit is contained in:
Dexter Lee 2021-06-04 15:13:32 -07:00 committed by GitHub
parent a483933eab
commit ee454eb5d6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
21 changed files with 349 additions and 77 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
datahub-kubernetes/README.md
View File

@ -84,10 +84,10 @@ prerequisites-zookeeper-0 1/1 Running 0
deploy Datahub by running the following
```(shell)
helm install datahub datahub/ --values datahub/quickstart-values.yaml
helm install datahub datahub/
```
Values in [quickstart-values.yaml](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub/quickstart-values.yaml)
Values in [values.yaml](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/datahub/values.yaml)
have been preset to point to the dependencies deployed using the [prerequisites](https://github.com/linkedin/datahub/tree/master/datahub-kubernetes/prerequisites)
chart with release name "prerequisites". If you deployed the helm chart using a different release name, update the quickstart-values.yaml file accordingly before installing.

4
datahub-kubernetes/datahub/Chart.yaml
View File

@ -4,10 +4,10 @@ description: A Helm chart for LinkedIn DataHub
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
version: 0.1.2
version: 0.2.0
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application.
appVersion: latest #0.3.1
appVersion: 0.8.1 #0.3.1
dependencies:
- name: datahub-gms
version: 0.2.1

32
datahub-kubernetes/datahub/README.md
View File

@ -17,44 +17,44 @@ helm install datahub datahub/
|-----|------|---------|-------------|
| datahub-frontend.enabled | bool | `true` | Enable Datahub Front-end |
| datahub-frontend.image.repository | string | `"linkedin/datahub-frontend-react"` | Image repository for datahub-frontend |
| datahub-frontend.image.tag | string | `"latest"` | Image tag for datahub-frontend |
| datahub-frontend.image.tag | string | `"v0.8.1"` | Image tag for datahub-frontend |
| datahub-gms.enabled | bool | `true` | Enable GMS |
| datahub-gms.image.repository | string | `"linkedin/datahub-gms"` | Image repository for datahub-gms |
| datahub-gms.image.tag | string | `"latest"` | Image tag for datahub-gms |
| datahub-gms.image.tag | string | `"v0.8.1"` | Image tag for datahub-gms |
| datahub-mae-consumer.enabled | bool | `true` | Enable MAE Consumer |
| datahub-mae-consumer.image.repository | string | `"linkedin/datahub-mae-consumer"` | Image repository for datahub-mae-consumer |
| datahub-mae-consumer.image.tag | string | `"latest"` | Image tag for datahub-mae-consumer |
| datahub-mae-consumer.image.tag | string | `"v0.8.1"` | Image tag for datahub-mae-consumer |
| datahub-mce-consumer.enabled | bool | `true` | Enable MCE Consumer |
| datahub-mce-consumer.image.repository | string | `"linkedin/datahub-mce-consumer"` | Image repository for datahub-mce-consumer |
| datahub-mce-consumer.image.tag | string | `"latest"` | Image tag for datahub-mce-consumer |
| datahub-mce-consumer.image.tag | string | `"v0.8.1"` | Image tag for datahub-mce-consumer |
| datahub-ingestion-cron.enabled | bool | `false` | Enable cronjob for periodic ingestion |
| elasticsearchSetupJob.enabled | bool | `true` | Enable setup job for elasicsearch |
| elasticsearchSetupJob.image.repository | string | `"linkedin/datahub-elasticsearch-setup"` | Image repository for elasticsearchSetupJob |
| elasticsearchSetupJob.image.tag | string | `"latest"` | Image repository for elasticsearchSetupJob |
| elasticsearchSetupJob.image.tag | string | `"v0.8.1"` | Image repository for elasticsearchSetupJob |
| kafkaSetupJob.enabled | bool | `true` | Enable setup job for kafka |
| kafkaSetupJob.image.repository | string | `"linkedin/datahub-kafka-setup"` | Image repository for kafkaSetupJob |
| kafkaSetupJob.image.tag | string | `"latest"` | Image repository for kafkaSetupJob |
| kafkaSetupJob.image.tag | string | `"v0.8.1"` | Image repository for kafkaSetupJob |
| mysqlSetupJob.enabled | bool | `false` | Enable setup job for mysql |
| mysqlSetupJob.image.repository | string | `""` | Image repository for mysqlSetupJob |
| mysqlSetupJob.image.tag | string | `""` | Image repository for mysqlSetupJob |
| global.datahub_analytics_enabled | boolean | true | Enable datahub usage analytics |
| global.datahub.appVersion | string | `"1.0"` | App version for annotation |
| global.datahub.gms.port | string | `"8080"` | Port of GMS service |
| global.elasticsearch.host | string | `"elasticsearch"` | Elasticsearch host name (endpoint) |
| global.elasticsearch.host | string | `"elasticsearch-master"` | Elasticsearch host name (endpoint) |
| global.elasticsearch.port | string | `"9200"` | Elasticsearch port |
| global.kafka.bootstrap.server | string | `"broker:9092"` | Kafka bootstrap servers (with port) |
| global.kafka.zookeeper.server | string | `"zookeeper:2181"` | Kafka zookeeper servers (with port) |
| global.kafka.schemaregistry.url | string | `"http://schema-registry:8081"` | URL to kafka schema registry |
| global.neo4j.host | string | `"neo4j:7474"` | Neo4j host address (with port) |
| global.neo4j.uri | string | `"bolt://neo4j"` | Neo4j URI |
| global.kafka.bootstrap.server | string | `"prerequisites-broker:9092"` | Kafka bootstrap servers (with port) |
| global.kafka.zookeeper.server | string | `"prerequisites-zookeeper:2181"` | Kafka zookeeper servers (with port) |
| global.kafka.schemaregistry.url | string | `"http://prerequisites-cp-schema-registry:8081"` | URL to kafka schema registry |
| global.neo4j.host | string | `"prerequisites-neo4j:7474"` | Neo4j host address (with port) |
| global.neo4j.uri | string | `"bolt://prerequisites-neo4j"` | Neo4j URI |
| global.neo4j.username | string | `"neo4j"` | Neo4j user name |
| global.neo4j.password.secretRef | string | `"neo4j-secrets"` | Secret that contains the Neo4j password |
| global.neo4j.password.secretKey | string | `"neo4j-password"` | Secret key that contains the Neo4j password |
| global.sql.datasource.driver | string | `"com.mysql.jdbc.Driver"` | Driver for the SQL database |
| global.sql.datasource.host | string | `"mysql:3306"` | SQL database host (with port) |
| global.sql.datasource.hostForMysqlClient | string | `"mysql"` | SQL database host (without port) |
| global.sql.datasource.url | string | `"jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | URL to access SQL database |
| global.sql.datasource.username | string | `"datahub"` | SQL user name |
| global.sql.datasource.host | string | `"prerequisites-mysql:3306"` | SQL database host (with port) |
| global.sql.datasource.hostForMysqlClient | string | `"prerequisites-mysql"` | SQL database host (without port) |
| global.sql.datasource.url | string | `"jdbc:mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false\u0026useSSL=true"` | URL to access SQL database |
| global.sql.datasource.username | string | `"root"` | SQL user name |
| global.sql.datasource.password.secretRef | string | `"mysql-secrets"` | Secret that contains the MySQL password |
| global.sql.datasource.password.secretKey | string | `"mysql-password"` | Secret key that contains the MySQL password |

2
datahub-kubernetes/datahub/charts/datahub-frontend/README.md
View File

@ -20,7 +20,7 @@ Current chart version is `0.2.0`
| global.datahub.gms.port | string | `"8080"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"linkedin/datahub-frontend-react"` | |
| image.tag | string | `"latest"` | |
| image.tag | string | `"v0.8.1"` | |
| imagePullSecrets | list | `[]` | |
| ingress.annotations | object | `{}` | |
| ingress.enabled | bool | `false` | |

2
datahub-kubernetes/datahub/charts/datahub-frontend/values.yaml
View File

@ -6,7 +6,7 @@ replicaCount: 1
image:
repository: linkedin/datahub-frontend-react
tag: "latest"
tag: "v0.8.1"
pullPolicy: Always
imagePullSecrets: []

2
datahub-kubernetes/datahub/charts/datahub-gms/README.md
View File

@ -38,7 +38,7 @@ Current chart version is `0.2.0`
| global.sql.datasource.password.secretKey | string | `"mysql-password"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"linkedin/datahub-gms"` | |
| image.tag | string | `"latest"` | |
| image.tag | string | `"v0.8.1"` | |
| imagePullSecrets | list | `[]` | |
| ingress.annotations | object | `{}` | |
| ingress.enabled | bool | `false` | |

2
datahub-kubernetes/datahub/charts/datahub-gms/values.yaml
View File

@ -7,7 +7,7 @@ replicaCount: 1
image:
repository: linkedin/datahub-gms
pullPolicy: Always
tag: latest
tag: v0.8.1
imagePullSecrets: []
nameOverride: ""

2
datahub-kubernetes/datahub/charts/datahub-ingestion-cron/README.md
View File

@ -8,7 +8,7 @@ A Helm chart for datahub's metadata-ingestion framework with kerberos authentica
|-----|------|---------|-------------|
| image.pullPolicy | string | `"Always"` | Image pull policy |
| image.repository | string | `"linkedin/datahub-ingestion"` | DataHub Ingestion image repository |
| image.tag | string | `"latest"` | DataHub Ingestion image tag |
| image.tag | string | `"v0.8.1"` | DataHub Ingestion image tag |
| imagePullSecrets | array | `[]` (does not add image pull secrets to deployed pods) | Docker registry secret names as an array |
| labels | string | `{}` | Metadata labels to be added to each crawling cron job |
| crons | type | `{}` | A map of crawling parameters per different technology being crawler, the key in the object will be used as the name for the new cron job |

2
datahub-kubernetes/datahub/charts/datahub-ingestion-cron/templates/cron.yaml
View File

@ -28,10 +28,10 @@ spec:
- name: {{ $jobName }}-crawler
image: "{{ $.Values.image.repository }}:{{ $.Values.image.tag }}"
imagePullPolicy: {{ $.Values.image.pullPolicy }}
{{- if .extraVolumeMounts }}
volumeMounts:
- name: recipe
mountPath: /etc/recipe
{{- if .extraVolumeMounts }}
{{- toYaml .extraVolumeMounts | nindent 14 }}
{{- end }}
command: ["/bin/sh", "-c", {{ default $defaultCommand .command }} ]

2
datahub-kubernetes/datahub/charts/datahub-ingestion-cron/values.yaml
View File

@ -4,7 +4,7 @@
image:
repository: linkedin/datahub-ingestion
tag: latest
tag: v0.8.1
pullPolicy: Always
imagePullSecrets: []

2
datahub-kubernetes/datahub/charts/datahub-mae-consumer/README.md
View File

@ -31,7 +31,7 @@ Current chart version is `0.2.0`
| global.hostAliases[0].ip | string | `"192.168.0.104"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"linkedin/datahub-mae-consumer"` | |
| image.tag | string | `"latest"` | |
| image.tag | string | `"v0.8.1"` | |
| imagePullSecrets | list | `[]` | |
| ingress.annotations | object | `{}` | |
| ingress.enabled | bool | `false` | |

8
datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/deployment.yaml
View File

@ -54,17 +54,21 @@ spec:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 9091
protocol: TCP
livenessProbe:
httpGet:
path: /actuator/health
port: 9091
port: http
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
readinessProbe:
httpGet:
path: /actuator/health
port: 9091
port: http
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}

15
datahub-kubernetes/datahub/charts/datahub-mae-consumer/templates/service.yaml Normal file
View File

@ -0,0 +1,15 @@
apiVersion: v1
kind: Service
metadata:
name: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }}
labels:
{{- include "datahub-mae-consumer.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.global.datahub.mae_consumer.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "datahub-mae-consumer.selectorLabels" . | nindent 4 }}

6
datahub-kubernetes/datahub/charts/datahub-mae-consumer/values.yaml
View File

@ -7,7 +7,7 @@ replicaCount: 1
image:
repository: linkedin/datahub-mae-consumer
pullPolicy: Always
tag: latest
tag: v0.8.1
imagePullSecrets: []
nameOverride: ""
@ -165,6 +165,10 @@ global:
secretRef: "neo4j-secrets"
secretKey: "neo4j-password"
datahub:
mae_consumer:
port: "9091"
hostAliases:
- ip: "192.168.0.104"
hostnames:

2
datahub-kubernetes/datahub/charts/datahub-mce-consumer/README.md
View File

@ -24,7 +24,7 @@ Current chart version is `0.2.0`
| global.hostAliases[0].ip | string | `"192.168.0.104"` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"linkedin/datahub-mce-consumer"` | |
| image.tag | string | `"latest"` | |
| image.tag | string | `"v0.8.1"` | |
| imagePullSecrets | list | `[]` | |
| ingress.annotations | object | `{}` | |
| ingress.enabled | bool | `false` | |

2
datahub-kubernetes/datahub/charts/datahub-mce-consumer/values.yaml
View File

@ -7,7 +7,7 @@ replicaCount: 1
image:
repository: linkedin/datahub-mce-consumer
pullPolicy: Always
tag: latest
tag: v0.8.1
imagePullSecrets: []
nameOverride: ""

28
datahub-kubernetes/datahub/quickstart-values.yaml
View File

@ -4,13 +4,13 @@ datahub-gms:
enabled: true
image:
repository: linkedin/datahub-gms
tag: "latest"
tag: "v0.8.1"
datahub-frontend:
enabled: true
image:
repository: linkedin/datahub-frontend-react
tag: "latest"
tag: "v0.8.1"
# Set up ingress to expose react front-end
ingress:
enabled: false
@ -19,31 +19,37 @@ datahub-mae-consumer:
enabled: true
image:
repository: linkedin/datahub-mae-consumer
tag: "latest"
tag: "v0.8.1"
datahub-mce-consumer:
enabled: true
image:
repository: linkedin/datahub-mce-consumer
tag: "latest"
tag: "v0.8.1"
elasticsearchSetupJob:
enabled: true
image:
repository: linkedin/datahub-elasticsearch-setup
tag: "latest"
tag: "v0.8.1"
kafkaSetupJob:
enabled: true
image:
repository: linkedin/datahub-kafka-setup
tag: "latest"
tag: "v0.8.1"
mysqlSetupJob:
enabled: true
image:
repository: acryldata/datahub-mysql-setup
tag: "latest"
tag: "v0.8.1"
datahubUpgrade:
enabled: true
image:
repository: acryldata/datahub-upgrade
tag: "v0.8.1"
datahub-ingestion-cron:
enabled: false
@ -53,12 +59,6 @@ global:
host: "elasticsearch-master"
port: "9200"
indexPrefix: demo
useSSL: "true"
auth:
username: test
password:
secretRef: test
secretKey: test
kafka:
bootstrap:
@ -91,4 +91,6 @@ global:
datahub:
gms:
port: "8080"
mae_consumer:
port: "9091"
appVersion: "1.0"

109
datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-cleanup-job-template.yml Normal file
View File

@ -0,0 +1,109 @@
{{- if .Values.datahubUpgrade.enabled -}}
# Job template for running datahub cleanup
# Creates a suspended cronJob that you can use to create an adhoc job when ready to run clean up.
# Run the following command to do so
# kubectl create job --from=cronjob/<<release-name>>-datahub-cleanup-job-template datahub-cleanup-job
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: {{ .Release.Name }}-datahub-cleanup-job-template
labels:
app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/version: {{ .Chart.AppVersion }}
helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
spec:
schedule: "* * * * *"
suspend: true
jobTemplate:
spec:
template:
spec:
{{- with .Values.global.hostAliases }}
hostAliases:
{{- toYaml . | nindent 12 }}
{{- end }}
{{- with .Values.datahubUpgrade.serviceAccount }}
serviceAccountName: {{ . }}
{{- end }}
volumes:
{{- with .Values.datahubUpgrade.extraVolumes }}
{{- toYaml . | nindent 12 }}
{{- end }}
restartPolicy: Never
securityContext:
runAsUser: 1000
fsGroup: 1000
containers:
- name: datahub-upgrade-job
image: "{{ .Values.datahubUpgrade.image.repository }}:{{ .Values.datahubUpgrade.image.tag }}"
args: [ "-u", "NoCodeDataMigrationCleanup" ]
env:
- name: DATAHUB_GMS_HOST
value: {{ printf "%s-%s" .Release.Name "datahub-gms" }}
- name: DATAHUB_GMS_PORT
value: "{{ .Values.global.datahub.gms.port }}"
- name: DATAHUB_MAE_CONSUMER_HOST
value: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }}
- name: DATAHUB_MAE_CONSUMER_PORT
value: "{{ .Values.global.datahub.mae_consumer.port }}"
- name: EBEAN_DATASOURCE_USERNAME
value: "{{ .Values.global.sql.datasource.username }}"
- name: EBEAN_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.global.sql.datasource.password.secretRef }}"
key: "{{ .Values.global.sql.datasource.password.secretKey }}"
- name: EBEAN_DATASOURCE_HOST
value: "{{ .Values.global.sql.datasource.host }}"
- name: EBEAN_DATASOURCE_URL
value: "{{ .Values.global.sql.datasource.url }}"
- name: EBEAN_DATASOURCE_DRIVER
value: "{{ .Values.global.sql.datasource.driver }}"
- name: KAFKA_BOOTSTRAP_SERVER
value: "{{ .Values.global.kafka.bootstrap.server }}"
- name: KAFKA_SCHEMAREGISTRY_URL
value: "{{ .Values.global.kafka.schemaregistry.url }}"
- name: ELASTICSEARCH_HOST
value: {{ .Values.global.elasticsearch.host | quote }}
- name: ELASTICSEARCH_PORT
value: {{ .Values.global.elasticsearch.port | quote }}
{{- with .Values.global.elasticsearch.useSSL }}
- name: ELASTICSEARCH_USE_SSL
value: {{ . | quote }}
{{- end }}
{{- with .Values.global.elasticsearch.auth }}
- name: ELASTICSEARCH_USERNAME
value: {{ .username }}
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .password.secretRef }}"
key: "{{ .password.secretKey }}"
{{- end }}
- name: NEO4J_HOST
value: "{{ .Values.global.neo4j.host }}"
- name: NEO4J_URI
value: "{{ .Values.global.neo4j.uri }}"
- name: NEO4J_USERNAME
value: "{{ .Values.global.neo4j.username }}"
- name: NEO4J_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.global.neo4j.password.secretRef }}"
key: "{{ .Values.global.neo4j.password.secretKey }}"
{{- with .Values.datahubUpgrade.extraEnvs }}
{{- toYaml . | nindent 16 }}
{{- end }}
volumeMounts:
{{- with .Values.datahubUpgrade.extraVolumeMounts }}
{{- toYaml . | nindent 16 }}
{{- end }}
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 300m
memory: 256Mi
{{- end -}}

107
datahub-kubernetes/datahub/templates/datahub-upgrade/datahub-upgrade-job.yml Normal file
View File

@ -0,0 +1,107 @@
{{- if .Values.datahubUpgrade.enabled -}}
apiVersion: batch/v1
kind: Job
metadata:
name: {{ .Release.Name }}-datahub-upgrade-job
labels:
app.kubernetes.io/managed-by: {{ .Release.Service | quote }}
app.kubernetes.io/instance: {{ .Release.Name | quote }}
app.kubernetes.io/version: {{ .Chart.AppVersion }}
helm.sh/chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
annotations:
# This is what defines this resource as a hook. Without this line, the
# job is considered part of the release.
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
template:
spec:
{{- with .Values.global.hostAliases }}
hostAliases:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.datahubUpgrade.serviceAccount }}
serviceAccountName: {{ . }}
{{- end }}
volumes:
{{- with .Values.datahubUpgrade.extraVolumes }}
{{- toYaml . | nindent 8}}
{{- end }}
restartPolicy: Never
securityContext:
runAsUser: 1000
fsGroup: 1000
containers:
- name: datahub-upgrade-job
image: "{{ .Values.datahubUpgrade.image.repository }}:{{ .Values.datahubUpgrade.image.tag }}"
args: [ "-u", "NoCodeDataMigration", "-a", "batchSize=1000", "-a", "batchDelayMs=100" ]
env:
- name: DATAHUB_GMS_HOST
value: {{ printf "%s-%s" .Release.Name "datahub-gms" }}
- name: DATAHUB_GMS_PORT
value: "{{ .Values.global.datahub.gms.port }}"
- name: DATAHUB_MAE_CONSUMER_HOST
value: {{ printf "%s-%s" .Release.Name "datahub-mae-consumer" }}
- name: DATAHUB_MAE_CONSUMER_PORT
value: "{{ .Values.global.datahub.mae_consumer.port }}"
- name: EBEAN_DATASOURCE_USERNAME
value: "{{ .Values.global.sql.datasource.username }}"
- name: EBEAN_DATASOURCE_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.global.sql.datasource.password.secretRef }}"
key: "{{ .Values.global.sql.datasource.password.secretKey }}"
- name: EBEAN_DATASOURCE_HOST
value: "{{ .Values.global.sql.datasource.host }}"
- name: EBEAN_DATASOURCE_URL
value: "{{ .Values.global.sql.datasource.url }}"
- name: EBEAN_DATASOURCE_DRIVER
value: "{{ .Values.global.sql.datasource.driver }}"
- name: KAFKA_BOOTSTRAP_SERVER
value: "{{ .Values.global.kafka.bootstrap.server }}"
- name: KAFKA_SCHEMAREGISTRY_URL
value: "{{ .Values.global.kafka.schemaregistry.url }}"
- name: ELASTICSEARCH_HOST
value: {{ .Values.global.elasticsearch.host | quote }}
- name: ELASTICSEARCH_PORT
value: {{ .Values.global.elasticsearch.port | quote }}
{{- with .Values.global.elasticsearch.useSSL }}
- name: ELASTICSEARCH_USE_SSL
value: {{ . | quote }}
{{- end }}
{{- with .Values.global.elasticsearch.auth }}
- name: ELASTICSEARCH_USERNAME
value: {{ .username }}
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .password.secretRef }}"
key: "{{ .password.secretKey }}"
{{- end }}
- name: NEO4J_HOST
value: "{{ .Values.global.neo4j.host }}"
- name: NEO4J_URI
value: "{{ .Values.global.neo4j.uri }}"
- name: NEO4J_USERNAME
value: "{{ .Values.global.neo4j.username }}"
- name: NEO4J_PASSWORD
valueFrom:
secretKeyRef:
name: "{{ .Values.global.neo4j.password.secretRef }}"
key: "{{ .Values.global.neo4j.password.secretKey }}"
{{- with .Values.datahubUpgrade.extraEnvs }}
{{- toYaml . | nindent 12 }}
{{- end }}
volumeMounts:
{{- with .Values.datahubUpgrade.extraVolumeMounts }}
{{- toYaml . | nindent 12 }}
{{- end }}
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 300m
memory: 256Mi
{{- end -}}

61
datahub-kubernetes/datahub/values.yaml
View File

@ -1,94 +1,101 @@
# Default values for datahub.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
# Values to start up datahub after starting up the datahub-prerequisites chart with "prerequisites" release name
# Copy this chart and change configuration as needed.
datahub-gms:
enabled: true
image:
repository: linkedin/datahub-gms
tag: "latest"
tag: "v0.8.1"
datahub-frontend:
enabled: true
image:
repository: linkedin/datahub-frontend-react
tag: "latest"
tag: "v0.8.1"
# Set up ingress to expose react front-end
ingress:
enabled: false
datahub-mae-consumer:
enabled: true
image:
repository: linkedin/datahub-mae-consumer
tag: "latest"
tag: "v0.8.1"
datahub-mce-consumer:
enabled: true
image:
repository: linkedin/datahub-mce-consumer
tag: "latest"
tag: "v0.8.1"
datahub-ingestion-cron:
enabled: false
image:
repository: linkedin/datahub-ingestion
tag: "latest"
tag: "v0.8.1"
elasticsearchSetupJob:
enabled: true
image:
repository: linkedin/datahub-elasticsearch-setup
tag: "latest"
tag: "v0.8.1"
kafkaSetupJob:
enabled: true
image:
repository: linkedin/datahub-kafka-setup
tag: "latest"
tag: "v0.8.1"
mysqlSetupJob:
enabled: true
image:
repository: acryldata/datahub-mysql-setup
tag: "latest"
tag: "v0.8.1"
datahubUpgrade:
enabled: true
image:
repository: acryldata/datahub-upgrade
tag: "v0.8.1"
global:
datahub_analytics_enabled: true
elasticsearch:
host: "elasticsearch"
host: "elasticsearch-master"
port: "9200"
kafka:
bootstrap:
server: "broker:9092"
server: "prerequisites-kafka:9092"
zookeeper:
server: "zookeeper:2181"
server: "prerequisites-zookeeper:2181"
schemaregistry:
url: "http://schema-registry:8081"
url: "http://prerequisites-cp-schema-registry:8081"
neo4j:
host: "neo4j:7474"
uri: "bolt://neo4j"
host: "prerequisites-neo4j-community:7474"
uri: "bolt://prerequisites-neo4j-community"
username: "neo4j"
password:
secretRef: "neo4j-secrets"
secretKey: "neo4j-password"
secretRef: neo4j-secrets
secretKey: neo4j-password
sql:
datasource:
host: "mysql:3306"
hostForMysqlClient: "mysql"
host: "prerequisites-mysql:3306"
hostForMysqlClient: "prerequisites-mysql"
port: "3306"
url: "jdbc:mysql://mysql:3306/datahub?verifyServerCertificate=false&useSSL=true"
url: "jdbc:mysql://prerequisites-mysql:3306/datahub?verifyServerCertificate=false&useSSL=true&useUnicode=yes&characterEncoding=UTF-8"
driver: "com.mysql.jdbc.Driver"
username: "datahub"
username: "root"
password:
secretRef: "mysql-secrets"
secretKey: "mysql-password"
secretRef: mysql-secrets
secretKey: mysql-root-password
datahub:
gms:
port: "8080"
datahub-mae-consumer:
mae_consumer:
port: "9091"
appVersion: "1.0"

32
docs/advanced/no-code-upgrade.md
View File

@ -92,11 +92,14 @@ docker images -a | grep acryldata/datahub-upgrade | awk '{print $3}' | xargs doc
Upgrade to latest helm charts by running the following after pulling latest master.
```(shell)
helm upgrade datahub datahub/ --values datahub/quickstart-values.yaml
helm upgrade datahub datahub/
```
This will upgrade all pods to version 0.8.0, and once all pods are up and ready, datahub-upgrade job will start, running
the above docker image to migrate to the new stores.
In the latest helm charts, we added a datahub-upgrade-job, which runs the above mentioned docker container to migrate to
the new storage layer. Note, the job will fail in the beginning as it waits for GMS and MAE consumer to be deployed with
the NoCode code. It will rerun until it runs successfully.
Once the storage layer has been migrated, subsequent runs of this job will be a noop.
### Step 3 (Optional): Cleaning Up
@ -132,7 +135,28 @@ documentation
#### Helm Deployments
TODO
Assuming the latest helm chart has been deployed in the previous step, datahub-cleanup-job-template cronJob should have
been created. You can check by running the following:
```
kubectl get cronjobs
```
You should see an output like below:
```
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
datahub-datahub-cleanup-job-template * * * * * True 0 <none> 12m
```
Note that the cronJob has been suspended. It is intended to be run in an adhoc fashion when ready to clean up. Make sure
the migration was successful and DataHub is working as expected. Then run the following command to run the clean up job:
```
kubectl create job --from=cronjob/<<release-name>>-datahub-cleanup-job-template datahub-cleanup-job
```
Replace release-name with the name of the helm release. If you followed the kubernetes guide, it should be "datahub".
## Support