yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-02 11:49:23 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(development): per project quickstart nuke (#14473)

Browse Source 
Co-authored-by: Esteban Gutierrez <esteban.gutierrez@acryl.io>
This commit is contained in:
david-leifker 2025-08-15 13:14:27 -05:00 committed by GitHub
parent 352df9ee25
commit 6c6abca29e
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 219 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

149
docker/README.md
View File

@ -87,3 +87,152 @@ something unique.
### Community Built Images
As the open source project grows, community members would like to contribute additions to the docker images. Not all contributions to the images can be accepted because those changes are not useful for all community members, it will increase build times, add dependencies and possible security vulns. In those cases this section can be used to point to `Dockerfiles` hosted by the community which build on top of the images published by the DataHub core team along with any container registry links where the result of those images are maintained.
# DataHub Docker Nuke Tasks
This document describes the nuke task system for cleaning up DataHub Docker containers and volumes.
## Overview
The nuke tasks provide a way to completely remove DataHub containers and volumes for different project namespaces. This is useful for:
- Cleaning up test environments
- Resetting development setups
- Isolating different project instances
- Troubleshooting container issues
## Available Tasks
### All Configurations Have Nuke Tasks
Every quickstart configuration automatically gets a nuke task for targeted cleanup:
- **`quickstartNuke`** - Removes containers and volumes for the default project namespace (`datahub`)
- **`quickstartDebugNuke`** - Removes containers and volumes for the debug configuration (`datahub`)
- **`quickstartCypressNuke`** - Removes containers and volumes for the cypress configuration (`dh-cypress`)
- **`quickstartDebugMinNuke`** - Removes containers and volumes for the debug-min configuration (`datahub`)
- **`quickstartDebugConsumersNuke`** - Removes containers and volumes for the debug-consumers configuration (`datahub`)
- **`quickstartPgNuke`** - Removes containers and volumes for the postgres configuration (`datahub`)
- **`quickstartPgDebugNuke`** - Removes containers and volumes for the debug-postgres configuration (`datahub`)
- **`quickstartSlimNuke`** - Removes containers and volumes for the backend configuration (`datahub`)
- **`quickstartSparkNuke`** - Removes containers and volumes for the spark configuration (`datahub`)
- **`quickstartStorageNuke`** - Removes containers and volumes for the storage configuration (`datahub`)
- **`quickstartBackendDebugNuke`** - Removes containers and volumes for the backend-debug configuration (`datahub`)
### Project Namespace Behavior
- **Default project namespace (`datahub`)**: Most configurations use this, so their nuke tasks will clean up containers in the same namespace
- **Custom project namespace (`dh-cypress`)**: The cypress configuration uses its own namespace for isolation
## Usage
### Basic Usage
```bash
# Remove containers and volumes for specific configurations
./gradlew quickstartDebugNuke # For debug configuration
./gradlew quickstartCypressNuke # For cypress configuration
./gradlew quickstartDebugMinNuke # For debug-min configuration
./gradlew quickstartPgNuke # For postgres configuration
# For general cleanup of all containers
./gradlew quickstartDown
```
### When to Use Each Task
- **Use specific nuke tasks when:**
- You want to clean up a specific configuration environment
- You need targeted cleanup without affecting other configurations
- You're working with a particular development setup (debug, postgres, cypress, etc.)
- **Use `quickstartDown` when:**
- You want to stop all running containers
- You need a general cleanup option
- You want to ensure all environments are stopped
## How It Works
1. **Volume Management**: Each nuke task sets `removeVolumes = true` for relevant configurations
2. **Container Cleanup**: Tasks are finalized by appropriate `ComposeDownForced` operations
3. **Project Isolation**: Each task operates within its own Docker Compose project namespace
4. **Configuration Respect**: Tasks respect the `projectName` settings in `quickstart_configs`
## Configuration
The nuke tasks are automatically generated based on the `quickstart_configs` in `docker/build.gradle`. To add a new nuke task:
1. Add a configuration to `quickstart_configs`:
```gradle
'quickstartCustom': [
profile: 'debug',
modules: [...],
// Optional: custom project name for isolation
additionalConfig: [
projectName: 'dh-custom'
]
]
```
2. The task `quickstartCustomNuke` will be automatically created
## Troubleshooting
### Task Not Found
- Ensure the configuration exists in `quickstart_configs`
- Check that the task name follows the pattern `{configName}Nuke`
### Containers Not Removed
- Verify the project namespace is correct
- Check that the configuration has the right `projectName` setting
- Ensure the task is targeting the correct `ComposeDownForced` operations
### Volume Persistence
- Check if `preserveVolumes` is set to `true` in the configuration
- Verify the `removeVolumes` setting is properly applied
## Related Commands
- **Start services**: `./gradlew quickstartDebug`, `./gradlew quickstartCypress`
- **Stop services**: `./gradlew quickstartDown`
- **Reload services**: `./gradlew debugReload`, `./gradlew cypressReload`
## Examples
### Complete Development Environment Reset
```bash
# Clean up specific debug environment
./gradlew quickstartDebugNuke
# Start fresh
./gradlew quickstartDebug
```
### Cypress Environment Isolation
```bash
# Clean up cypress environment
./gradlew quickstartCypressNuke
# Start fresh cypress environment
./gradlew quickstartCypress
```
### Mixed Environment Management
```bash
# Clean up only cypress (leaving main environment intact)
./gradlew quickstartCypressNuke
# Clean up only debug environment (leaving cypress intact)
./gradlew quickstartDebugNuke
# Clean up only postgres environment (leaving others intact)
./gradlew quickstartPgNuke
```

27
docker/build.gradle
View File

@ -340,14 +340,21 @@ tasks.register('minDockerCompose2.20', Exec) {
args '-c', 'echo -e "$(docker compose version --short)\n2.20"|sort --version-sort --check=quiet --reverse'
}
tasks.register('quickstartNuke') {
group = 'quickstart'
doFirst {
quickstart_configs.each { taskName, config ->
// Create nuke tasks for all configurations
quickstart_configs.each { taskName, config ->
def actualProjectName = config.additionalConfig?.projectName ?: project_name
tasks.register("${taskName}Nuke") {
group = 'quickstart'
description = "Nuke containers and volumes for configuration: ${taskName} (project: ${actualProjectName})"
doFirst {
// Set removeVolumes for this specific configuration
dockerCompose."${taskName}".removeVolumes = !config.preserveVolumes
}
finalizedBy(tasks.matching { task ->
task.name == "${taskName}ComposeDownForced"
})
}
finalizedBy(tasks.withType(ComposeDownForced))
}
tasks.register('quickstartDown') {
@ -385,8 +392,11 @@ quickstart_configs.each { taskName, config ->
if (containersToRestart) {
def composeFiles = dockerCompose."${taskName}".useComposeFiles.get()
def composeFileArgs = composeFiles.collectMany { ['-f', it] }
// Use the actual project name from config, fallback to default
def actualProjectName = config.additionalConfig?.projectName ?: project_name
def cmd = ["docker compose -p ${project_name} --profile ${config.profile}"] + composeFileArgs + ['restart'] + containersToRestart
def cmd = ["docker compose -p ${actualProjectName} --profile ${config.profile}"] + composeFileArgs + ['restart'] + containersToRestart
println(cmd.join(" "))
commandLine 'bash', '-c', cmd.join(" ")
} else {
@ -409,8 +419,11 @@ quickstart_configs.each { taskName, config ->
}
def composeFiles = dockerCompose."${taskName}".useComposeFiles.get()
def composeFileArgs = composeFiles.collectMany { ['-f', it] }
// Use the actual project name from config, fallback to default
def actualProjectName = config.additionalConfig?.projectName ?: project_name
def cmd = ["docker compose -p ${project_name} --profile ${config.profile}"] + composeFileArgs + ['up', '-d', '--no-deps'] + containersToRestart
def cmd = ["docker compose -p ${actualProjectName} --profile ${config.profile}"] + composeFileArgs + ['up', '-d', '--no-deps'] + containersToRestart
println(cmd.join(" "))
commandLine 'bash', '-c', cmd.join(" ")
}

11
docker/profiles/README.md
View File

@ -25,7 +25,16 @@ Alternatively, you can use the gradle tasks defined in `docker/build.gradle`:
Use Control-c (`^c`) to terminate the running system. This will automatically stop all running containers.
To remove the containers use the following:
To remove the containers and volumes, you can use the gradle nuke tasks:
```bash
# Remove containers and volumes for specific projects
./gradlew quickstartNuke # For default project (datahub)
./gradlew quickstartDebugNuke # For debug project (datahub)
./gradlew quickstartCypressNuke # For cypress project (dh-cypress)
```
Alternatively, you can use docker compose directly:
```bash
docker compose --profile <profile name> rm

24
docs/developers.md
View File

@ -116,30 +116,42 @@ The frontend will be available at `http://localhost:3000` and will automatically
### Refreshing components of quickStart
To refresh any of the running system started by `./gradlew quickStartDebug`, run
To refresh any of the running system started by `./gradlew quickstartDebug`, run
```shell
./gradlew debugReload
```
This will build any changed components and restart those containers that had changes.
There are a few other quickStart\* variants, like quickStartDebugMin, quickStartDebugConsumers
There are a few other quickstart\* variants, like quickstartDebugMin, quickstartDebugConsumers
For each of those variants, there is a corresponding reloadTask.
For `./gradlew quickStartDebugConsumers`, the reload command is `./gradlew debugConsumersReload`
For `./gradlew quickStartDebugMin`, the reload command is `./gradlew debugMinReload`
For `./gradlew quickstartDebugConsumers`, the reload command is `./gradlew debugConsumersReload`
For `./gradlew quickstartDebugMin`, the reload command is `./gradlew debugMinReload`
A full restart using `./gradlew quickStartDebug` is recommended if there are significant changes and the setup/system update containers need to be run again.
A full restart using `./gradlew quickstartDebug` is recommended if there are significant changes and the setup/system update containers need to be run again.
For incremental changes, the `debugReload*` variants can be used.
### Cleaning up containers and volumes
To completely remove containers and volumes for a specific project, you can use the nuke tasks:
```shell
# Remove containers and volumes for specific projects
./gradlew quickstartDebugNuke # For debug project
./gradlew quickstartCypressNuke # For cypress project (dh-cypress)
```
> **Note**: These are Gradle nuke tasks. For CLI-based cleanup, see `datahub docker nuke` in the [quickstart guide](quickstart.md).
### Using .env to configure settings of services started by quickstart
To start datahub with a customized set of environment variables, .env files can be created in the docker/profiles folder.
For example, an env file `my-settings.env` can be created in docker/profiles folder and loaded using
```shell
DATAHUB_LOCAL_COMMON_ENV=my-settings.env ./gradlew quickStartDebug
DATAHUB_LOCAL_COMMON_ENV=my-settings.env ./gradlew quickstartDebug
```
To refresh the containers due to code changes, `debugReload` task can be used.

31
docs/docker/development.md
View File

@ -40,8 +40,8 @@ acryldata/datahub-elasticsearch-setup debug 4d935be7c62c
At this point it is possible to view the DataHub UI at `http://localhost:9002` as you normally would with quickstart.
Like `quickStartDebug`, there are a few other tasks that bring up a different set of containers, for example
`quickStartDebugConsumers` will also bring up mce-consumer and mae-consumer.
Like `quickstartDebug`, there are a few other tasks that bring up a different set of containers, for example
`quickstartDebugConsumers` will also bring up mce-consumer and mae-consumer.
## Reloading
@ -56,13 +56,13 @@ The restart can be performed using following gradle task.
This single task will build the artifacts that were modified and restart only those containers that were affected by the rebuilt artifacts.
For each of the `quickStartDebug` variants, there is a corresponding `debugReload` task.
For `quickStartDebugConsumers`, the reload task is `debugConsumersReload`
For each of the `quickstartDebug` variants, there is a corresponding `debugReload` task.
For `quickstartDebugConsumers`, the reload task is `debugConsumersReload`
`debugReload` is generally much faster than re-running `quickStartDebug` and is recommended after an initial bringup of all services via `quickStartDebug` followed
`debugReload` is generally much faster than re-running `quickstartDebug` and is recommended after an initial bringup of all services via `quickstartDebug` followed
by loading the incremental changes using `debugReload`.
If there are significant changes to the code, for example due to pulling the latest code, it is recommended to start with a `quickStartDebug` and then iterate using `debugReload`
If there are significant changes to the code, for example due to pulling the latest code, it is recommended to start with a `quickstartDebug` and then iterate using `debugReload`
# Setting environment variables via env files
@ -70,15 +70,15 @@ You can define different sets of environment variables for all the containers in
To use the env file, run
```shell
DATAHUB_LOCAL_COMMON_ENV=my-settings.env ./gradlew quickStartDebug
DATAHUB_LOCAL_COMMON_ENV=my-settings.env ./gradlew quickstartDebug
```
The `debugReload` process continues to work, but the restarted containers will use the same settings that were present at the time of running `./gradlew quickStartDebug`.
The `debugReload` process continues to work, but the restarted containers will use the same settings that were present at the time of running `./gradlew quickstartDebug`.
If you need to reload the containers with a different env file or changes made to the env file, a task `debugReloadEnv` builds the artifacts that have code changes
and recreates all the containers that refer to these the env file via the DATAHUB_LOCAL_COMMON_ENV environment variable.
`debugReloadEnv` also has variants for all the `quickStartDebug` variants. For example, `quickStartDebugConsumers` has `debugConsumersReloadEnv`
`debugReloadEnv` also has variants for all the `quickstartDebug` variants. For example, `quickstartDebugConsumers` has `debugConsumersReloadEnv`
## Start/Stop
@ -96,6 +96,19 @@ Resume containers for further debugging.
docker compose -p datahub start
```
## Cleanup
To completely remove containers and volumes for a specific project, you can use the nuke tasks:
```shell
# Remove containers and volumes for specific projects
./gradlew quickstartNuke # For default project (datahub)
./gradlew quickstartDebugNuke # For debug project (datahub)
./gradlew quickstartCypressNuke # For cypress project (dh-cypress)
```
> **Note**: These are Gradle nuke tasks. For CLI-based cleanup, see `datahub docker nuke` in the [quickstart guide](../quickstart.md).
## Debugging
The default debugging process uses your local code and enables debugging by default for both GMS and the frontend. Attach