# Docker Compose Profiles This directory contains a set of docker compose definitions which are designed to run several configurations for quickstart use-cases as well as development use-cases. These configurations cover a few of the wide variety of infrastructure configurations that DataHub can operate on. Requirements: - Using profiles requires docker compose >= 2.20. - If using the debug/development profiles, you will need to have built the `debug` docker images locally. See the Development Profiles section for more details. ```bash $ cd docker/profiles $ docker compose --profile up ``` Use Control-c (`^c`) to terminate the running system. This will automatically stop all running containers. To remove the containers use the following: ```bash docker compose --profile rm ``` Please refer to docker's documentation for more details. The following sections detail a few of the profiles and their intended use-cases. For a complete list of profiles and their configuration please see the table at the end of each section. ## Quickstart Profiles Quickstart profiles are primarily a way to test drive DataHub features before committing to a production ready deployment. A couple of these profiles are also used in our continuous integration (CI) tests. Note: Quickstart profiles use docker images with the `head` tag. These images up updated when changes are committed to the DataHub github repository. This can be overridden to use a stable release tag by prefixing the commands with `DATAHUB_VERSION=v0.12.1` for example. ### `quickstart` This is the default configuration MySQL and OpenSearch for the storage and GMS running with integrated consumers. ### `quickstart-consumers` This configuration is identical to `quickstart` how it runs standalone consumers instead of consumers integrated with the GMS container. ### `quickstart-postgres` Identical to `quickstart` with Postgres instead of MySQL. ### `quickstart-cassandra` Uses Cassandra as the primary data store along with Neo4j as the graph database. ### `quickstart-storage` Just run the `quickstart` data stores without the DataHub components. This mode is useful for debugging when running the frontend and GMS components outside of docker. ### Quickstart Profiles Table | Profile Name | MySQL | Postgres | Cassandra | Neo4j | Frontend | GMS | Actions | SystemUpdate | MAE | MCE | Kafka | OpenSearch | | -------------------- | ----- | -------- | --------- | ----- | -------- | --- | ------- | ------------ | --- | --- | ----- | ---------- | | quickstart | X | | | | X | X | X | X | | | X | X | | quickstart-frontend | X | | | | X | | | X | | | X | X | | quickstart-backend | X | | | | | X | X | X | | | X | X | | quickstart-postgres | | X | | | X | X | X | X | | | X | X | | quickstart-cassandra | | | X | X | X | X | X | X | | | X | X | | quickstart-consumers | X | | | | X | X | X | X | X | X | X | X | | quickstart-storage | X | | | | | | | | | | X | X | ## Development Profiles - Runs `debug` tagged images - JVM Debug Mode Enabled - Exposes local jars and scripts to the containers - Can run non-default one-off configurations (neo4j, cassandra, elasticsearch) The docker images used are the `debug` images which are created by building locally. These images are created by running the gradle command. ```bash ./gradlew dockerTagDebug ``` For a complete list of profiles see the table at the end of this section. ### `quickstart-backend` Run everything except for the `frontend` component. Useful for running just a local (non-docker) frontend. ### `quickstart-frontend` Runs everything except for the GMS. Useful for running just a local (non-docker) GMS instance. ### Development Profiles Table | Profile Name | MySQL | Postgres | Cassandra | Neo4j | Frontend | GMS | Actions | SystemUpdate | MAE | MCE | Kafka | OpenSearch | Elasticsearch | Localstack (AWS) | | ------------------- | ----- | -------- | --------- | ----- | -------- | --- | ------- | ------------ | --- | --- | ----- | ---------- | ------------- | ---------------- | | debug | X | | | | X | X | X | X | | | X | X | | | | debug-frontend | X | | | | X | | | X | | | X | X | | | | debug-backend | X | | | | | X | X | X | | | X | X | | | | debug-postgres | | X | | | X | X | X | X | | | X | X | | | | debug-cassandra | | | X | | X | X | X | X | | | X | X | | | | debug-consumers | X | | | | X | X | X | X | X | X | X | X | | | | debug-neo4j | X | | | X | X | X | X | X | | | X | X | | | | debug-elasticsearch | X | | | | X | X | X | X | | | X | | X | | | debug-backend-aws | X | | | | | X | X | X | | | X | X | | X | ## Advanced Setups ### Version Mixing In some cases, it might be useful to debug upgrade scenarios where there are intentional version miss-matches. It is possible to override individual component versions. Note: This only works for `non-debug` profiles because of the file mounts when in `debug` which would run older containers but still pickup the latest application jars. In this example we are interested in upgrading two components (the `mae-consumer` and the `mce-consumer`) to a fresh build `v0.15.1-SNAPSHOT` while maintaining older components on `v0.14.1` (especially the `system-update` container). This configuration reproduces the situation where the consumers were upgraded prior to running the latest version of `system-update`. In this scenario we expect the consumers to block their startup waiting for the successful completion of a newer `system-update`. `DATAHUB_VERSION` - specifies the default component version of `v0.14.1` `DATAHUB_MAE_VERSION` - specifies an override of just the `mae-consumer` to version `v0.15.1-SNAPSHOT`[1] `DATAHUB_MCE_VERSION` - specifies an override of just the `mce-consumer` to version `v0.15.1-SNAPSHOT`[1] ```shell DATAHUB_MAE_VERSION="v0.15.1-SNAPSHOT" DATAHUB_MCE_VERSION="v0.15.1-SNAPSHOT" DATAHUB_VERSION="v0.14.1" ./gradlew quickstart ``` [1] Image versions were `v0.15.1-SNAPSHOT` built locally prior to running the command.