datahub/docs/developers.md

---
title: "Local Development"
---

# DataHub Developer's Guide

## Requirements

- [Java 17 JDK](https://openjdk.org/projects/jdk/17/)
- [Python 3.10](https://www.python.org/downloads/release/python-3100/)
- [Docker](https://www.docker.com/)
- [Docker Compose >=2.20](https://docs.docker.com/compose/)
- Docker engine with at least 8GB of memory to run tests.

On macOS, these can be installed using [Homebrew](https://brew.sh/).

```shell
# Install Java
brew install openjdk@17

# Install Python
brew install python@3.10  # you may need to add this to your PATH
# alternatively, you can use pyenv to manage your python versions

# Install docker and docker compose
brew install --cask docker
```

## Building the Project

Fork and clone the repository if haven't done so already

```shell
git clone https://github.com/{username}/datahub.git
```

Change into the repository's root directory

```shell
cd datahub
```

Use [gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) to build the project

```shell
./gradlew build
```

Note that the above will also run run tests and a number of validations which makes the process considerably slower.

We suggest partially compiling DataHub according to your needs:

- Build Datahub's backend GMS (Generalized metadata service):

  ```
  ./gradlew :metadata-service:war:build
  ```

- Build Datahub's frontend:

  ```
  ./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint
  ```

- Build DataHub's command line tool:

  ```
  ./gradlew :metadata-ingestion:installDev
  ```

- Build DataHub's documentation:

  ```
  ./gradlew :docs-website:yarnLintFix :docs-website:build -x :metadata-ingestion:runPreFlightScript
  # To preview the documentation
  ./gradlew :docs-website:serve
  ```

## Deploying Local Versions

Run just once to have the local `datahub` cli tool installed in your $PATH

```shell
cd smoke-test/
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools
pip install -r requirements.txt
cd ../
```

Once you have compiled & packaged the project or appropriate module you can deploy the entire system via docker-compose by running:

```shell
./gradlew quickstart
```

Replace whatever container you want in the existing deployment.
I.e, replacing datahub's backend (GMS):

```shell
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-gms)
```

Running the local version of the frontend

```shell
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-frontend-react)
```

## IDE Support

The recommended IDE for DataHub development is [IntelliJ IDEA](https://www.jetbrains.com/idea/).
You can run the following command to generate or update the IntelliJ project file.

```shell
./gradlew idea
```

Open `datahub.ipr` in IntelliJ to start developing!

For consistency please import and auto format the code using [LinkedIn IntelliJ Java style](../gradle/idea/LinkedIn%20Style.xml).

## Windows Compatibility

For optimal performance and compatibility, we strongly recommend building on a Mac or Linux system.
Please note that we do not actively support Windows in a non-virtualized environment.

If you must use Windows, one workaround is to build within a virtualized environment, such as a VM(Virtual Machine) or [WSL(Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl).
This approach can help ensure that your build environment remains isolated and stable, and that your code is compiled correctly.

## Common Build Issues

#### Getting `Unsupported class file major version 57`

You're probably using a Java version that's too new for gradle. Run the following command to check your Java version

```shell
java --version
```

While it may be possible to build and run DataHub using newer versions of Java, we currently only support [Java 17](https://openjdk.org/projects/jdk/17/) (aka Java 17).

#### Getting `cannot find symbol` error for `javax.annotation.Generated`

Similar to the previous issue, please use Java 17 to build the project.
You can install multiple version of Java on a single machine and switch between them using the `JAVA_HOME` environment variable. See [this document](https://docs.oracle.com/cd/E21454_01/html/821-2531/inst_jdk_javahome_t.html) for more details.

#### `:metadata-models:generateDataTemplate` task fails with `java.nio.file.InvalidPathException: Illegal char <:> at index XX` or `Caused by: java.lang.IllegalArgumentException: 'other' has different root` error

This is a [known issue](https://github.com/linkedin/rest.li/issues/287) when building the project on Windows due a bug in the Pegasus plugin. Please refer to [Windows Compatibility](/docs/developers.md#windows-compatibility).

#### Various errors related to `generateDataTemplate` or other `generate` tasks

As we generate quite a few files from the models, it is possible that old generated files may conflict with new model changes. When this happens, a simple `./gradlew clean` should reosolve the issue.

#### `Execution failed for task ':metadata-service:restli-servlet-impl:checkRestModel'`

This generally means that an [incompatible change](https://linkedin.github.io/rest.li/modeling/compatibility_check) was introduced to the rest.li API in GMS. You'll need to rebuild the snapshots/IDL by running the following command once

```shell
./gradlew :metadata-service:restli-servlet-impl:build -Prest.model.compatibility=ignore
```

#### `java.io.IOException: No space left on device`

This means you're running out of space on your disk to build. Please free up some space or try a different disk.

#### `Build failed` for task `./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint`

This could mean that you need to update your [Yarn](https://yarnpkg.com/getting-started/install) version