docs: add setup instructions for mac dependencies (#9346)

Co-authored-by: Hyejin Yoon <0327jane@gmail.com>
This commit is contained in:
Harshal Sheth 2023-12-01 14:23:11 -05:00 committed by GitHub
parent 36c7813f89
commit 7b0a8f422b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -4,33 +4,53 @@ title: "Local Development"
# DataHub Developer's Guide # DataHub Developer's Guide
## Pre-requirements ## Requirements
- [Java 11 SDK](https://openjdk.org/projects/jdk/11/)
- [Python 3.10] (https://www.python.org/downloads/release/python-3100/)
- [Docker](https://www.docker.com/)
- [Docker Compose](https://docs.docker.com/compose/)
- Docker engine with at least 8GB of memory to run tests.
:::note - Both [Java 11 JDK](https://openjdk.org/projects/jdk/11/) and [Java 8 JDK](https://openjdk.java.net/projects/jdk8/)
- [Python 3.10](https://www.python.org/downloads/release/python-3100/)
- [Docker](https://www.docker.com/)
- [Docker Compose](https://docs.docker.com/compose/)
- Docker engine with at least 8GB of memory to run tests.
Do not try to use a JDK newer than JDK 11. The build process does not work with newer JDKs currently. :::caution
::: Do not try to use a JDK newer than JDK 11. The build process does not currently work with newer JDKs versions.
:::
On macOS, these can be installed using [Homebrew](https://brew.sh/).
```shell
# Install Java 8 and 11
brew tap homebrew/cask-versions
brew install java11
brew install --cask zulu8
# Install Python
brew install python@3.10 # you may need to add this to your PATH
# alternatively, you can use pyenv to manage your python versions
# Install docker and docker compose
brew install --cask docker
```
## Building the Project ## Building the Project
Fork and clone the repository if haven't done so already Fork and clone the repository if haven't done so already
```
```shell
git clone https://github.com/{username}/datahub.git git clone https://github.com/{username}/datahub.git
``` ```
Change into the repository's root directory Change into the repository's root directory
```
```shell
cd datahub cd datahub
``` ```
Use [gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) to build the project Use [gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) to build the project
```
```shell
./gradlew build ./gradlew build
``` ```
@ -38,29 +58,37 @@ Note that the above will also run run tests and a number of validations which ma
We suggest partially compiling DataHub according to your needs: We suggest partially compiling DataHub according to your needs:
- Build Datahub's backend GMS (Generalized metadata service): - Build Datahub's backend GMS (Generalized metadata service):
```
./gradlew :metadata-service:war:build
```
- Build Datahub's frontend:
```
./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint
```
- Build DataHub's command line tool:
```
./gradlew :metadata-ingestion:installDev
```
- Build DataHub's documentation:
```
./gradlew :docs-website:yarnLintFix :docs-website:build -x :metadata-ingestion:runPreFlightScript
# To preview the documentation
./gradlew :docs-website:serve
```
## Deploying local versions ```
./gradlew :metadata-service:war:build
```
- Build Datahub's frontend:
```
./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint
```
- Build DataHub's command line tool:
```
./gradlew :metadata-ingestion:installDev
```
- Build DataHub's documentation:
```
./gradlew :docs-website:yarnLintFix :docs-website:build -x :metadata-ingestion:runPreFlightScript
# To preview the documentation
./gradlew :docs-website:serve
```
## Deploying Local Versions
Run just once to have the local `datahub` cli tool installed in your $PATH Run just once to have the local `datahub` cli tool installed in your $PATH
```
```shell
cd smoke-test/ cd smoke-test/
python3 -m venv venv python3 -m venv venv
source venv/bin/activate source venv/bin/activate
@ -70,31 +98,37 @@ cd ../
``` ```
Once you have compiled & packaged the project or appropriate module you can deploy the entire system via docker-compose by running: Once you have compiled & packaged the project or appropriate module you can deploy the entire system via docker-compose by running:
```
```shell
./gradlew quickstart ./gradlew quickstart
``` ```
Replace whatever container you want in the existing deployment. Replace whatever container you want in the existing deployment.
I.e, replacing datahub's backend (GMS): I.e, replacing datahub's backend (GMS):
```
```shell
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-gms) (cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-gms)
``` ```
Running the local version of the frontend Running the local version of the frontend
```
```shell
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-frontend-react) (cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-frontend-react)
``` ```
## IDE Support ## IDE Support
The recommended IDE for DataHub development is [IntelliJ IDEA](https://www.jetbrains.com/idea/). The recommended IDE for DataHub development is [IntelliJ IDEA](https://www.jetbrains.com/idea/).
You can run the following command to generate or update the IntelliJ project file You can run the following command to generate or update the IntelliJ project file.
```
```shell
./gradlew idea ./gradlew idea
``` ```
Open `datahub.ipr` in IntelliJ to start developing! Open `datahub.ipr` in IntelliJ to start developing!
For consistency please import and auto format the code using [LinkedIn IntelliJ Java style](../gradle/idea/LinkedIn%20Style.xml). For consistency please import and auto format the code using [LinkedIn IntelliJ Java style](../gradle/idea/LinkedIn%20Style.xml).
## Windows Compatibility ## Windows Compatibility
For optimal performance and compatibility, we strongly recommend building on a Mac or Linux system. For optimal performance and compatibility, we strongly recommend building on a Mac or Linux system.
@ -105,37 +139,41 @@ This approach can help ensure that your build environment remains isolated and s
## Common Build Issues ## Common Build Issues
### Getting `Unsupported class file major version 57` #### Getting `Unsupported class file major version 57`
You're probably using a Java version that's too new for gradle. Run the following command to check your Java version You're probably using a Java version that's too new for gradle. Run the following command to check your Java version
```
```shell
java --version java --version
``` ```
While it may be possible to build and run DataHub using newer versions of Java, we currently only support [Java 11](https://openjdk.org/projects/jdk/11/) (aka Java 11). While it may be possible to build and run DataHub using newer versions of Java, we currently only support [Java 11](https://openjdk.org/projects/jdk/11/) (aka Java 11).
### Getting `cannot find symbol` error for `javax.annotation.Generated` #### Getting `cannot find symbol` error for `javax.annotation.Generated`
Similar to the previous issue, please use Java 1.8 to build the project. Similar to the previous issue, please use Java 1.8 to build the project.
You can install multiple version of Java on a single machine and switch between them using the `JAVA_HOME` environment variable. See [this document](https://docs.oracle.com/cd/E21454_01/html/821-2531/inst_jdk_javahome_t.html) for more details. You can install multiple version of Java on a single machine and switch between them using the `JAVA_HOME` environment variable. See [this document](https://docs.oracle.com/cd/E21454_01/html/821-2531/inst_jdk_javahome_t.html) for more details.
### `:metadata-models:generateDataTemplate` task fails with `java.nio.file.InvalidPathException: Illegal char <:> at index XX` or `Caused by: java.lang.IllegalArgumentException: 'other' has different root` error #### `:metadata-models:generateDataTemplate` task fails with `java.nio.file.InvalidPathException: Illegal char <:> at index XX` or `Caused by: java.lang.IllegalArgumentException: 'other' has different root` error
This is a [known issue](https://github.com/linkedin/rest.li/issues/287) when building the project on Windows due a bug in the Pegasus plugin. Please refer to [Windows Compatibility](/docs/developers.md#windows-compatibility). This is a [known issue](https://github.com/linkedin/rest.li/issues/287) when building the project on Windows due a bug in the Pegasus plugin. Please refer to [Windows Compatibility](/docs/developers.md#windows-compatibility).
### Various errors related to `generateDataTemplate` or other `generate` tasks #### Various errors related to `generateDataTemplate` or other `generate` tasks
As we generate quite a few files from the models, it is possible that old generated files may conflict with new model changes. When this happens, a simple `./gradlew clean` should reosolve the issue. As we generate quite a few files from the models, it is possible that old generated files may conflict with new model changes. When this happens, a simple `./gradlew clean` should reosolve the issue.
### `Execution failed for task ':metadata-service:restli-servlet-impl:checkRestModel'` #### `Execution failed for task ':metadata-service:restli-servlet-impl:checkRestModel'`
This generally means that an [incompatible change](https://linkedin.github.io/rest.li/modeling/compatibility_check) was introduced to the rest.li API in GMS. You'll need to rebuild the snapshots/IDL by running the following command once This generally means that an [incompatible change](https://linkedin.github.io/rest.li/modeling/compatibility_check) was introduced to the rest.li API in GMS. You'll need to rebuild the snapshots/IDL by running the following command once
```
```shell
./gradlew :metadata-service:restli-servlet-impl:build -Prest.model.compatibility=ignore ./gradlew :metadata-service:restli-servlet-impl:build -Prest.model.compatibility=ignore
``` ```
### `java.io.IOException: No space left on device` #### `java.io.IOException: No space left on device`
This means you're running out of space on your disk to build. Please free up some space or try a different disk. This means you're running out of space on your disk to build. Please free up some space or try a different disk.
### `Build failed` for task `./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint` #### `Build failed` for task `./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint`
This could mean that you need to update your [Yarn](https://yarnpkg.com/getting-started/install) version This could mean that you need to update your [Yarn](https://yarnpkg.com/getting-started/install) version