mirror of
https://github.com/datahub-project/datahub.git
synced 2025-11-03 04:10:43 +00:00
docs: add setup instructions for mac dependencies (#9346)
Co-authored-by: Hyejin Yoon <0327jane@gmail.com>
This commit is contained in:
Harshal Sheth
GitHub
parent
36c7813f89
commit
7b0a8f422b
@ -4,33 +4,53 @@ title: "Local Development"
|
||||
|
||||
# DataHub Developer's Guide
|
||||
|
||||
## Pre-requirements
|
||||
- [Java 11 SDK](https://openjdk.org/projects/jdk/11/)
|
||||
- [Python 3.10] (https://www.python.org/downloads/release/python-3100/)
|
||||
- [Docker](https://www.docker.com/)
|
||||
- [Docker Compose](https://docs.docker.com/compose/)
|
||||
- Docker engine with at least 8GB of memory to run tests.
|
||||
## Requirements
|
||||
|
||||
:::note
|
||||
- Both [Java 11 JDK](https://openjdk.org/projects/jdk/11/) and [Java 8 JDK](https://openjdk.java.net/projects/jdk8/)
|
||||
- [Python 3.10](https://www.python.org/downloads/release/python-3100/)
|
||||
- [Docker](https://www.docker.com/)
|
||||
- [Docker Compose](https://docs.docker.com/compose/)
|
||||
- Docker engine with at least 8GB of memory to run tests.
|
||||
|
||||
Do not try to use a JDK newer than JDK 11. The build process does not work with newer JDKs currently.
|
||||
:::caution
|
||||
|
||||
:::
|
||||
Do not try to use a JDK newer than JDK 11. The build process does not currently work with newer JDKs versions.
|
||||
|
||||
:::
|
||||
|
||||
On macOS, these can be installed using [Homebrew](https://brew.sh/).
|
||||
|
||||
```shell
|
||||
# Install Java 8 and 11
|
||||
brew tap homebrew/cask-versions
|
||||
brew install java11
|
||||
brew install --cask zulu8
|
||||
|
||||
# Install Python
|
||||
brew install python@3.10 # you may need to add this to your PATH
|
||||
# alternatively, you can use pyenv to manage your python versions
|
||||
|
||||
# Install docker and docker compose
|
||||
brew install --cask docker
|
||||
```
|
||||
|
||||
## Building the Project
|
||||
|
||||
Fork and clone the repository if haven't done so already
|
||||
```
|
||||
|
||||
```shell
|
||||
git clone https://github.com/{username}/datahub.git
|
||||
```
|
||||
|
||||
Change into the repository's root directory
|
||||
```
|
||||
|
||||
```shell
|
||||
cd datahub
|
||||
```
|
||||
|
||||
Use [gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) to build the project
|
||||
```
|
||||
|
||||
```shell
|
||||
./gradlew build
|
||||
```
|
||||
|
||||
@ -38,29 +58,37 @@ Note that the above will also run run tests and a number of validations which ma
|
||||
|
||||
We suggest partially compiling DataHub according to your needs:
|
||||
|
||||
- Build Datahub's backend GMS (Generalized metadata service):
|
||||
```
|
||||
./gradlew :metadata-service:war:build
|
||||
```
|
||||
- Build Datahub's frontend:
|
||||
```
|
||||
./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint
|
||||
```
|
||||
- Build DataHub's command line tool:
|
||||
```
|
||||
./gradlew :metadata-ingestion:installDev
|
||||
```
|
||||
- Build DataHub's documentation:
|
||||
```
|
||||
./gradlew :docs-website:yarnLintFix :docs-website:build -x :metadata-ingestion:runPreFlightScript
|
||||
# To preview the documentation
|
||||
./gradlew :docs-website:serve
|
||||
```
|
||||
- Build Datahub's backend GMS (Generalized metadata service):
|
||||
|
||||
## Deploying local versions
|
||||
```
|
||||
./gradlew :metadata-service:war:build
|
||||
```
|
||||
|
||||
- Build Datahub's frontend:
|
||||
|
||||
```
|
||||
./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint
|
||||
```
|
||||
|
||||
- Build DataHub's command line tool:
|
||||
|
||||
```
|
||||
./gradlew :metadata-ingestion:installDev
|
||||
```
|
||||
|
||||
- Build DataHub's documentation:
|
||||
|
||||
```
|
||||
./gradlew :docs-website:yarnLintFix :docs-website:build -x :metadata-ingestion:runPreFlightScript
|
||||
# To preview the documentation
|
||||
./gradlew :docs-website:serve
|
||||
```
|
||||
|
||||
## Deploying Local Versions
|
||||
|
||||
Run just once to have the local `datahub` cli tool installed in your $PATH
|
||||
```
|
||||
|
||||
```shell
|
||||
cd smoke-test/
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
@ -70,34 +98,40 @@ cd ../
|
||||
```
|
||||
|
||||
Once you have compiled & packaged the project or appropriate module you can deploy the entire system via docker-compose by running:
|
||||
```
|
||||
|
||||
```shell
|
||||
./gradlew quickstart
|
||||
```
|
||||
|
||||
Replace whatever container you want in the existing deployment.
|
||||
I.e, replacing datahub's backend (GMS):
|
||||
```
|
||||
|
||||
```shell
|
||||
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-gms)
|
||||
```
|
||||
|
||||
Running the local version of the frontend
|
||||
```
|
||||
|
||||
```shell
|
||||
(cd docker && COMPOSE_DOCKER_CLI_BUILD=1 DOCKER_BUILDKIT=1 docker-compose -p datahub -f docker-compose-without-neo4j.yml -f docker-compose-without-neo4j.override.yml -f docker-compose.dev.yml up -d --no-deps --force-recreate --build datahub-frontend-react)
|
||||
```
|
||||
|
||||
## IDE Support
|
||||
The recommended IDE for DataHub development is [IntelliJ IDEA](https://www.jetbrains.com/idea/).
|
||||
You can run the following command to generate or update the IntelliJ project file
|
||||
```
|
||||
|
||||
The recommended IDE for DataHub development is [IntelliJ IDEA](https://www.jetbrains.com/idea/).
|
||||
You can run the following command to generate or update the IntelliJ project file.
|
||||
|
||||
```shell
|
||||
./gradlew idea
|
||||
```
|
||||
|
||||
Open `datahub.ipr` in IntelliJ to start developing!
|
||||
|
||||
For consistency please import and auto format the code using [LinkedIn IntelliJ Java style](../gradle/idea/LinkedIn%20Style.xml).
|
||||
|
||||
|
||||
## Windows Compatibility
|
||||
|
||||
For optimal performance and compatibility, we strongly recommend building on a Mac or Linux system.
|
||||
For optimal performance and compatibility, we strongly recommend building on a Mac or Linux system.
|
||||
Please note that we do not actively support Windows in a non-virtualized environment.
|
||||
|
||||
If you must use Windows, one workaround is to build within a virtualized environment, such as a VM(Virtual Machine) or [WSL(Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl).
|
||||
@ -105,37 +139,41 @@ This approach can help ensure that your build environment remains isolated and s
|
||||
|
||||
## Common Build Issues
|
||||
|
||||
### Getting `Unsupported class file major version 57`
|
||||
#### Getting `Unsupported class file major version 57`
|
||||
|
||||
You're probably using a Java version that's too new for gradle. Run the following command to check your Java version
|
||||
```
|
||||
|
||||
```shell
|
||||
java --version
|
||||
```
|
||||
|
||||
While it may be possible to build and run DataHub using newer versions of Java, we currently only support [Java 11](https://openjdk.org/projects/jdk/11/) (aka Java 11).
|
||||
|
||||
### Getting `cannot find symbol` error for `javax.annotation.Generated`
|
||||
#### Getting `cannot find symbol` error for `javax.annotation.Generated`
|
||||
|
||||
Similar to the previous issue, please use Java 1.8 to build the project.
|
||||
You can install multiple version of Java on a single machine and switch between them using the `JAVA_HOME` environment variable. See [this document](https://docs.oracle.com/cd/E21454_01/html/821-2531/inst_jdk_javahome_t.html) for more details.
|
||||
|
||||
### `:metadata-models:generateDataTemplate` task fails with `java.nio.file.InvalidPathException: Illegal char <:> at index XX` or `Caused by: java.lang.IllegalArgumentException: 'other' has different root` error
|
||||
#### `:metadata-models:generateDataTemplate` task fails with `java.nio.file.InvalidPathException: Illegal char <:> at index XX` or `Caused by: java.lang.IllegalArgumentException: 'other' has different root` error
|
||||
|
||||
This is a [known issue](https://github.com/linkedin/rest.li/issues/287) when building the project on Windows due a bug in the Pegasus plugin. Please refer to [Windows Compatibility](/docs/developers.md#windows-compatibility).
|
||||
This is a [known issue](https://github.com/linkedin/rest.li/issues/287) when building the project on Windows due a bug in the Pegasus plugin. Please refer to [Windows Compatibility](/docs/developers.md#windows-compatibility).
|
||||
|
||||
### Various errors related to `generateDataTemplate` or other `generate` tasks
|
||||
#### Various errors related to `generateDataTemplate` or other `generate` tasks
|
||||
|
||||
As we generate quite a few files from the models, it is possible that old generated files may conflict with new model changes. When this happens, a simple `./gradlew clean` should reosolve the issue.
|
||||
As we generate quite a few files from the models, it is possible that old generated files may conflict with new model changes. When this happens, a simple `./gradlew clean` should reosolve the issue.
|
||||
|
||||
### `Execution failed for task ':metadata-service:restli-servlet-impl:checkRestModel'`
|
||||
#### `Execution failed for task ':metadata-service:restli-servlet-impl:checkRestModel'`
|
||||
|
||||
This generally means that an [incompatible change](https://linkedin.github.io/rest.li/modeling/compatibility_check) was introduced to the rest.li API in GMS. You'll need to rebuild the snapshots/IDL by running the following command once
|
||||
```
|
||||
|
||||
```shell
|
||||
./gradlew :metadata-service:restli-servlet-impl:build -Prest.model.compatibility=ignore
|
||||
```
|
||||
|
||||
### `java.io.IOException: No space left on device`
|
||||
#### `java.io.IOException: No space left on device`
|
||||
|
||||
This means you're running out of space on your disk to build. Please free up some space or try a different disk.
|
||||
|
||||
### `Build failed` for task `./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint`
|
||||
#### `Build failed` for task `./gradlew :datahub-frontend:dist -x yarnTest -x yarnLint`
|
||||
|
||||
This could mean that you need to update your [Yarn](https://yarnpkg.com/getting-started/install) version
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user