The upgrade jobs take arguments as command line args to the job itself rather than environment variables for job specific configuration. The RestoreIndices job is specified through the `-u RestoreIndices` upgrade ID parameter and then additional parameters are specified like `-a batchSize=1000`.
The following configurations are available:
### Time-Based Filtering
*`lePitEpochMs`: Restore records created before this timestamp (in milliseconds)
*`gePitEpochMs`: Restore records created after this timestamp (in milliseconds)
### Pagination and Performance Options
*`urnBasedPagination`: Enable key-based pagination instead of offset-based pagination. Recommended for large datasets as it's typically more efficient.
*`startingOffset`: When using default pagination, start from this offset
*`lastUrn`: Resume from a specific URN when using URN-based pagination
*`lastAspect`: Used with lastUrn to resume from a specific aspect, preventing reprocessing
*`numThreads`: Number of concurrent threads for processing restoration, only used with default offset based paging
*`batchSize`: Configures the size of each batch as the job pages through rows
*`batchDelayMs`: Adds a delay in between each batch to avoid overloading backend systems
### Content Filtering
*`aspectNames`: Comma-separated list of aspects to restore (e.g., "ownership,status")
*`urnLike`: SQL LIKE pattern to filter URNs (e.g., "urn:li:dataset%")
### Nuclear option
*`clean`: This option wipes out the current indices by running deletes of all the documents to guarantee a consistent state with SQL. This is generally not recommended unless there is significant data corruption on the instance.
### Helm
These are available in the helm charts as configurations for Kubernetes deployments under the `datahubUpgrade.restoreIndices.args` path which will set them up as args to the pod command.
If you are on a custom docker-compose deployment, run the following command (you need to checkout [the source repository](https://github.com/datahub-project/datahub)) from the root of the repo to send MAE for each aspect in the local database.
By default this command will not clear the search and graph indices before restoring, thous potentially leading to inconsistencies between the local database and the indices, in case aspects were previously deleted in the local database but were not removed from the correponding index.
If you need to clear the search and graph indices before restoring, add `-a clean` to the end of the command. Please take note that the search and graph services might not be fully functional during reindexing when the indices are cleared.
```shell
./docker/datahub-upgrade/datahub-upgrade.sh -u RestoreIndices -a clean
Once the job completes, your indices will have been restored.
:::info
By default the restore indices job template will not clear the search and graph indices before restoring, thous potentially leading to inconsistencies between the local database and the indices, in case aspects were previously deleted in the local database but were not removed from the correponding index.
If you need to clear the search and graph indices before restoring, modify the `values.yaml` for your deployment and overwrite the default arguments of the restore indices job template to include the `-a clean` argument. Please take note that the search and graph services might not be fully functional during reindexing when the indices are cleared.
```yaml
datahubUpgrade:
restoreIndices:
image:
args:
- "-u"
- "RestoreIndices"
- "-a"
- "batchSize=1000" # default value of datahubUpgrade.batchSize
- "-a"
- "batchDelayMs=100" # default value of datahubUpgrade.batchDelayMs
