yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-11-09 16:03:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ingest): metabase - allow configuring how database engines get mapped to platforms (#3869)

Browse Source
This commit is contained in:
iasoon 2022-01-12 09:52:55 +01:00 committed by GitHub
parent 5f701ebb3c
commit 988baeb467
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 14 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
metadata-ingestion/source_docs/metabase.md
View File

@ -81,10 +81,18 @@ sink:
| `username` | ✅ | | Metabase username. | | `username` | ✅ | | Metabase username. |
| `password` | ✅ | | Metabase password. | | `password` | ✅ | | Metabase password. |
| `database_alias_map` | | | Database name map to use when constructing dataset URN. | | `database_alias_map` | | | Database name map to use when constructing dataset URN. |
| `engine_platform_map`| | | Custom mappings between metabase database engines and DataHub platforms |
| `default_schema` | | `public` | Default schema name to use when schema is not provided in an SQL query | | `default_schema` | | `public` | Default schema name to use when schema is not provided in an SQL query |
| `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. | | `env` | | `"PROD"` | Environment to use in namespace when constructing URNs. |
Metabase databases will be mapped to a DataHub platform based on the engine listed in the
[api/database](https://www.metabase.com/docs/latest/api-documentation.html#database) response. This mapping can be
customized by using the `engine_platform_map` config option. For example, to map databases using the `athena` engine to
the underlying datasets in the `glue` platform, the following snippet can be used:
```yml
engine_platform_map:
athena: glue
```
DataHub will try to determine database name from Metabase [api/database](https://www.metabase.com/docs/latest/api-documentation.html#database) DataHub will try to determine database name from Metabase [api/database](https://www.metabase.com/docs/latest/api-documentation.html#database)
payload. However, the name can be overridden from `database_alias_map` for a given database connected to Metabase. payload. However, the name can be overridden from `database_alias_map` for a given database connected to Metabase.

5
metadata-ingestion/src/datahub/ingestion/source/metabase.py
View File

@ -42,6 +42,7 @@ class MetabaseConfig(ConfigModel):
username: Optional[str] = None username: Optional[str] = None
password: Optional[str] = None password: Optional[str] = None
database_alias_map: Optional[dict] = None database_alias_map: Optional[dict] = None
engine_platform_map: Optional[dict] = None
default_schema: str = "public" default_schema: str = "public"
env: str = builder.DEFAULT_ENV env: str = builder.DEFAULT_ENV
@ -477,6 +478,10 @@ class MetabaseSource(Source):
"sqlserver": "mssql", "sqlserver": "mssql",
"bigquery-cloud-sdk": "bigquery", "bigquery-cloud-sdk": "bigquery",
} }
if self.config.engine_platform_map is not None:
engine_mapping.update(self.config.engine_platform_map)
if engine in engine_mapping: if engine in engine_mapping:
platform = engine_mapping[engine] platform = engine_mapping[engine]
else: else: