yujunjun/datahub
2
0
Fork 0
mirror of https://github.com/datahub-project/datahub.git synced 2025-08-18 14:16:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ingest): postgres - ignore information_schema tables by default (#4069)

Browse Source
This commit is contained in:
Kevin Hu 2022-02-10 02:20:25 -05:00 committed by GitHub
parent 076848ff55
commit 9bdc9af7b9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 8 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
metadata-ingestion/source_docs/postgres.md
View File

@ -17,7 +17,7 @@ This plugin extracts the following:
- Table, row, and column statistics via optional [SQL profiling](./sql_profiles.md) - Table, row, and column statistics via optional [SQL profiling](./sql_profiles.md)
| Capability | Status | Details | | Capability | Status | Details |
|-------------------|--------|------------------------------------------| | --------------- | ------ | ----------------------------- |
| Data Containers | ✔️ | | | Data Containers | ✔️ | |
| Data Domains | ✔️ | [link](../../docs/domains.md) | | Data Domains | ✔️ | [link](../../docs/domains.md) |
@ -53,7 +53,7 @@ Note that a `.` is used to denote nested fields in the YAML recipe.
As a SQL-based service, the Athena integration is also supported by our SQL profiler. See [here](./sql_profiles.md) for more details on configuration. As a SQL-based service, the Athena integration is also supported by our SQL profiler. See [here](./sql_profiles.md) for more details on configuration.
| Field | Required | Default | Description | | Field | Required | Default | Description |
|--------------------------------|----------|----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | ------------------------------ | -------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `username` | | | PostgreSQL username. | | `username` | | | PostgreSQL username. |
| `password` | | | PostgreSQL password. | | `password` | | | PostgreSQL password. |
| `host_port` | ✅ | | PostgreSQL host URL. | | `host_port` | ✅ | | PostgreSQL host URL. |

2
metadata-ingestion/src/datahub/ingestion/source/sql/postgres.py
View File

@ -9,6 +9,7 @@ import sqlalchemy.dialects.postgresql as custom_types
# https://geoalchemy-2.readthedocs.io/en/latest/core_tutorial.html#reflecting-tables. # https://geoalchemy-2.readthedocs.io/en/latest/core_tutorial.html#reflecting-tables.
from geoalchemy2 import Geometry # noqa: F401 from geoalchemy2 import Geometry # noqa: F401
from datahub.configuration.common import AllowDenyPattern
from datahub.ingestion.source.sql.sql_common import ( from datahub.ingestion.source.sql.sql_common import (
BasicSQLAlchemyConfig, BasicSQLAlchemyConfig,
SQLAlchemySource, SQLAlchemySource,
@ -29,6 +30,7 @@ register_custom_type(custom_types.HSTORE, MapTypeClass)
class PostgresConfig(BasicSQLAlchemyConfig): class PostgresConfig(BasicSQLAlchemyConfig):
# defaults # defaults
scheme = "postgresql+psycopg2" scheme = "postgresql+psycopg2"
schema_pattern = AllowDenyPattern(deny=["information_schema"])
def get_identifier(self: BasicSQLAlchemyConfig, schema: str, table: str) -> str: def get_identifier(self: BasicSQLAlchemyConfig, schema: str, table: str) -> str:
regular = f"{schema}.{table}" regular = f"{schema}.{table}"