7.7 KiB
| description |
|---|
| This guide will help you configure metadata ingestion workflows using the Snowflake connector. |
Snowflake in the UI
- Ensure your system meets the requirements
- Visit the Services page
- Initiate new service creation
- Select service type
- Name and describe your service
- Configure service connection
- Configure metadata ingestion
- Review configuration and save
1. Ensure your system meets the requirements
Please ensure that your host system meets the requirements listed below.
OpenMetadata (version 0.9.0 or later)
To deploy OpenMetadata, follow the procedure Try OpenMetadata in Docker.
2. Visit the Services page
You may configure scheduled ingestion workflows from the Services page in the OpenMetadata UI. To visit the Services page, select Services from the Settings menu.
3. Initiate a new service creation
From the Database Service UI, click the Add New Service button to add your Snowflake service to OpenMetadata for metadata ingestion.
4. Select service type
Select Snowflake as the service type.
5. Name and describe your service
Provide a name and description for your service as illustrated below.
Name
OpenMetadata uniquely identifies services by their Name. Provide a name that distinguishes your deployment from other services, including other Snowflake services that you might be ingesting metadata from.
Description
Provide a description for your Snowflake service that enables other users to determine whether it might provide data of interest to them.
6. Configure service connection
In this step, we will configure the connection settings required for this connector. Please follow the instructions below to ensure that you've configured the connector to read from your Snowflake service as desired.
Host
Enter fully qualified hostname for your Snowflake deployment in the Host field.
Port
Enter the port number on which your Snowflake deployment listens for client connections in the Port field.
Username
Enter username of your Snowflake user in the Username field. The user specified should be authorized to read all databases you want to include in the metadata ingestion workflow.
Password
Enter the password for your Snowflake user in the Password field.
Database (optional)
If you want to limit metadata ingestion to a single database, enter the name of this database in the Database field. If no value is entered for this field, the connector will ingest metadata from all databases that the specified user is authorized to read.
7. Configure metadata ingestion
In this step we will configure the metadata ingestion settings for your Snowflake deployment. Please follow the instructions below to ensure that you've configured the connector to read from your Snowflake service as desired.
Ingestion name
OpenMetadata will pre-populate the Ingestion name field. You may modify the Ingestion name, but if you do, please ensure it is unique for this service.
Include (Table Filter Pattern)
Use to table filter patterns to control whether or not to include tables as part of metadata ingestion and data profiling.
Explicitly include tables by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all tables with names matching one or more of the supplied regular expressions. All other tables will be excluded. See the figure above for an example.
Exclude (Table Filter Pattern)
Explicitly exclude tables by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all tables with names matching one or more of the supplied regular expressions. All other tables will be included. See the figure above for an example.
Include (Schema Filter Pattern)
Use to schema filter patterns to control whether or not to include schemas as part of metadata ingestion and data profiling.
Explicitly include schemas by adding a list of comma-separated regular expressions to the Include field. OpenMetadata will include all schemas with names matching one or more of the supplied regular expressions. All other schemas will be excluded.
Exclude (Schema Filter Pattern)
Explicitly exclude schemas by adding a list of comma-separated regular expressions to the Exclude field. OpenMetadata will exclude all schemas with names matching one or more of the supplied regular expressions. All other schemas will be included.
Include views (toggle)
Set the Include views toggle to the on position to control whether or not to include views as part of metadata ingestion and data profiling.
Explicitly include views by adding the following key-value pair in the source.config field of your configuration file.
Enable data profiler (toggle)
The data profiler ingests usage information for tables. This enables you to assess the frequency of use, reliability, and other details.
When enabled, the data profiler will run as part of metadata ingestion. Running the data profiler increases the amount of time it takes for metadata ingestion, but provides the benefits mentioned above.
Set the Enable data profiler toggle to the on position to enable the data profiler.
Ingest sample data (toggle)
Set the Ingest sample data toggle to the on position to control whether or not to generate sample data to include in table views in the OpenMetadata user interface.
Every
Use the Every drop down menu to select the interval at which you want to ingest metadata. Your options are as follows:
- Hour: Ingest metadata once per hour
- Day: Ingest metadata once per day
- Week: Ingest metadata once per week
Day
The Day selector is only active when ingesting metadata once per week. Use the Day selector to set the day of the week on which to ingest metadata.
Minute
The Minute dropdown is only active when ingesting metadata once per hour. Use the Minute drop down menu to select the minute of the hour at which to begin ingesting metadata.
Time
The Time drop down menus are active when ingesting metadata either once per day or once per week. Use the time drop downs to select the time of day at which to begin ingesting metadata.
Start date (UTC)
Use the Start date selector to choose the date at which to begin ingesting metadata according to the defined schedule.
End date (UTC)
Use the End date selector to choose the date at which to stop ingesting metadata according to the defined schedule. If no end date is set, metadata ingestion will continue according to the defined schedule indefinitely.
8. Review configuration and save
Review your configuration settings. If they match what you intended, click Save to create the service and schedule metadata ingestion.
If something doesn't look right, click the Previous button to return to the appropriate step and change the settings as needed.
