datahub/metadata-ingestion/docs/sources/looker/lookml_post.md

Configuration Notes

API-Based Lineage Extraction and Reachable Views

When use_api_for_view_lineage: true is enabled, DataHub uses the LookerQueryAPIBasedViewUpstream implementation to extract lineage. This approach:

• Uses SQL from Looker API: The system queries the Looker API to generate fully resolved SQL statements for views, which are then parsed to extract column-level and table-level lineage. This provides more accurate lineage than regex-based parsing.

• Works Only for Reachable Views: The Looker Query API requires an explore name to generate SQL queries. Therefore, this method only works for views that are reachable from explores defined in your LookML model files. A view is considered "reachable" if it is referenced by at least one explore (either directly or through joins).

• Fallback Behavior: Views that are not reachable from any explore cannot use the API-based approach and will automatically fall back to regex-based parsing. If emit_reachable_views_only: true (default), unreachable views are skipped entirely.

Example:

source:
  type: lookml
  config:
    # Enable API-based lineage (requires reachable views)
    use_api_for_view_lineage: true

    # Control whether unreachable views are processed
    # If true (default), only views referenced by explores are processed
    # If false, all views are processed, but unreachable ones use regex parsing
    emit_reachable_views_only: true

When a view is not reachable:

• If emit_reachable_views_only: true: The view is skipped and a warning is logged
• If emit_reachable_views_only: false: The view is processed using regex-based parsing (may have limited lineage accuracy)

Liquid Template Support and Limitations

1. Handling Liquid Templates

   If a view contains a liquid template, for example:

   sql_table_name: {{ user_attributes['db'] }}.kafka_streaming.events
   

   where db=ANALYTICS_PROD, you need to specify the values of those variables in the liquid_variables configuration as shown below:

   liquid_variables:
     user_attributes:
       db: ANALYTICS_PROD
   

2. Resolving LookML Constants

   If a view contains a LookML constant, for example:

   sql_table_name: @{db}.kafka_streaming.events;
   

   Ingestion attempts to resolve it's value by looking at project manifest files

   manifest.lkml
     constant: db {
         value: "ANALYTICS_PROD"
     }
   

   • If the constant's value is not resolved or incorrectly resolved, you can specify lookml_constants configuration in ingestion recipe as shown below. The constant value in recipe takes precedence over constant values resolved from manifest.

      ```yml
      lookml_constants:
        db: ANALYTICS_PROD
      ```
     

Limitations:

• Supported: Simple variable interpolation ({{ var }}) and condition directives ({% condition filter_name %} field {% endcondition %})
• Unsupported: Conditional logic with if/else/endif and custom Looker tags like date_start, date_end, and parameter

Additional Notes

Important: Unsupported templates may cause lineage extraction to fail for some assets.

Although liquid variables and LookML constants can be used anywhere in LookML code, their values are currently resolved only for LookML views by DataHub LookML ingestion. This behavior is sufficient since LookML ingestion processes only views and their upstream dependencies.

Multi-Project LookML (Advanced)

Looker projects support organization as multiple git repos, with remote includes that can refer to projects that are stored in a different repo. If your Looker implementation uses multi-project setup, you can configure the LookML source to pull in metadata from your remote projects as well.

If you are using local or remote dependencies, you will see include directives in your lookml files that look like this:

include: "//e_flights/views/users.view.lkml"
include: "//e_commerce/public/orders.view.lkml"

Also, you will see projects that are being referred to listed in your manifest.lkml file. Something like this:

project_name: this_project

local_dependency: {
    project: "my-remote-project"
}

remote_dependency: ga_360_block {
  url: "https://github.com/llooker/google_ga360"
  ref: "0bbbef5d8080e88ade2747230b7ed62418437c21"
}

To ingest Looker repositories that are including files defined in other projects, you will need to use the project_dependencies directive within the configuration section. Consider the following scenario:

• Your primary project refers to a remote project called my_remote_project
• The remote project is homed in the GitHub repo my_org/my_remote_project
• You have provisioned a GitHub deploy key and stored the credential in the environment variable (or UI secret), ${MY_REMOTE_PROJECT_DEPLOY_KEY}

In this case, you can add this section to your recipe to activate multi-project LookML ingestion.

source:
  type: lookml
  config:
    ... other config variables

    project_dependencies:
      my_remote_project:
         repo: my_org/my_remote_project
         deploy_key: ${MY_REMOTE_PROJECT_DEPLOY_KEY}

Under the hood, DataHub will check out your remote repository using the provisioned deploy key, and use it to navigate includes that you have in the model files from your primary project.

If you have the remote project checked out locally, and do not need DataHub to clone the project for you, you can provide DataHub directly with the path to the project like the config snippet below:

source:
  type: lookml
  config:
    ... other config variables

    project_dependencies:
      my_remote_project: /path/to/local_git_clone_of_remote_project

:::note

This is not the same as ingesting the remote project as a primary Looker project because DataHub will not be processing the model files that might live in the remote project. If you want to additionally include the views accessible via the models in the remote project, create a second recipe where your remote project is the primary project.

:::

Handling Large Views with Many Fields

For Looker views with a large number of fields (100+), DataHub automatically uses field splitting to ensure reliable lineage extraction. This feature splits large field sets into manageable chunks, processes them in parallel, and combines the results.

:::important

API Configuration Required: Field splitting requires Looker API credentials to be configured. You must:

1. Provide the api configuration section with your Looker credentials
2. Set use_api_for_view_lineage: true to enable API-based lineage extraction

Without API configuration, field splitting will not be available and the system will fall back to regex-based parsing, which may fail for large views.

Reachable Views Only: The LookerQueryAPIBasedViewUpstream implementation (used for field splitting) works by querying the Looker API to generate SQL statements for views. This approach only works for reachable views - views that are referenced by explores defined in your LookML model files. Views that are not reachable from any explore cannot be queried via the Looker API and will fall back to regex-based parsing. The emit_reachable_views_only configuration option controls whether only reachable views are processed.

:::

When Field Splitting is Used

Field splitting is automatically triggered when:

• use_api_for_view_lineage: true is set
• Looker API credentials are provided
• A view has more fields than the configured threshold (default: 100 fields)

You can adjust this threshold based on your needs:

source:
  type: lookml
  config:
    # Adjust the threshold for field splitting (default: 100)
    field_threshold_for_splitting: 100

When to adjust the threshold:

• Lower the threshold (e.g., 50) if you experience SQL parsing failures with views that have 50-100 fields
• Raise the threshold (e.g., 150) if your views consistently have 100+ fields and you want to minimize API calls

Partial Lineage Results

By default, DataHub will return partial lineage results even if some field chunks fail to parse. This ensures you get lineage information for working fields rather than complete failure.

source:
  type: lookml
  config:
    # Allow partial lineage when some chunks fail (default: true)
    allow_partial_lineage_results: true

When to disable:

• Set to false if you want strict validation and prefer complete failure over partial results
• Useful for debugging to identify problematic views that need attention

Individual Field Fallback

When a chunk of fields fails, DataHub can automatically attempt to process each field individually. This helps:

• Maximize lineage extraction by processing working fields
• Identify specific problematic fields that cause issues
• Provide detailed reporting on which fields fail

source:
  type: lookml
  config:
    # Enable individual field processing when chunks fail (default: true)
    enable_individual_field_fallback: true

When to disable:

• Set to false if you want faster processing and don't need to identify problematic fields
• Useful if you know all fields in a view are valid and want to skip the fallback overhead

Parallel Processing Performance

Field chunks are processed in parallel to improve performance. You can control the number of worker threads:

source:
  type: lookml
  config:
    # Number of parallel workers (default: 10, max: 100)
    max_workers_for_parallel_processing: 10

Performance tuning:

• Increase workers (e.g., 20-30) for faster processing if you have many large views and sufficient system resources
• Decrease workers (e.g., 5) if you're hitting API rate limits or have limited system resources
• Set to 1 to process sequentially (useful for debugging)

Important: The maximum allowed value is 100 to prevent resource exhaustion. Values above 100 will be automatically capped with a warning.

Complete Configuration Example

Here's a complete example configuration for handling large views:

source:
  type: lookml
  config:
    base_folder: /path/to/lookml

    # API configuration (REQUIRED for field splitting)
    api:
      base_url: "https://your-instance.cloud.looker.com"
      client_id: ${LOOKER_CLIENT_ID}
      client_secret: ${LOOKER_CLIENT_SECRET}

    # Enable API-based lineage extraction (REQUIRED for field splitting)
    use_api_for_view_lineage: true

    # Optional: Enable API caching for better performance
    use_api_cache_for_view_lineage: true

    # Large view handling configuration
    field_threshold_for_splitting: 100 # Split views with >100 fields
    allow_partial_lineage_results: true # Return partial results on errors
    enable_individual_field_fallback: true # Process fields individually on chunk failure
    max_workers_for_parallel_processing: 10 # Parallel processing workers

Important Notes:

• The api section with credentials is required for field splitting to work
• use_api_for_view_lineage: true must be set to enable API-based lineage extraction
• Without API configuration, field splitting features are not available
• Reachable Views Only: Field splitting via LookerQueryAPIBasedViewUpstream only works for views that are reachable from explores. The Looker Query API requires an explore name to generate SQL, so views not referenced by any explore will use regex-based parsing instead
• The emit_reachable_views_only configuration (default: true) controls whether unreachable views are processed at all

Check ingestion logs for:

• Field splitting statistics: View 'view_name' has X fields, exceeding threshold of Y. Splitting into multiple queries
• Success rates: Combined results for view 'view_name': X tables, Y column lineages, success rate: Z%
• Problematic fields: Warnings about specific fields that fail processing

Common issues:

• Field splitting not working: Verify use_api_for_view_lineage: true and API credentials are configured
• Low success rate (<50%): Consider lowering field_threshold_for_splitting or investigating problematic fields
• API rate limiting: Reduce max_workers_for_parallel_processing to decrease concurrent requests
• Memory issues: Reduce max_workers_for_parallel_processing if you experience memory pressure

Troubleshooting Large View Lineage Extraction

If you have Looker views with many fields (100+) and are experiencing lineage extraction issues, the following troubleshooting steps can help:

:::important

Prerequisites: Field splitting requires Looker API configuration. Ensure you have:

• api section with valid credentials configured
• use_api_for_view_lineage: true enabled

:::

Issue: Field splitting not working

Symptoms:

• Large views still fail even with field splitting configuration
• No field splitting messages in logs
• Views fall back to regex-based parsing

Solutions:

1. Verify API configuration:

   source:
     type: lookml
     config:
       api:
         base_url: "https://your-instance.cloud.looker.com"
         client_id: ${LOOKER_CLIENT_ID}
         client_secret: ${LOOKER_CLIENT_SECRET}
       use_api_for_view_lineage: true # Must be enabled
   

2. Check API credentials:

   • Verify credentials have admin privileges (required for API access)
   • Test API connection separately if needed
   • Check logs for authentication errors

3. Verify view-to-explore mapping:

   • Field splitting requires views to be mapped to explores (views must be reachable from explores)
   • Check logs for warnings about missing explore mappings
   • Ensure your views are referenced by at least one explore in your model files
   • If emit_reachable_views_only: true (default), unreachable views are skipped entirely

Issue: Lineage extraction fails for large views

Symptoms:

• Views with 100+ fields show no lineage
• Error messages about SQL parsing failures
• Incomplete lineage information

Solutions:

1. Verify field splitting is working: Check your ingestion logs for messages like:

   View 'your_view' has 150 fields, exceeding threshold of 100. Splitting into multiple queries for partial lineage.
   

   If you don't see this message, field splitting may not be triggered. Lower the threshold:

   field_threshold_for_splitting: 50 # Lower threshold
   

2. Check success rates: Look for statistics in logs:

   Combined results for view 'your_view': 5 tables, 120 column lineages, success rate: 80.0%
   

   • High success rate (>80%): System is working well
   • Medium success rate (50-80%): Some fields may be problematic, but partial lineage is available
   • Low success rate (<50%): Consider investigating specific fields or lowering threshold

3. Enable individual field fallback: If chunks are failing, enable individual field processing to identify problematic fields:

   enable_individual_field_fallback: true
   

   Check logs for warnings about specific fields that fail.

4. Adjust parallel processing: If you're hitting API rate limits, reduce workers:

   max_workers_for_parallel_processing: 5 # Reduce from default 10
   

Issue: Slow processing for large views

Symptoms:

• Ingestion takes a long time for views with many fields
• Processing appears sequential

Solutions:

1. Increase parallel workers:

   max_workers_for_parallel_processing: 20 # Increase from default 10
   

   Note: Monitor system resources and API rate limits

2. Enable API caching:

   use_api_cache_for_view_lineage: true # Enable server-side caching
   

3. Verify parallel processing is active: Check logs for concurrent processing indicators. If processing appears sequential, verify max_workers_for_parallel_processing is set correctly.

Issue: Memory or resource exhaustion

Symptoms:

• Ingestion process runs out of memory
• System becomes unresponsive during ingestion

Solutions:

1. Reduce parallel workers:

   max_workers_for_parallel_processing: 5 # Reduce concurrent processing
   

2. Process sequentially:

   max_workers_for_parallel_processing: 1 # Disable parallel processing
   

3. Increase chunk size:

   field_threshold_for_splitting: 150 # Larger chunks = fewer concurrent operations
   

Issue: Incomplete lineage for some fields

Symptoms:

• Some fields show lineage, others don't
• Partial lineage information available

Solutions:

1. This is expected behavior when allow_partial_lineage_results: true (default)

   • Partial lineage is better than no lineage
   • Check logs for specific fields that fail

2. To identify problematic fields:

   • Enable enable_individual_field_fallback: true (default)
   • Check logs for warnings about specific fields
   • Review those fields in Looker to identify issues

3. For strict validation:

   allow_partial_lineage_results: false # Fail completely if any chunk fails
   

   Note: This may result in no lineage for large views if any chunk fails

Best Practices

1. Start with defaults: The default configuration works well for most cases
2. Monitor logs: Check field splitting statistics and success rates
3. Tune gradually: Adjust one parameter at a time and monitor results
4. Consider your environment:
   • High-resource systems: Can increase max_workers_for_parallel_processing
   • Rate-limited APIs: Should decrease max_workers_for_parallel_processing
   • Many problematic fields: Enable enable_individual_field_fallback
   • Strict validation needs: Disable allow_partial_lineage_results

Debugging LookML Parsing Errors

If you see messages like my_file.view.lkml': "failed to load view file: Unable to find a matching expression for '<literal>' on line 5" in the failure logs, it indicates a parsing error for the LookML file.

The first thing to check is that the Looker IDE can validate the file without issues. You can check this by clicking this "Validate LookML" button in the IDE when in development mode.

If that's not the issue, it might be because DataHub's parser, which is based on the joshtemple/lkml library, is slightly more strict than the official Looker parser. Note that there's currently only one known discrepancy between the two parsers, and it's related to using leading colons in blocks.

To check if DataHub can parse your LookML file syntax, you can use the lkml CLI tool. If this raises an exception, DataHub will fail to parse the file.

pip install lkml

lkml path/to/my_file.view.lkml