mirror of
https://github.com/open-metadata/OpenMetadata.git
synced 2025-11-02 19:48:17 +00:00
Fixes: 11540 Added api_version and domain fields to Salesforce (#11618)
* Added sales new fields * fixed image after merge * Fix Failing Test --------- Co-authored-by: mohitdeuex <mohit.y@deuexsolutions.com> Co-authored-by: Mohit Yadav <105265192+mohityadav766@users.noreply.github.com> Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
This commit is contained in:
parent
5164858b4b
commit
e6a7ffc57c
@ -7,8 +7,9 @@ source:
|
||||
username: username
|
||||
password: password
|
||||
securityToken: securityToken
|
||||
scheme: salesforce
|
||||
sobjectName: sobjectName
|
||||
# salesforceApiVersion: 42.0
|
||||
# salesforceDomain: login
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -18,4 +19,4 @@ sink:
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: http://localhost:8585/api
|
||||
authProvider: no-auth
|
||||
authProvider: no-auth
|
||||
@ -35,6 +35,8 @@ def get_connection(connection: SalesforceConnection) -> Engine:
|
||||
connection.username,
|
||||
password=connection.password.get_secret_value(),
|
||||
security_token=connection.securityToken.get_secret_value(),
|
||||
domain=connection.salesforceDomain,
|
||||
version=connection.salesforceApiVersion,
|
||||
)
|
||||
|
||||
|
||||
|
||||
@ -105,9 +105,22 @@ This is a sample config for Salesforce:
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**salesforceApiVersion**: Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version. Enter the numerical value in the field, For example `42.0`.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**salesforceDomain**: When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -123,7 +136,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=11 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -131,7 +144,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=10 %}
|
||||
{% codeInfo srNumber=12 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -141,13 +154,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -181,16 +194,22 @@ source:
|
||||
sobjectName: sobjectName
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
salesforceApiVersion: 42.0
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
salesforceDomain: login
|
||||
```
|
||||
```yaml {% srNumber=8 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -221,13 +240,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=11 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=10 %}
|
||||
```yaml {% srNumber=12 %}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: "http://localhost:8585/api"
|
||||
@ -269,7 +288,7 @@ Create a Python file in your Airflow DAGs directory with the following contents:
|
||||
{% codeInfoContainer %}
|
||||
|
||||
|
||||
{% codeInfo srNumber=11 %}
|
||||
{% codeInfo srNumber=13 %}
|
||||
|
||||
#### Import necessary modules
|
||||
|
||||
@ -279,7 +298,7 @@ Here we are also importing all the basic requirements to parse YAMLs, handle dat
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=12 %}
|
||||
{% codeInfo srNumber=14 %}
|
||||
|
||||
**Default arguments for all tasks in the Airflow DAG.**
|
||||
|
||||
@ -287,19 +306,19 @@ Here we are also importing all the basic requirements to parse YAMLs, handle dat
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=13 %}
|
||||
{% codeInfo srNumber=15 %}
|
||||
|
||||
- **config**: Specifies config for the metadata ingestion as we prepare above.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=14 %}
|
||||
{% codeInfo srNumber=16 %}
|
||||
|
||||
- **metadata_ingestion_workflow()**: This code defines a function `metadata_ingestion_workflow()` that loads a YAML configuration, creates a `Workflow` object, executes the workflow, checks its status, prints the status to the console, and stops the workflow.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=15 %}
|
||||
{% codeInfo srNumber=17 %}
|
||||
|
||||
- **DAG**: creates a DAG using the Airflow framework, and tune the DAG configurations to whatever fits with your requirements
|
||||
- For more Airflow DAGs creation details visit [here](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#declaring-a-dag).
|
||||
@ -313,7 +332,7 @@ By updating the `YAML configuration`, you will be able to extract metadata from
|
||||
|
||||
{% codeBlock fileName="filename.py" %}
|
||||
|
||||
```python {% srNumber=11 %}
|
||||
```python {% srNumber=13 %}
|
||||
import pathlib
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
@ -330,7 +349,7 @@ except ModuleNotFoundError:
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=12 %}
|
||||
```python {% srNumber=14 %}
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email": ["username@org.com"],
|
||||
@ -343,7 +362,7 @@ default_args = {
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=13 %}
|
||||
```python {% srNumber=15 %}
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
@ -351,7 +370,7 @@ config = """
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=14 %}
|
||||
```python {% srNumber=16 %}
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
@ -363,7 +382,7 @@ def metadata_ingestion_workflow():
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=15 %}
|
||||
```python {% srNumber=17 %}
|
||||
with DAG(
|
||||
"sample_data",
|
||||
default_args=default_args,
|
||||
|
||||
@ -105,9 +105,22 @@ This is a sample config for Salesforce:
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**salesforceApiVersion**: Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version. Enter the numerical value in the field, For example `42.0`.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**salesforceDomain**: When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -123,7 +136,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=11 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -131,7 +144,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=10 %}
|
||||
{% codeInfo srNumber=12 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -141,13 +154,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -181,16 +194,22 @@ source:
|
||||
sobjectName: sobjectName
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
salesforceApiVersion: 42.0
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
salesforceDomain: login
|
||||
```
|
||||
```yaml {% srNumber=8 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -221,13 +240,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=11 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=10 %}
|
||||
```yaml {% srNumber=12 %}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: "http://localhost:8585/api"
|
||||
|
||||
@ -191,6 +191,9 @@ desired.
|
||||
- **Password**: Password to connect to Salesforce.
|
||||
- **Security Token**: Salesforce Security Token is required to access the metadata through APIs. You can checkout [this doc](https://help.salesforce.com/s/articleView?id=sf.user_security_token.htm&type=5) on how to get the security token.
|
||||
- **Salesforce Object Name**: Specify the Salesforce Object Name in case you want to ingest a specific object. If left blank, we will ingest all the Objects.
|
||||
- **Salesforce API Version**: Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version. Enter the numerical value in the field, For example `42.0`.
|
||||
- **Salesforce Domain**: When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Salesforce during the connection. These details must be added as Key-Value pairs.
|
||||
- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Salesforce during the connection. These details must be added as Key-Value pairs.
|
||||
- In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`
|
||||
|
||||
@ -105,9 +105,22 @@ This is a sample config for Salesforce:
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**salesforceApiVersion**: Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version. Enter the numerical value in the field, For example `42.0`.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**salesforceDomain**: When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -123,7 +136,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=11 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -131,7 +144,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=10 %}
|
||||
{% codeInfo srNumber=12 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -141,13 +154,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -181,16 +194,22 @@ source:
|
||||
sobjectName: sobjectName
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
salesforceApiVersion: 42.0
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
salesforceDomain: login
|
||||
```
|
||||
```yaml {% srNumber=8 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -221,13 +240,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=11 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=10 %}
|
||||
```yaml {% srNumber=12 %}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: "http://localhost:8585/api"
|
||||
@ -269,7 +288,7 @@ Create a Python file in your Airflow DAGs directory with the following contents:
|
||||
{% codeInfoContainer %}
|
||||
|
||||
|
||||
{% codeInfo srNumber=11 %}
|
||||
{% codeInfo srNumber=13 %}
|
||||
|
||||
#### Import necessary modules
|
||||
|
||||
@ -279,7 +298,7 @@ Here we are also importing all the basic requirements to parse YAMLs, handle dat
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=12 %}
|
||||
{% codeInfo srNumber=14 %}
|
||||
|
||||
**Default arguments for all tasks in the Airflow DAG.**
|
||||
|
||||
@ -287,19 +306,19 @@ Here we are also importing all the basic requirements to parse YAMLs, handle dat
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=13 %}
|
||||
{% codeInfo srNumber=15 %}
|
||||
|
||||
- **config**: Specifies config for the metadata ingestion as we prepare above.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=14 %}
|
||||
{% codeInfo srNumber=16 %}
|
||||
|
||||
- **metadata_ingestion_workflow()**: This code defines a function `metadata_ingestion_workflow()` that loads a YAML configuration, creates a `Workflow` object, executes the workflow, checks its status, prints the status to the console, and stops the workflow.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=15 %}
|
||||
{% codeInfo srNumber=17 %}
|
||||
|
||||
- **DAG**: creates a DAG using the Airflow framework, and tune the DAG configurations to whatever fits with your requirements
|
||||
- For more Airflow DAGs creation details visit [here](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#declaring-a-dag).
|
||||
@ -313,7 +332,7 @@ By updating the `YAML configuration`, you will be able to extract metadata from
|
||||
|
||||
{% codeBlock fileName="filename.py" %}
|
||||
|
||||
```python {% srNumber=11 %}
|
||||
```python {% srNumber=13 %}
|
||||
import pathlib
|
||||
import yaml
|
||||
from datetime import timedelta
|
||||
@ -330,7 +349,7 @@ except ModuleNotFoundError:
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=12 %}
|
||||
```python {% srNumber=14 %}
|
||||
default_args = {
|
||||
"owner": "user_name",
|
||||
"email": ["username@org.com"],
|
||||
@ -343,7 +362,7 @@ default_args = {
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=13 %}
|
||||
```python {% srNumber=15 %}
|
||||
config = """
|
||||
<your YAML configuration>
|
||||
"""
|
||||
@ -351,7 +370,7 @@ config = """
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=14 %}
|
||||
```python {% srNumber=16 %}
|
||||
def metadata_ingestion_workflow():
|
||||
workflow_config = yaml.safe_load(config)
|
||||
workflow = Workflow.create(workflow_config)
|
||||
@ -363,7 +382,7 @@ def metadata_ingestion_workflow():
|
||||
|
||||
```
|
||||
|
||||
```python {% srNumber=15 %}
|
||||
```python {% srNumber=17 %}
|
||||
with DAG(
|
||||
"sample_data",
|
||||
default_args=default_args,
|
||||
|
||||
@ -105,9 +105,22 @@ This is a sample config for Salesforce:
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
|
||||
**salesforceApiVersion**: Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version. Enter the numerical value in the field, For example `42.0`.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
|
||||
**salesforceDomain**: When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
#### Source Configuration - Source Config
|
||||
|
||||
{% codeInfo srNumber=8 %}
|
||||
{% codeInfo srNumber=10 %}
|
||||
|
||||
The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetadata/blob/main/openmetadata-spec/src/main/resources/json/schema/metadataIngestion/databaseServiceMetadataPipeline.json):
|
||||
|
||||
@ -123,7 +136,7 @@ The `sourceConfig` is defined [here](https://github.com/open-metadata/OpenMetada
|
||||
|
||||
#### Sink Configuration
|
||||
|
||||
{% codeInfo srNumber=9 %}
|
||||
{% codeInfo srNumber=11 %}
|
||||
|
||||
To send the metadata to OpenMetadata, it needs to be specified as `type: metadata-rest`.
|
||||
|
||||
@ -131,7 +144,7 @@ To send the metadata to OpenMetadata, it needs to be specified as `type: metadat
|
||||
|
||||
#### Workflow Configuration
|
||||
|
||||
{% codeInfo srNumber=10 %}
|
||||
{% codeInfo srNumber=12 %}
|
||||
|
||||
The main property here is the `openMetadataServerConfig`, where you can define the host and security provider of your OpenMetadata installation.
|
||||
|
||||
@ -141,13 +154,13 @@ For a simple, local installation using our docker containers, this looks like:
|
||||
|
||||
#### Advanced Configuration
|
||||
|
||||
{% codeInfo srNumber=6 %}
|
||||
{% codeInfo srNumber=8 %}
|
||||
|
||||
**Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
{% /codeInfo %}
|
||||
|
||||
{% codeInfo srNumber=7 %}
|
||||
{% codeInfo srNumber=9 %}
|
||||
|
||||
**Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Athena during the connection. These details must be added as Key-Value pairs.
|
||||
|
||||
@ -181,16 +194,22 @@ source:
|
||||
sobjectName: sobjectName
|
||||
```
|
||||
```yaml {% srNumber=6 %}
|
||||
salesforceApiVersion: 42.0
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
salesforceDomain: login
|
||||
```
|
||||
```yaml {% srNumber=8 %}
|
||||
# connectionOptions:
|
||||
# key: value
|
||||
```
|
||||
```yaml {% srNumber=7 %}
|
||||
```yaml {% srNumber=9 %}
|
||||
# connectionArguments:
|
||||
# key: value
|
||||
```
|
||||
|
||||
|
||||
```yaml {% srNumber=8 %}
|
||||
```yaml {% srNumber=10 %}
|
||||
sourceConfig:
|
||||
config:
|
||||
type: DatabaseMetadata
|
||||
@ -221,13 +240,13 @@ source:
|
||||
# - table4
|
||||
```
|
||||
|
||||
```yaml {% srNumber=9 %}
|
||||
```yaml {% srNumber=11 %}
|
||||
sink:
|
||||
type: metadata-rest
|
||||
config: {}
|
||||
```
|
||||
|
||||
```yaml {% srNumber=10 %}
|
||||
```yaml {% srNumber=12 %}
|
||||
workflowConfig:
|
||||
openMetadataServerConfig:
|
||||
hostPort: "http://localhost:8585/api"
|
||||
|
||||
@ -191,6 +191,9 @@ desired.
|
||||
- **Password**: Password to connect to Salesforce.
|
||||
- **Security Token**: Salesforce Security Token is required to access the metadata through APIs. You can checkout [this doc](https://help.salesforce.com/s/articleView?id=sf.user_security_token.htm&type=5) on how to get the security token.
|
||||
- **Salesforce Object Name**: Specify the Salesforce Object Name in case you want to ingest a specific object. If left blank, we will ingest all the Objects.
|
||||
- **Salesforce API Version**: Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version. Enter the numerical value in the field, For example `42.0`.
|
||||
- **Salesforce Domain**: When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
- **Connection Options (Optional)**: Enter the details for any additional connection options that can be sent to Salesforce during the connection. These details must be added as Key-Value pairs.
|
||||
- **Connection Arguments (Optional)**: Enter the details for any additional connection arguments such as security or protocol configs that can be sent to Salesforce during the connection. These details must be added as Key-Value pairs.
|
||||
- In case you are using Single-Sign-On (SSO) for authentication, add the `authenticator` details in the Connection Arguments as a Key-Value pair as follows: `"authenticator" : "sso_login_url"`
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 122 KiB After Width: | Height: | Size: 118 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 122 KiB After Width: | Height: | Size: 118 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 122 KiB After Width: | Height: | Size: 118 KiB |
@ -5,8 +5,10 @@ import static org.openmetadata.common.utils.CommonUtil.listOf;
|
||||
import static org.openmetadata.service.Entity.ALL_RESOURCES;
|
||||
|
||||
import java.util.ArrayList;
|
||||
import java.util.HashSet;
|
||||
import java.util.List;
|
||||
import java.util.Random;
|
||||
import java.util.Set;
|
||||
import org.junit.jupiter.api.Test;
|
||||
import org.openmetadata.schema.entity.policies.accessControl.Rule;
|
||||
|
||||
@ -26,15 +28,17 @@ class CompiledRuleTest {
|
||||
void testResourceMatch() {
|
||||
// Create a random list of resources
|
||||
Random random = new Random();
|
||||
List<String> ruleResources = new ArrayList<>();
|
||||
for (String resource : RESOURCE_LIST) {
|
||||
if (random.nextBoolean() && !resource.equalsIgnoreCase(ALL_RESOURCES)) {
|
||||
ruleResources.add(resource);
|
||||
Set<String> ruleResources = new HashSet<>();
|
||||
for (int i = 0; i < 3; i++) {
|
||||
String res = RESOURCE_LIST.get(random.nextInt(4));
|
||||
if (!res.equalsIgnoreCase(ALL_RESOURCES)) {
|
||||
ruleResources.add(res);
|
||||
}
|
||||
}
|
||||
|
||||
assertTrue(ruleResources.size() > 0); // Ensure we are setting at least one resource in a rule
|
||||
|
||||
CompiledRule rule = new CompiledRule(new Rule().withName("test").withResources(ruleResources));
|
||||
CompiledRule rule = new CompiledRule(new Rule().withName("test").withResources(new ArrayList<>(ruleResources)));
|
||||
for (String resource : RESOURCE_LIST) {
|
||||
assertEquals(
|
||||
rule.matchResource(resource), ruleResources.contains(resource), "Resource name " + resource + " not matched");
|
||||
|
||||
@ -47,6 +47,18 @@
|
||||
"description": "Optional name to give to the database in OpenMetadata. If left blank, we will use default as the database name.",
|
||||
"type": "string"
|
||||
},
|
||||
"salesforceApiVersion": {
|
||||
"title": "Salesforce API Version",
|
||||
"description": "API version of the Salesforce instance",
|
||||
"type": "string",
|
||||
"default": "42.0"
|
||||
},
|
||||
"salesforceDomain": {
|
||||
"title": "Salesforce Domain",
|
||||
"description": "Domain of Salesforce instance",
|
||||
"type": "string",
|
||||
"default": "login"
|
||||
},
|
||||
"connectionOptions": {
|
||||
"title": "Connection Options",
|
||||
"$ref": "../connectionBasicType.json#/definitions/connectionOptions"
|
||||
|
||||
@ -49,6 +49,23 @@ Database Service > Database > Schema > Table
|
||||
In the case of Salesforce, we won't have a Database as such. If you'd like to see your data in a database named something other than `default`, you can specify the name in this field.
|
||||
$$
|
||||
|
||||
$$section
|
||||
### Salesforce API Version $(id="salesforceApiVersion")
|
||||
|
||||
The version of the Salesforce API to use.
|
||||
|
||||
Follow the steps mentioned [here](https://help.salesforce.com/s/articleView?id=000386929&type=1) to get the API version.
|
||||
Enter the numerical value in the field, For example `42.0`.
|
||||
|
||||
$$
|
||||
|
||||
$$section
|
||||
### Salesforce Domain $(id="salesforceDomain")
|
||||
|
||||
When connecting to Salesforce, you can specify the domain to use for accessing the platform. The common domains include `login` and `test`, and you can also utilize Salesforce My Domain.
|
||||
By default, the domain `login` is used for accessing Salesforce.
|
||||
|
||||
$$
|
||||
|
||||
$$section
|
||||
### Connection Options $(id="connectionOptions")
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user