datahub/docs/api/openapi/openapi-structured-properties.md

# Structured Properties - DataHub OpenAPI v2 Guide

This guides walks through the process of creating and using a Structured Property using the `v2` version
of the DataHub OpenAPI implementation. Note that this refers to DataHub's OpenAPI version and not the version of OpenAPI itself.

Requirements:
* curl
* jq

## Structured Property Definition

Before a structured property can be added to an entity it must first be defined. Here is an example
structured property being created against a local quickstart instance.

### Create Property Definition

Example Request:

```shell
curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/propertyDefinition' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "qualifiedName": "my.test.MyProperty01",
    "displayName": "MyProperty01",
    "valueType": "urn:li:dataType:datahub.string",
    "allowedValues": [
        {
            "value": {"string": "foo"},
            "description": "test foo value"
        },
        {
            "value": {"string": "bar"},
            "description": "test bar value"
        }
    ],
    "cardinality": "SINGLE",
    "entityTypes": [
        "urn:li:entityType:datahub.dataset"
    ],
    "description": "test description"
}' | jq
```

### Read Property Definition

Example Request:

```shell
curl -X 'GET' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/propertyDefinition' \
  -H 'accept: application/json' | jq
```

Example Response:

```json
{
  "value": {
    "allowedValues": [
      {
        "value": {
          "string": "foo"
        },
        "description": "test foo value"
      },
      {
        "value": {
          "string": "bar"
        },
        "description": "test bar value"
      }
    ],
    "qualifiedName": "my.test.MyProperty01",
    "displayName": "MyProperty01",
    "valueType": "urn:li:dataType:datahub.string",
    "description": "test description",
    "entityTypes": [
      "urn:li:entityType:datahub.dataset"
    ],
    "cardinality": "SINGLE"
  }
}
```

### Delete Property Definition

There are two types of deletion present in DataHub: `hard` and `soft` delete. As of the current release only the `soft` delete
is supported for Structured Properties. See the subsections below for more details.

#### Soft Delete

A `soft` deleted Structured Property does not remove any underlying data on the Structured Property entity
or the Structured Property's values written to other entities. The `soft` delete is 100% reversible with zero data loss.
When a Structured Property is `soft` deleted, a few operations are not available.

Structured Property Soft Delete Effects:

* Entities with a `soft` deleted Structured Property value will not return the `soft` deleted properties
* Updates to a `soft` deleted Structured Property's definition are denied
* Adding a `soft` deleted Structured Property's value to an entity is denied
* Search filters using a `soft` deleted Structured Property will be denied

The following command will `soft` delete the test property `MyProperty01` created in this guide by writing
to the `status` aspect.

```shell
curl -X 'POST' \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/status?systemMetadata=false' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
"removed": true
}' | jq
```

Removing the `soft` delete from the Structured Property can be done by either `hard` deleting the `status` aspect or
changing the `removed` boolean to `false.

```shell
curl -X 'POST' \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty01/status?systemMetadata=false' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
"removed": false
}' | jq
```

#### Hard Delete

⚠ **Not Implemented** ⚠

## Applying Structured Properties

Structured Properties can now be added to entities which have the `structuredProperties` as aspect. In the following
example we'll attach and remove properties to an example dataset entity with urn `urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)`.

### Set Structured Property Values

This will set/replace all structured properties on the entity. See `PATCH` operations to add/remove a single property.

```shell
curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "properties": [
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
      "values": [
        {"string": "foo"}
      ]
    }
  ]
}' | jq
```

### Patch Structured Property Value

For this example, we'll extend create a second structured property and apply both properties to the same
dataset used previously. After this your system should include both `my.test.MyProperty01` and `my.test.MyProperty02`.

```shell
curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/structuredProperty/urn%3Ali%3AstructuredProperty%3Amy.test.MyProperty02/propertyDefinition' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
    "qualifiedName": "my.test.MyProperty02",
    "displayName": "MyProperty02",
    "valueType": "urn:li:dataType:datahub.string",
    "allowedValues": [
        {
            "value": {"string": "foo2"},
            "description": "test foo2 value"
        },
        {
            "value": {"string": "bar2"},
            "description": "test bar2 value"
        }
    ],
    "cardinality": "SINGLE",
    "entityTypes": [
        "urn:li:entityType:datahub.dataset"
    ]
}' | jq
```

This command will attach one of each of the two properties to our test dataset `urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)`.

```shell
curl -X 'POST' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "properties": [
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
      "values": [
        {"string": "foo"}
      ]
    },
    {
      "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02",
      "values": [
        {"string": "bar2"}
      ]
    }
  ]
}' | jq
```

#### Remove Structured Property Value

The expected state of our test dataset include 2 structured properties. We'd like to remove the first one and preserve
the second property.

```shell
curl -X 'PATCH' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json-patch+json' \
  -d '{
        "patch": [
            {
                "op": "remove",
                "path": "/properties/urn:li:structuredProperty:my.test.MyProperty01"
            }
        ],
        "arrayPrimaryKeys": {
            "properties": [
                "propertyUrn"
            ]
        }
      }' | jq
```

The response will show that the expected property has been removed.

```json
{
  "urn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
  "aspects": {
    "structuredProperties": {
      "value": {
        "properties": [
          {
            "values": [
              {
                "string": "bar2"
              }
            ],
            "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02"
          }
        ]
      }
    }
  }
}
```

#### Add Structured Property Value

In this example, we'll add the property back with a different value, preserving the existing property.

```shell
curl -X 'PATCH' -v \
  'http://localhost:8080/openapi/v2/entity/dataset/urn%3Ali%3Adataset%3A%28urn%3Ali%3AdataPlatform%3Ahive%2CSampleHiveDataset%2CPROD%29/structuredProperties' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json-patch+json' \
  -d '{
        "patch": [
            {
                "op": "add",
                "path": "/properties/urn:li:structuredProperty:my.test.MyProperty01",
                "value": {
                    "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01",
                    "values": [
                        {
                            "string": "bar"
                        }
                    ]
                }
            }
        ],
        "arrayPrimaryKeys": {
            "properties": [
                "propertyUrn"
            ]
        }
    }' | jq
```

The response shows that the property was re-added with the new value `bar` instead of the previous value `foo`.

```json
{
    "urn": "urn:li:dataset:(urn:li:dataPlatform:hive,SampleHiveDataset,PROD)",
    "aspects": {
        "structuredProperties": {
            "value": {
                "properties": [
                    {
                        "values": [
                            {
                                "string": "bar2"
                            }
                        ],
                        "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty02"
                    },
                    {
                        "values": [
                            {
                                "string": "bar"
                            }
                        ],
                        "propertyUrn": "urn:li:structuredProperty:my.test.MyProperty01"
                    }
                ]
            }
        }
    }
}
```