mirror of https://github.com/open-metadata/OpenMetadata.git synced 2025-07-09 18:21:23 +00:00

Pere Miquel Brull 222a8f8984

[Docs] - SSO updates & Connectors workflow config (#12241 )

* Rename docs and clean SSO

* Add connector partials

* Add connector partials

* Rename path

2023-06-30 12:25:11 +02:00

1.4 KiB

Raw Blame History

title	slug
Bulk Sink	/sdk/python/build-connector/bulk-sink

BulkSink

BulkSink is an optional component in the workflow. It can be used to bulk update the records generated in a workflow. It needs to be used in conjunction with Stage

API

@dataclass  # type: ignore[misc]
class BulkSink(Closeable, metaclass=ABCMeta):
ctx: WorkflowContext

    @classmethod
    @abstractmethod
    def create(cls, config_dict: dict, metadata_config_dict: dict, ctx: WorkflowContext) -> "BulkSink":
        pass

    @abstractmethod
    def write_records(self) -> None:
        pass

    @abstractmethod
    def get_status(self) -> BulkSinkStatus:
        pass

    @abstractmethod
    def close(self) -> None:
        pass

create method is called during the workflow instantiation and creates an instance of the bulk sink.

write_records this method is called only once in Workflow. Its developer responsibility is to make bulk actions inside this method. Such as read the entire file or store to generate the API calls to external services.

get_status to report the status of the bulk_sink ex: how many records, failures or warnings etc.

close gets called before the workflow stops. Can be used to clean up any connections or other resources.

Example

Example implementation

1.4 KiB Raw Blame History

BulkSink

API

Example

1.4 KiB

Raw Blame History