yujunjun/openspg
1
0
Fork 0
mirror of https://github.com/OpenSPG/openspg.git synced 2025-12-27 15:14:19 +00:00
Code Issues Packages Projects Releases Wiki Activity

merge master

Browse Source
This commit is contained in:
baifuyu 2023-12-11 15:34:02 +08:00
commit 7b67e48ad6
53 changed files with 282 additions and 212 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.github/workflows/pr-title-checker.yml vendored
View File

@ -12,6 +12,7 @@ jobs:
name: Validate PR title
runs-on: ubuntu-latest
steps:
# https://www.conventionalcommits.org/en/v1.0.0/#summary
- uses: amannn/action-semantic-pull-request@v5
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

4
README.md
View File

@ -4,7 +4,7 @@
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](./LICENSE)
OpenSPG is a knowledge graph engine developed by Ant Group in collaboration with OpenKG, based on the SPG (Semantic-enhanced Programmable Graph) framework, which is a summary of Ant Group's years of experience in constructing and applying diverse domain knowledge graphs in the financial scenarios.
[OpenSPG](https://spg.openkg.cn/en-US) is a knowledge graph engine developed by Ant Group in collaboration with OpenKG, based on the SPG (Semantic-enhanced Programmable Graph) framework, which is a summary of Ant Group's years of experience in constructing and applying diverse domain knowledge graphs in the financial scenarios.
![OpenSPG Architecture](https://mdn.alipayobjects.com/huamei_xgb3qj/afts/img/A*DmdvRKo5pyYAAAAAAAAAAAAADtmcAQ/original)
@ -13,7 +13,7 @@ OpenSPG is a knowledge graph engine developed by Ant Group in collaboration with
SPG (Semantic-enhanced Programmable Graph): semantic-enhanced programmable framework is a set of semantic representation framework based on property graph precipitated by Ant Knowledge Graph platform after years of supporting business in the financial field. It creatively integrates LPG structural and RDF semantic, which overcomes the problem that RDF/OWL semantic complexity cannot be industrially landed, and fully inherits the advantages of LPG structural simplicity and compatibility with big data system. The framework defines and represents knowledge semantics from three aspects. First, SPG explicitly defines the formal representation and programmable framework of "knowledge", so that it can be defined, programmed, understood and processed by machines. Secondly, SPG achieves compatibility and progressive advancement between knowledge levels, supporting the construction of knowledge graphs and the continuous iterative evolution of incomplete data states in industrial-level scenarios. Finally, SPG serves as an effective bridge between big data and AI technology systems, facilitating the efficient transformation of massive data into knowledge-based insights. By doing so, it enhances the value and application potential of the data. With the SPG framework, we can construct and manage graph data more efficiently, and at the same time, we can better support business requirements and application scenarios. Since SPG framework has good scalability and flexibility, new business scenarios can quickly build their domain models and solutions by extending the domain knowledge model and developing new
operators.
For a detailed introduction to SPG, please refer to **the [《SPG White Paper》](https://spg.openkg.cn/ "SPG White Paper") jointly released by Ant Group and OpenKG**.
For a detailed introduction to SPG, please refer to **the [《SPG White Paper》](https://spg.openkg.cn/en-US "SPG White Paper") jointly released by Ant Group and OpenKG**.
# OpenSPG

2
README_cn.md
View File

@ -2,7 +2,7 @@
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](./LICENSE)
OpenSPG 是蚂蚁集团结合多年金融领域多元场景知识图谱构建与应用业务经验的总结，并与OpenKG联合推出的基于SPG(Semantic-enhanced Programmable Graph)框架研发的知识图谱引擎。
(OpenSPG)[https://spg.openkg.cn] 是蚂蚁集团结合多年金融领域多元场景知识图谱构建与应用业务经验的总结，并与OpenKG联合推出的基于SPG(Semantic-enhanced Programmable Graph)框架研发的知识图谱引擎。
![OpenSPG Architecture](https://mdn.alipayobjects.com/huamei_xgb3qj/afts/img/A*YYSpQoItezMAAAAAAAAAAAAADtmcAQ/original)

12
builder/core/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.builder</groupId>
<artifactId>openspg-builder-parent</artifactId>
<artifactId>builder-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
@ -28,23 +28,23 @@
<artifactId>api-http-client</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-graph-store</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-graph-store-tugraph</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-table-store</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-search-engine</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-search-engine-elasticsearch</artifactId>
</dependency>
<dependency>

2
builder/model/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.builder</groupId>
<artifactId>openspg-builder-parent</artifactId>
<artifactId>builder-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>

2
builder/pom.xml
View File

@ -23,7 +23,7 @@
</parent>
<groupId>com.antgroup.openspg.builder</groupId>
<artifactId>openspg-builder-parent</artifactId>
<artifactId>builder-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<licenses>

4
builder/runner/local/pom.xml
View File

@ -1,10 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.builder</groupId>
<artifactId>openspg-builder-parent</artifactId>
<artifactId>builder-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

8
cloudext/impl/graph-store/tugraph/pom.xml
View File

@ -15,17 +15,17 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-impl-graph-store-tugraph</artifactId>
<dependencies>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-graph-store</artifactId>
</dependency>
<dependency>

8
cloudext/impl/object-store/local/pom.xml
View File

@ -15,17 +15,17 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-impl-object-store-local</artifactId>
<dependencies>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-object-store</artifactId>
</dependency>
<dependency>

8
cloudext/impl/search-engine/elasticsearch/pom.xml
View File

@ -15,17 +15,17 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-impl-search-engine-elasticsearch</artifactId>
<dependencies>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-search-engine</artifactId>
</dependency>
<dependency>

8
cloudext/impl/table-store/local/pom.xml
View File

@ -15,17 +15,17 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
<relativePath>../../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-impl-table-store-local</artifactId>
<dependencies>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-table-store</artifactId>
</dependency>
<dependency>

6
cloudext/interface/graph-store/pom.xml
View File

@ -15,10 +15,10 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-interface-graph-store</artifactId>

6
cloudext/interface/object-store/pom.xml
View File

@ -15,10 +15,10 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-interface-object-store</artifactId>

6
cloudext/interface/search-engine/pom.xml
View File

@ -15,10 +15,10 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-interface-search-engine</artifactId>

6
cloudext/interface/table-store/pom.xml
View File

@ -15,10 +15,10 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
<relativePath>../../pom.xml</relativePath>
</parent>
<artifactId>cloudext-interface-table-store</artifactId>

53
cloudext/pom.xml Normal file
View File

@ -0,0 +1,53 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Copyright 2023 Ant Group CO., Ltd.
~
~ Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
~ in compliance with the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software distributed under the License
~ is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
~ or implied.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg</groupId>
<artifactId>openspg-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../pom.xml</relativePath>
</parent>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<licenses>
<license>
<name>Apache 2.0 License</name>
<url>http://www.apache.org/licenses/LICENSE-2.0.html</url>
<distribution>repo</distribution>
</license>
</licenses>
<modules>
<module>interface/graph-store</module>
<module>interface/object-store</module>
<module>interface/table-store</module>
<module>interface/search-engine</module>
<module>impl/graph-store/tugraph</module>
<module>impl/object-store/local</module>
<module>impl/table-store/local</module>
<module>impl/search-engine/elasticsearch</module>
</modules>
<properties/>
<dependencyManagement>
<dependencies/>
</dependencyManagement>
</project>

28
pom.xml
View File

@ -30,17 +30,9 @@
<modules>
<module>common/util</module>
<module>server</module>
<module>builder</module>
<module>reasoner</module>
<module>cloudext/interface/graph-store</module>
<module>cloudext/interface/object-store</module>
<module>cloudext/interface/table-store</module>
<module>cloudext/interface/search-engine</module>
<module>cloudext/impl/graph-store/tugraph</module>
<module>cloudext/impl/object-store/local</module>
<module>cloudext/impl/table-store/local</module>
<module>cloudext/impl/search-engine/elasticsearch</module>
<module>builder</module>
<module>cloudext</module>
</modules>
<properties>
@ -96,42 +88,42 @@
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-graph-store</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-search-engine</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-object-store</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-table-store</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-graph-store-tugraph</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-object-store-local</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-table-store-local</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-search-engine-elasticsearch</artifactId>
<version>${project.version}</version>
</dependency>

3
python/knext/api/client.py
View File

@ -1,4 +1,3 @@
from knext.client.builder import BuilderClient
from knext.client.schema import SchemaClient
from knext.client.reasoner import ReasonerClient
@ -10,5 +9,5 @@ __all__ = [
"SchemaClient",
"ReasonerClient",
"OperatorClient",
"SearchClient"
"SearchClient",
]

3
python/knext/api/component.py
View File

@ -1,4 +1,3 @@
from knext.component.builder import UserDefinedExtractor, LLMBasedExtractor
from knext.component.builder import SPGTypeMapping, RelationMapping
from knext.component.builder import CsvSourceReader
@ -13,5 +12,5 @@ __all__ = [
"SPGTypeMapping",
"RelationMapping",
"KGSinkWriter",
"Component"
"Component",
]

34
python/knext/chain/base.py
View File

@ -17,14 +17,18 @@ class Chain(Runnable, RESTable):
def to_rest(self):
pass
def __rshift__(self, other: Union[
Type['Chain'],
List[Type['Chain']],
Type['Component'],
List[Type['Component']],
None
]):
def __rshift__(
self,
other: Union[
Type["Chain"],
List[Type["Chain"]],
Type["Component"],
List[Type["Component"]],
None,
],
):
from knext.component.base import Component
if not other:
return self
if not isinstance(other, list):
@ -34,7 +38,11 @@ class Chain(Runnable, RESTable):
if not o:
dag_list.append(o.dag)
if isinstance(o, Component):
end_nodes = [node for node, out_degree in self.dag.out_degree() if out_degree == 0 or node.last]
end_nodes = [
node
for node, out_degree in self.dag.out_degree()
if out_degree == 0 or node.last
]
dag = nx.DiGraph(self.dag)
if len(end_nodes) > 0:
for end_node in end_nodes:
@ -43,8 +51,14 @@ class Chain(Runnable, RESTable):
dag_list.append(dag)
elif isinstance(o, Chain):
combined_dag = nx.compose(self.dag, o.dag)
end_nodes = [node for node, out_degree in self.dag.out_degree() if out_degree == 0 or node.last]
start_nodes = [node for node, in_degree in o.dag.in_degree() if in_degree == 0]
end_nodes = [
node
for node, out_degree in self.dag.out_degree()
if out_degree == 0 or node.last
]
start_nodes = [
node for node, in_degree in o.dag.in_degree() if in_degree == 0
]
if len(end_nodes) > 0 and len(start_nodes) > 0:
for end_node in end_nodes:

5
python/knext/chain/builder_chain.py
View File

@ -12,7 +12,10 @@ class BuilderChain(RESTable, Chain):
source: SourceReader
process: Union[SPGExtractor, Mapping, ]
process: Union[
SPGExtractor,
Mapping,
]
sink: SinkWriter

2
python/knext/client/base.py
View File

@ -2,4 +2,4 @@ from abc import ABC
class Client(ABC):
pass
pass

5
python/knext/common/restable.py
View File

@ -5,13 +5,12 @@ from knext import rest
class RESTable(ABC):
@property
def upstream_types(self) -> Type['RESTable']:
def upstream_types(self) -> Type["RESTable"]:
raise NotImplementedError("To be implemented in subclass")
@property
def downstream_types(self) -> Type['RESTable']:
def downstream_types(self) -> Type["RESTable"]:
raise NotImplementedError("To be implemented in subclass")
def to_rest(self) -> rest.Node:

3
python/knext/common/schema_helper.py
View File

@ -10,13 +10,10 @@ class SchemaHelper(ABC):
class SPGTypeHelper(SchemaHelper):
def __init__(self, type_name: str):
super().__init__(type_name)
class PropertyHelper(SchemaHelper):
def __init__(self, type_name: str):
super().__init__(type_name)

28
python/knext/component/base.py
View File

@ -54,14 +54,18 @@ class Component(Runnable, RESTable, ABC):
def __eq__(self, other):
return hash(self) == hash(other)
def __rshift__(self, other: Union[
Type['Chain'],
List[Type['Chain']],
Type['Component'],
List[Type['Component']],
None
]):
def __rshift__(
self,
other: Union[
Type["Chain"],
List[Type["Chain"]],
Type["Component"],
List[Type["Component"]],
None,
],
):
from knext.chain.base import Chain
if not other:
return self
if not isinstance(other, list):
@ -83,8 +87,14 @@ class Component(Runnable, RESTable, ABC):
elif isinstance(o, Chain):
dag = nx.DiGraph()
dag.add_node(self)
end_nodes = [node for node, out_degree in dag.out_degree() if out_degree == 0 or node.last]
start_nodes = [node for node, in_degree in o.dag.in_degree() if in_degree == 0]
end_nodes = [
node
for node, out_degree in dag.out_degree()
if out_degree == 0 or node.last
]
start_nodes = [
node for node, in_degree in o.dag.in_degree() if in_degree == 0
]
if len(end_nodes) > 0 and len(start_nodes) > 0:
for end_node in end_nodes:

7
python/knext/component/builder/__init__.py
View File

@ -1,5 +1,8 @@
from knext.component.builder.extractor import UserDefinedExtractor, LLMBasedExtractor, SPGExtractor
from knext.component.builder.extractor import (
UserDefinedExtractor,
LLMBasedExtractor,
SPGExtractor,
)
from knext.component.builder.mapping import SPGTypeMapping, RelationMapping, Mapping
from knext.component.builder.source_reader import CsvSourceReader, SourceReader
from knext.component.builder.sink_writer import KGSinkWriter, SinkWriter

5
python/knext/component/builder/base.py
View File

@ -14,7 +14,6 @@ class ComponentLabelEnum(str, Enum):
class BuilderComponent(Component, ABC):
@property
def type(self):
return ComponentTypeEnum.Builder
@ -29,7 +28,6 @@ class BuilderComponent(Component, ABC):
class SourceReader(BuilderComponent, ABC):
@property
def upstream_types(self):
return None
@ -44,7 +42,6 @@ class SourceReader(BuilderComponent, ABC):
class SPGExtractor(BuilderComponent, ABC):
@property
def upstream_types(self):
return Union[SourceReader, SPGExtractor]
@ -59,7 +56,6 @@ class SPGExtractor(BuilderComponent, ABC):
class Mapping(BuilderComponent, ABC):
@property
def upstream_types(self):
return Union[SourceReader, SPGExtractor]
@ -74,7 +70,6 @@ class Mapping(BuilderComponent, ABC):
class SinkWriter(BuilderComponent, ABC):
@property
def upstream_types(self):
return Union[Mapping]

18
python/knext/component/builder/mapping.py
View File

@ -12,7 +12,6 @@ from knext.operator.op import LinkOp
from knext.operator.spg_record import SPGRecord
class MappingTypeEnum(str, Enum):
SPGType = "SPG_TYPE"
Relation = "RELATION"
@ -67,7 +66,12 @@ class SPGTypeMapping(Mapping):
def output_keys(self):
return self.output_fields
def add_field(self, source_field: str, target_field: Union[str, PropertyHelper], link_strategy: Union[LinkStrategyEnum ,LinkOp] = None):
def add_field(
self,
source_field: str,
target_field: Union[str, PropertyHelper],
link_strategy: Union[LinkStrategyEnum, LinkOp] = None,
):
"""Adds a field mapping from source data to property of spg_type.
:param source_field: The source field to be mapped.
@ -94,7 +98,7 @@ class SPGTypeMapping(Mapping):
Transforms `EntityMappingComponent` to REST model `MappingNodeConfig`.
"""
schema = {}
#TODO generate schema with link_strategy
# TODO generate schema with link_strategy
mapping_filters = [
rest.MappingFilter(column_name=name, column_value=value)
@ -236,7 +240,12 @@ class SubGraphMapping(Mapping):
def output_keys(self):
return self.output_fields
def add_field(self, source_field: str, target_field: Union[str, PropertyHelper], link_strategy: Union[LinkStrategyEnum ,LinkOp] = None):
def add_field(
self,
source_field: str,
target_field: Union[str, PropertyHelper],
link_strategy: Union[LinkStrategyEnum, LinkOp] = None,
):
"""Adds a field mapping from source data to property of spg_type.
:param source_field: The source field to be mapped.
@ -282,4 +291,3 @@ class SubGraphMapping(Mapping):
def submit(self):
pass

4
python/knext/examples/medical/builder/job/hospital_department.py
View File

@ -27,9 +27,7 @@ class HospitalDepartment(BuilderJob):
start_row=1,
)
mapping = EntityMappingComponent(
spg_type_name=Medical.HospitalDepartment
)
mapping = EntityMappingComponent(spg_type_name=Medical.HospitalDepartment)
sink = SinkToKgComponent()

144
python/knext/examples/riskmining/schema/riskmining_schema_helper.py
View File

@ -3,90 +3,90 @@
# PLEASE DO NOT MODIFY THIS FILE!!!
#
class RiskMining:
def __init__(self):
self.App = self.App()
self.Cert = self.Cert()
self.Company = self.Company()
self.Device = self.Device()
self.Person = self.Person()
self.TaxOfRiskApp = self.TaxOfRiskApp()
self.TaxOfRiskUser = self.TaxOfRiskUser()
def __init__(self):
self.App = self.App()
self.Cert = self.Cert()
self.Company = self.Company()
self.Device = self.Device()
self.Person = self.Person()
self.TaxOfRiskApp = self.TaxOfRiskApp()
self.TaxOfRiskUser = self.TaxOfRiskUser()
class App:
__typename__ = "RiskMining.App"
description = "description"
id = "id"
name = "name"
belongTo = "belongTo"
useCert = "useCert"
riskMark = "riskMark"
class App:
__typename__ = "RiskMining.App"
description = "description"
id = "id"
name = "name"
belongTo = "belongTo"
useCert = "useCert"
riskMark = "riskMark"
def __init__(self):
pass
def __init__(self):
pass
class Cert:
__typename__ = "RiskMining.Cert"
description = "description"
id = "id"
name = "name"
certNum = "certNum"
class Cert:
__typename__ = "RiskMining.Cert"
description = "description"
id = "id"
name = "name"
certNum = "certNum"
def __init__(self):
pass
def __init__(self):
pass
class Company:
__typename__ = "RiskMining.Company"
description = "description"
id = "id"
name = "name"
hasPhone = "hasPhone"
class Company:
__typename__ = "RiskMining.Company"
description = "description"
id = "id"
name = "name"
hasPhone = "hasPhone"
def __init__(self):
pass
def __init__(self):
pass
class Device:
__typename__ = "RiskMining.Device"
description = "description"
id = "id"
name = "name"
install = "install"
umid = "umid"
class Device:
__typename__ = "RiskMining.Device"
description = "description"
id = "id"
name = "name"
install = "install"
umid = "umid"
def __init__(self):
pass
def __init__(self):
pass
class Person:
__typename__ = "RiskMining.Person"
description = "description"
id = "id"
name = "name"
hasPhone = "hasPhone"
belongTo = "belongTo"
age = "age"
class Person:
__typename__ = "RiskMining.Person"
description = "description"
id = "id"
name = "name"
hasPhone = "hasPhone"
belongTo = "belongTo"
age = "age"
def __init__(self):
pass
def __init__(self):
pass
class TaxOfRiskApp:
__typename__ = "RiskMining.TaxOfRiskApp"
description = "description"
id = "id"
name = "name"
alias = "alias"
stdId = "stdId"
class TaxOfRiskApp:
__typename__ = "RiskMining.TaxOfRiskApp"
description = "description"
id = "id"
name = "name"
alias = "alias"
stdId = "stdId"
def __init__(self):
pass
def __init__(self):
pass
class TaxOfRiskUser:
__typename__ = "RiskMining.TaxOfRiskUser"
description = "description"
id = "id"
name = "name"
alias = "alias"
stdId = "stdId"
def __init__(self):
pass
class TaxOfRiskUser:
__typename__ = "RiskMining.TaxOfRiskUser"
description = "description"
id = "id"
name = "name"
alias = "alias"
stdId = "stdId"
def __init__(self):
pass

1
python/knext/operator/builtin/online_runner.py
View File

@ -6,7 +6,6 @@ from NN4K.invoker.base import ModelInvoker
class BuiltInOnlineLLMBasedExtractOp(ExtractOp):
def __init__(self, params: Dict[str, str] = None):
"""

4
python/knext/operator/op.py
View File

@ -99,7 +99,9 @@ class PromptOp(ExtractOp, ABC):
f"{self.__class__.__name__} need to implement `build_prompt` method."
)
def parse_response(self, response: str) -> Union[List[Dict[str, str]], List[SPGRecord]]:
def parse_response(
self, response: str
) -> Union[List[Dict[str, str]], List[SPGRecord]]:
raise NotImplementedError(
f"{self.__class__.__name__} need to implement `parse_response` method."
)

4
python/knext/operator/spg_record.py
View File

@ -17,9 +17,7 @@ from typing import Dict, Any, List
class SPGRecord:
"""Data structure in operator, used to store entity information."""
def __init__(
self, spg_type_name: str = "", properties: Dict[str, str] = None
):
def __init__(self, spg_type_name: str = "", properties: Dict[str, str] = None):
self._spg_type_name = None
self._properties = None

2
server/api/facade/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/api/http-client/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/api/http-server/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/arks/sofaboot/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/biz/common/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/biz/reasoner/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/biz/schema/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

4
server/cloudext/impl/computing/local/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
</parent>
@ -29,7 +29,7 @@
<artifactId>cloudext-interface-computing</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-table-store</artifactId>
</dependency>
<dependency>

2
server/cloudext/impl/job-scheduler/local/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../../pom.xml</relativePath>
</parent>

2
server/cloudext/interface/computing/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

2
server/cloudext/interface/job-scheduler/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

2
server/common/model/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

10
server/common/service/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>
@ -33,15 +33,15 @@
<artifactId>api-http-client</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-graph-store</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-object-store</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-search-engine</artifactId>
</dependency>
<dependency>
@ -53,7 +53,7 @@
<artifactId>cloudext-interface-computing</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-interface-table-store</artifactId>
</dependency>
<dependency>

2
server/core/reasoner/model/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

2
server/core/reasoner/service/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

2
server/core/schema/model/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>

10
server/core/schema/service/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../../pom.xml</relativePath>
</parent>
@ -33,19 +33,19 @@
<artifactId>common-service</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-graph-store-tugraph</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-object-store-local</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-table-store-local</artifactId>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-search-engine-elasticsearch</artifactId>
</dependency>
<dependency>

2
server/infra/dao/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../../pom.xml</relativePath>
</parent>

2
server/pom.xml
View File

@ -23,7 +23,7 @@
</parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>pom</packaging>
<licenses>

4
server/test/pom.xml
View File

@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>com.antgroup.openspg.server</groupId>
<artifactId>openspg-server-parent</artifactId>
<artifactId>server-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
</parent>
@ -40,7 +40,7 @@
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.antgroup.openspg</groupId>
<groupId>com.antgroup.openspg.cloudext</groupId>
<artifactId>cloudext-impl-graph-store-tugraph</artifactId>
<scope>test</scope>
</dependency>