refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
# coding: utf-8
|
|
|
|
# Copyright 2023 OpenSPG Authors
|
|
|
|
#
|
|
|
|
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
|
|
|
|
# in compliance with the License. You may obtain a copy of the License at
|
|
|
|
#
|
|
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
#
|
|
|
|
# Unless required by applicable law or agreed to in writing, software distributed under the License
|
|
|
|
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
|
|
|
|
# or implied.
|
|
|
|
|
|
|
|
|
|
|
|
"""
|
|
|
|
knext
|
|
|
|
|
|
|
|
No description provided (generated by Openapi Generator https://github.com/openapitools/openapi-generator) # noqa: E501
|
|
|
|
|
|
|
|
The version of the OpenAPI document: 1.0.0
|
|
|
|
Generated by: https://openapi-generator.tech
|
|
|
|
"""
|
|
|
|
|
|
|
|
|
|
|
|
import pprint
|
|
|
|
import re # noqa: F401
|
|
|
|
|
|
|
|
import six
|
|
|
|
|
|
|
|
from knext.common.rest.configuration import Configuration
|
|
|
|
|
|
|
|
|
|
|
|
class WriterGraphRequest(object):
|
|
|
|
"""NOTE: This class is auto generated by OpenAPI Generator.
|
|
|
|
Ref: https://openapi-generator.tech
|
|
|
|
|
|
|
|
Do not edit the class manually.
|
|
|
|
"""
|
|
|
|
|
|
|
|
"""
|
|
|
|
Attributes:
|
|
|
|
openapi_types (dict): The key is attribute name
|
|
|
|
and the value is attribute type.
|
|
|
|
attribute_map (dict): The key is attribute name
|
|
|
|
and the value is json key in definition.
|
|
|
|
"""
|
|
|
|
openapi_types = {
|
|
|
|
"project_id": "int",
|
|
|
|
"operation": "str",
|
|
|
|
"sub_graph": "object",
|
|
|
|
"enable_lead_to": "bool",
|
2025-03-12 10:13:03 +08:00
|
|
|
"token": "str"
|
refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
attribute_map = {
|
|
|
|
"project_id": "projectId",
|
|
|
|
"operation": "operation",
|
|
|
|
"sub_graph": "subGraph",
|
|
|
|
"enable_lead_to": "enableLeadTo",
|
2025-03-12 10:13:03 +08:00
|
|
|
"token": "token"
|
refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
}
|
|
|
|
|
|
|
|
def __init__(
|
2025-03-12 10:13:03 +08:00
|
|
|
self,
|
|
|
|
project_id=None,
|
|
|
|
operation=None,
|
|
|
|
sub_graph=None,
|
|
|
|
enable_lead_to=None,
|
|
|
|
token=None,
|
|
|
|
local_vars_configuration=None,
|
refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
): # noqa: E501
|
|
|
|
"""WriterGraphRequest - a model defined in OpenAPI""" # noqa: E501
|
|
|
|
if local_vars_configuration is None:
|
|
|
|
local_vars_configuration = Configuration()
|
|
|
|
self.local_vars_configuration = local_vars_configuration
|
|
|
|
|
|
|
|
self._project_id = None
|
|
|
|
self._operation = None
|
|
|
|
self._sub_graph = None
|
|
|
|
self._enable_lead_to = None
|
2025-03-12 10:13:03 +08:00
|
|
|
self._token = None
|
refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
self.discriminator = None
|
|
|
|
|
|
|
|
if project_id is not None:
|
|
|
|
self.project_id = project_id
|
|
|
|
if operation is not None:
|
|
|
|
self.operation = operation
|
|
|
|
if sub_graph is not None:
|
|
|
|
self.sub_graph = sub_graph
|
|
|
|
if enable_lead_to is not None:
|
|
|
|
self.enable_lead_to = enable_lead_to
|
2025-03-12 10:13:03 +08:00
|
|
|
if token is not None:
|
|
|
|
self.token = token
|
refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
|
|
|
|
@property
|
|
|
|
def project_id(self):
|
|
|
|
"""Gets the project_id of this WriterGraphRequest. # noqa: E501
|
|
|
|
|
|
|
|
|
|
|
|
:return: The project_id of this WriterGraphRequest. # noqa: E501
|
|
|
|
:rtype: int
|
|
|
|
"""
|
|
|
|
return self._project_id
|
|
|
|
|
|
|
|
@project_id.setter
|
|
|
|
def project_id(self, project_id):
|
|
|
|
"""Sets the project_id of this WriterGraphRequest.
|
|
|
|
|
|
|
|
|
|
|
|
:param project_id: The project_id of this WriterGraphRequest. # noqa: E501
|
|
|
|
:type: int
|
|
|
|
"""
|
|
|
|
|
|
|
|
self._project_id = project_id
|
|
|
|
|
|
|
|
@property
|
|
|
|
def operation(self):
|
|
|
|
"""Gets the operation of this WriterGraphRequest. # noqa: E501
|
|
|
|
|
|
|
|
|
|
|
|
:return: The operation of this WriterGraphRequest. # noqa: E501
|
|
|
|
:rtype: str
|
|
|
|
"""
|
|
|
|
return self._operation
|
|
|
|
|
|
|
|
@operation.setter
|
|
|
|
def operation(self, operation):
|
|
|
|
"""Sets the operation of this WriterGraphRequest.
|
|
|
|
|
|
|
|
|
|
|
|
:param operation: The operation of this WriterGraphRequest. # noqa: E501
|
|
|
|
:type: str
|
|
|
|
"""
|
|
|
|
|
|
|
|
self._operation = operation
|
|
|
|
|
|
|
|
@property
|
|
|
|
def sub_graph(self):
|
|
|
|
"""Gets the sub_graph of this WriterGraphRequest. # noqa: E501
|
|
|
|
|
|
|
|
|
|
|
|
:return: The sub_graph of this WriterGraphRequest. # noqa: E501
|
|
|
|
:rtype: object
|
|
|
|
"""
|
|
|
|
return self._sub_graph
|
|
|
|
|
|
|
|
@sub_graph.setter
|
|
|
|
def sub_graph(self, sub_graph):
|
|
|
|
"""Sets the sub_graph of this WriterGraphRequest.
|
|
|
|
|
|
|
|
|
|
|
|
:param sub_graph: The sub_graph of this WriterGraphRequest. # noqa: E501
|
|
|
|
:type: object
|
|
|
|
"""
|
|
|
|
|
|
|
|
self._sub_graph = sub_graph
|
|
|
|
|
|
|
|
@property
|
|
|
|
def enable_lead_to(self):
|
|
|
|
"""Gets the enable_lead_to of this WriterGraphRequest. # noqa: E501
|
|
|
|
|
|
|
|
|
|
|
|
:return: The enable_lead_to of this WriterGraphRequest. # noqa: E501
|
|
|
|
:rtype: bool
|
|
|
|
"""
|
|
|
|
return self._enable_lead_to
|
|
|
|
|
|
|
|
@enable_lead_to.setter
|
|
|
|
def enable_lead_to(self, enable_lead_to):
|
|
|
|
"""Sets the enable_lead_to of this WriterGraphRequest.
|
|
|
|
|
|
|
|
|
|
|
|
:param enable_lead_to: The enable_lead_to of this WriterGraphRequest. # noqa: E501
|
|
|
|
:type: bool
|
|
|
|
"""
|
|
|
|
|
|
|
|
self._enable_lead_to = enable_lead_to
|
|
|
|
|
2025-03-12 10:13:03 +08:00
|
|
|
@property
|
|
|
|
def token(self):
|
|
|
|
"""Gets the token of this WriterGraphRequest. # noqa: E501
|
|
|
|
|
|
|
|
|
|
|
|
:return: The token of this WriterGraphRequest. # noqa: E501
|
|
|
|
:rtype: str
|
|
|
|
"""
|
|
|
|
return self._token
|
|
|
|
|
|
|
|
@token.setter
|
|
|
|
def token(self, token):
|
|
|
|
"""Sets the token of this WriterGraphRequest.
|
|
|
|
|
|
|
|
|
|
|
|
:param token: The token of this WriterGraphRequest. # noqa: E501
|
|
|
|
:type: str
|
|
|
|
"""
|
|
|
|
|
|
|
|
self._token = token
|
|
|
|
|
refactor(all): kag v0.6 (#174)
* add path find
* fix find path
* spg guided relation extraction
* fix dict parse with same key
* rename graphalgoclient to graphclient
* rename graphalgoclient to graphclient
* file reader supports http url
* add checkpointer class
* parser supports checkpoint
* add build
* remove incorrect logs
* remove logs
* update examples
* update chain checkpointer
* vectorizer batch size set to 32
* add a zodb backended checkpointer
* add a zodb backended checkpointer
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* 增加solver
* add kag
* update outline splitter
* add main test
* add op
* code refactor
* add tools
* fix outline splitter
* fix outline prompt
* graph api pass
* commit with page rank
* add search api and graph api
* add markdown report
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* update markdown reader
* update pdf reader
* raise extractor failure
* add default expr
* add log
* merge jc reader features
* rm import
* add build
* fix zodb based checkpointer
* add thread for zodb IO
* fix(common): resolve mutlithread conflict in zodb IO
* fix(common): load existing zodb checkpoints
* update examples
* update examples
* fix zodb writer
* add docstring
* fix jieba version mismatch
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* commit kag_config-tc.yaml
1、rename type to register_name
2、put a uniqe & specific name to register_name
3、rename reader to scanner
4、rename parser to reader
5、rename num_parallel to num_parallel_file, rename chain_level_num_paralle to num_parallel_chain_of_file
6、rename kag_extractor to schema_free_extractor, schema_base_extractor to schema_constraint_extractor
7、pre-define llm & vectorize_model and refer them in the yaml file
Issues to be resolved:
1、examples of event extract & spg extract
2、statistic of indexer, such as nums of nodes & edges extracted, ratio of llm invoke.
3、Exceptions such as Debt, account does not exist should be thrown in llm invoke.
4、conf of solver need to be re-examined.
* 1、fix bug in base_table_splitter
* 1、fix bug in base_table_splitter
* 1、fix bug in default_chain
* update outline splitter
* add main test
* add markdown report
* code refactor
* fix outline splitter
* fix outline prompt
* update markdown reader
* fix vectorizer num batch compute
* add retry for vectorize model call
* update markdown reader
* raise extractor failure
* rm parser
* run pipeline
* add config option of whether to perform llm config check, default to false
* fix
* recover pdf reader
* several components can be null for default chain
* 支持完整qa运行
* add if
* remove unused code
* 使用chunk兜底
* excluded source relation to choose
* add generate
* default recall 10
* add local memory
* 排除相似边
* 增加保护
* 修复并发问题
* add debug logger
* 支持topk参数化
* 支持chunk截断和调整spo select 的prompt
* 增加查询请求保护
* 增加force_chunk配置
* fix entity linker algorithm
* 增加sub query改写
* fix md reader dup in test
* fix
* merge knext to kag parallel
* fix package
* 修复指标下跌问题
* scanner update
* scanner update
* add doc and update example scripts
* fix
* add bridge to spg server
* add format
* fix bridge
* update conf for baike
* disable ckpt for spg server runner
* llm invoke error default raise exceptions
* chore(version): bump version to X.Y.Z
* update default response generation prompt
* add method getSummarizationMetrics
* fix(common): fix project conf empty error
* fix typo
* 增加上报信息
* 修改main solver
* postprocessor support spg server
* 修改solver支持名
* fix language
* 修改chunker接口,增加openapi
* rename vectorizer to vectorize_model in spg server config
* generate_random_string start with gen
* add knext llm vector checker
* add knext llm vector checker
* add knext llm vector checker
* solver移除默认值
* udpate yaml and register_name for baike
* udpate yaml and register_name for baike
* remove config key check
* 修复llmmodule
* fix knext project
* udpate yaml and register_name for examples
* udpate yaml and register_name for examples
* Revert "udpate yaml and register_name for examples"
This reverts commit b3fa5ca9ba749e501133ac67bd8746027ab839d9.
* update register name
* fix
* fix
* support multiple resigter names
* update component
* update reader register names (#183)
* fix markdown reader
* fix llm client for retry
* feat(common): add processed chunk id checkpoint (#185)
* update reader register names
* add processed chunk id checkpoint
* feat(example): add example config (#186)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* add max_workers parameter for getSummarizationMetrics to make it faster
* add csqa data generation script generate_data.py
* commit generated csqa builder and solver data
* add csqa basic project files
* adjust split_length and num_threads_per_chain to match lightrag settings
* ignore ckpt dirs
* add csqa evaluation script eval.py
* save evaluation scripts summarization_metrics.py and factual_correctness.py
* save LightRAG output csqa_lightrag_answers.json
* ignore KAG output csqa_kag_answers.json
* add README.md for CSQA
* fix(solver): fix solver pipeline conf (#191)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* update links and file paths
* reformat csqa kag_config.yaml
* reformat csqa python files
* reformat getSummarizationMetrics and compare_summarization_answers
* fix(solver): fix solver config (#192)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* add except
* fix typo in csqa README.md
* feat(conf): support reinitialize config for call from java side (#199)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* revert default response generation prompt
* update project list
* add README.md for the hotpotqa, 2wiki and musique examples
* 增加spo检索
* turn off kag config dump by default
* turn off knext schema dump by default
* add .gitignore and fix kag_config.yaml
* add README.md for the medicine example
* add README.md for the supplychain example
* bugfix for risk mining
* use exact out
* refactor(solver): format solver code (#205)
* update reader register names
* add processed chunk id checkpoint
* add example config file
* update solver pipeline config
* fix project create
* fix main solver conf
* support reinitialize config for java call
* black format
---------
Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
Co-authored-by: huaidong.xhd <huaidong.xhd@antgroup.com>
2025-01-03 17:10:51 +08:00
|
|
|
def to_dict(self):
|
|
|
|
"""Returns the model properties as a dict"""
|
|
|
|
result = {}
|
|
|
|
|
|
|
|
for attr, _ in six.iteritems(self.openapi_types):
|
|
|
|
value = getattr(self, attr)
|
|
|
|
if isinstance(value, list):
|
|
|
|
result[attr] = list(
|
|
|
|
map(lambda x: x.to_dict() if hasattr(x, "to_dict") else x, value)
|
|
|
|
)
|
|
|
|
elif hasattr(value, "to_dict"):
|
|
|
|
result[attr] = value.to_dict()
|
|
|
|
elif isinstance(value, dict):
|
|
|
|
result[attr] = dict(
|
|
|
|
map(
|
|
|
|
lambda item: (item[0], item[1].to_dict())
|
|
|
|
if hasattr(item[1], "to_dict")
|
|
|
|
else item,
|
|
|
|
value.items(),
|
|
|
|
)
|
|
|
|
)
|
|
|
|
else:
|
|
|
|
result[attr] = value
|
|
|
|
|
|
|
|
return result
|
|
|
|
|
|
|
|
def to_str(self):
|
|
|
|
"""Returns the string representation of the model"""
|
|
|
|
return pprint.pformat(self.to_dict())
|
|
|
|
|
|
|
|
def __repr__(self):
|
|
|
|
"""For `print` and `pprint`"""
|
|
|
|
return self.to_str()
|
|
|
|
|
|
|
|
def __eq__(self, other):
|
|
|
|
"""Returns true if both objects are equal"""
|
|
|
|
if not isinstance(other, WriterGraphRequest):
|
|
|
|
return False
|
|
|
|
|
|
|
|
return self.to_dict() == other.to_dict()
|
|
|
|
|
|
|
|
def __ne__(self, other):
|
|
|
|
"""Returns true if both objects are not equal"""
|
|
|
|
if not isinstance(other, WriterGraphRequest):
|
|
|
|
return True
|
|
|
|
|
|
|
|
return self.to_dict() != other.to_dict()
|