KAG/knext/common/env.py
zhuzhongshu123 13cea5f6fe
feat(kag): update to v0.7 (#456)
* add think cost

* update csv scanner

* add final rerank

* add reasoner

* add iterative planner

* fix dpr search

* fix dpr search

* add reference data

* move odps import

* update requirement.txt

* update 2wiki

* add missing file

* fix markdown reader

* add iterative planning

* update version

* update runner

* update 2wiki example

* update bridge

* merge solver and solver_new

* add cur day

* writer delete

* update multi process

* add missing files

* fix report

* add chunk retrieved executor

* update try in stream runner result

* add path

* add math executor

* update hotpotqa example

* remove log

* fix python coder solver

* update hotpotqa example

* fix python coder solver

* update config

* fix bad

* add log

* remove unused code

* commit with task thought

* move kag model to common

* add default chat llm

* fix

* use static planner

* support chunk graph node

* add args

* support naive rag

* llm client support tool calls

* add default async

* add openai

* fix result

* fix markdown reader

* fix thinker

* update asyncio interface

* feat(solver): add mcp support (#444)

* 上传mcp client相关代码

* 1、完成一套mcp client的调用,从pipeline到planner、executor
2、允许json中传入多个mcp_server,通过大模型进行调用并选择
3、调通baidu_map_mcp的使用

* 1、schema

* bugfix:删减冗余代码

---------

Co-authored-by: wanxingyu.wxy <wanxingyu.wxy@antgroup.com>

* fix affairqa after solver refactor

* fix affairqa after solver refactor

* fix readme

* add params

* update version

* update mcp executor

* update mcp executor

* solver add mcp executor

* add missing file

* add mpc executor

* add executor

* x

* update

* fix requirement

* fix main llm config

* fix solver

* bugfix:修复invoke函数调用逻辑

* chg eva

* update example

* add kag layer

* add step task

* support dot refresh

* support dot refresh

* support dot refresh

* support dot refresh

* add retrieved num

* add retrieved num

* add pipelineconf

* update ppr

* update musique prompts

* update

* add to_dict for BuilderComponentData

* async build

* add deduce prompt

* add deduce prompt

* add deduce prompt

* fix reader

* add deduce prompt

* add page thinker report

* modify prmpt

* add step status

* add self cognition

* add self cognition

* add memory graph storage

* add now time

* update memory config

* add now time

* chg graph loader

* 添加prqa数据集和代码

* bugfix:prqa调用逻辑修复

* optimize:优化代码逻辑,生成答案规范化

* add retry py code

* update memory graph

* update memory graph

* fix

* fix ner

* add with_out_refer generator prompt

* fix

* close ckpt

* fix query

* fix query

* update version

* add llm checker

* add llm checker

* 1、上传evalutor.py以及修改gold_answer.json格式
2、优化代码逻辑
3、修改README.md文件

* update exp

* update exp

* rerank support

* add static rewrite query

* recall more chunks

* fix graph load

* add static rewrite query

* fix bugs

* add finish check

* add finish check

* add finish check

* add finish check

* 1、上传evalutor.py的结果
2、优化代码逻辑,优化readme文件

* add lf retry

* add memory graph api

* fix reader api

* add ner

* add metrics

* fix bug

* remove ner

* add reraise fo retry

* add edge prop to memory graph

* add memory graph

* 1、评测数据集结果修正
2、优化evaluator.py代码
3、删除结果不存在而gold_answer中有答案的问题

* 删除评测结果文件

* fix knext host addr

* async eva

* add lf prompt

* add lf prompt

* add config

* add retry

* add unknown check

* add rc result

* add rc result

* add rc result

* add rc result

* 依据kag pipeline格式修改代码逻辑并通过测试

* bugfix:删除冗余代码

* fix report prompt

* bugfix:触发重试机制

* bugfix:中文符号错误

* fix rethinker prompt

* update version to 0.6.2b78

* update version

* 1、修改evaluator.py,通过大模型计算准确率,符合最新调用逻辑
2、修改prompt,让没有回答的结果重复测试

* update affairqa for evaluate

* update affairqa for evaluate

* bugfix:修正数据集

* bugfix:修正数据集

* bugfix:修正数据集

* fix name conflict

* bugfix:删除错误问题

* bugfix:文件名命名错误导致evaluator失败

* update for affairqa eval

* bugfix:修改代码保持evaluate逻辑一致

* x

* update for affairqa readme

* remove temp eval scripts

* bugfix for math deduce

* merge 0.6.2_dev

* merge 0.6.2_dev

* fix

* update client addr

* updated version

* update for affairqa eval

* evaUtils 支持中文

* fix affairqa eval:

* remove unused example

* update kag config

* fix default value

* update readme

* fix init

* 注释信息修改,并添加部分class说明

* update example config

* Tc 0.7.0 (#459)

* 提交affairQA 代码

* fix affairqa eval

---------

Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>

* fix all examples

* reformat

---------

Co-authored-by: peilong <peilong.zpl@antgroup.com>
Co-authored-by: 锦呈 <zhangxinhong.zxh@antgroup.com>
Co-authored-by: wanxingyu.wxy <wanxingyu.wxy@antgroup.com>
Co-authored-by: zhengke.gzk <zhengke.gzk@antgroup.com>
2025-04-17 17:23:52 +08:00

157 lines
5.0 KiB
Python

# -*- coding: utf-8 -*-
# Copyright 2023 OpenSPG Authors
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except
# in compliance with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied.
from collections import OrderedDict
import logging
import os
import sys
import json
from ruamel.yaml import YAML
from pathlib import Path
from typing import Union, Optional
yaml = YAML()
yaml.default_flow_style = False
yaml.indent(mapping=2, sequence=4, offset=2)
logger = logging.getLogger(__name__)
DEFAULT_HOST_ADDR = "http://127.0.0.1:8887"
class Environment:
_instance = None
_config = None
def __new__(cls):
if cls._instance is None:
cls._instance = super(Environment, cls).__new__(cls)
try:
log_config = cls._instance.config.get("log", {})
value = log_config.get("level", "INFO")
logging.basicConfig(level=logging.getLevelName(value),
format='%(asctime)s - %(levelname)s - %(name)s - %(message)s',
datefmt='%Y-%m-%d %H:%M:%S')
except:
logger.info("logger info not set")
return cls._instance
@property
def config(self):
closest_config = self._closest_config()
if not hasattr(self, '_config_path') or self._config_path != closest_config:
self._config_path = closest_config
self._config = self.get_config()
if self._config is None:
self._config = self.get_config()
return self._config
@property
def project_path(self):
config_path = self._closest_config()
return os.path.abspath(os.path.dirname(config_path))
@property
def config_path(self):
if not hasattr(self, '_config_path') or self._config_path is None:
self._config_path = self._closest_config()
return self._config_path
@property
def project_config(self):
return self.config.get("project", {})
@property
def id(self):
id = self.project_config.get("id", None)
if id is None:
logger.warning("can not find id in project config")
if os.getenv("KAG_PROJECT_ID",None):
return os.getenv("KAG_PROJECT_ID")
else:
raise Exception(
"project id not restore in spgserver, please restore project first"
)
return id
@property
def project_id(self):
return self.id
@property
def namespace(self):
if os.getenv("KAG_PROJECT_NAMESPACE"):
return os.getenv("KAG_PROJECT_NAMESPACE")
namespace = self.project_config.get("namespace", None)
if namespace is None:
raise Exception("project namespace is not defined")
return namespace
@property
def name(self):
return self.namespace
@property
def host_addr(self):
host_addr = self.project_config.get("host_addr", None)
if host_addr is None:
host_addr = DEFAULT_HOST_ADDR
logger.warning(f"project host_addr is not defined, use default host_addr: {host_addr}")
return host_addr
def get_config(self):
"""
Get knext config file as a ConfigParser.
"""
local_cfg_path = self._closest_config()
try:
with open(local_cfg_path) as f:
local_cfg = yaml.load(f)
except Exception as e:
raise Exception(f"failed to load config from {local_cfg_path}, error: {e}")
projdir = ""
if local_cfg_path:
projdir = str(Path(local_cfg_path).parent)
if projdir not in sys.path:
sys.path.append(projdir)
return local_cfg
def _closest_config(
self,
path: Union[str, os.PathLike] = ".",
prev_path: Optional[Union[str, os.PathLike]] = None,
) -> str:
"""
Return the path to the closest kag_config.yaml file by traversing the current
directory and its parents
"""
if prev_path is not None and str(path) == str(prev_path):
return ""
path = Path(path).resolve()
cfg_files = list(path.glob("*.yaml"))
cfg_file = next((f for f in cfg_files if f.name == "kag_config.yaml"), cfg_files[0] if cfg_files else None)
if cfg_file and cfg_file.exists():
return str(cfg_file)
if path.parent == path:
raise FileNotFoundError("No kag_config.yaml file found in current directory or any parent directories")
return self._closest_config(path.parent, path)
def dump(self, path=None, **kwargs):
with open(path or self._config_path, "w") as f:
yaml.dump(self._config, f)
env = Environment()