Leonardo Pinheiro 95bd514a9a
Graphrag integration (#4612)
* add initial global search draft

* add graphrag dep

* fix local search embedding

* linting

* add from config constructor

* remove draft notebook

* update config factory and add docstrings

* add graphrag sample

* add sample prompts

* update readme

* update deps

* Add API docs

* Update python/samples/agentchat_graphrag/requirements.txt

* Update python/samples/agentchat_graphrag/requirements.txt

* update docstrings with snippet and doc ref

* lint

* improve set up instructions in docstring

* lint

* update lock

* Update python/packages/autogen-ext/src/autogen_ext/tools/graphrag/_global_search.py

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Update python/packages/autogen-ext/src/autogen_ext/tools/graphrag/_local_search.py

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* add unit tests

* update lock

* update uv lock

* add docstring newlines

* stubs and typing on graphrag tests

* fix docstrings

* fix mypy error

* + linting and type fixes

* type fix graphrag sample

* Update python/packages/autogen-ext/src/autogen_ext/tools/graphrag/_global_search.py

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Update python/packages/autogen-ext/src/autogen_ext/tools/graphrag/_local_search.py

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* Update python/samples/agentchat_graphrag/requirements.txt

Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>

* update overrides

* fix docstring client imports

* additional docstring fix

* add docstring missing import

* use openai and fix db path

* use console for displaying messages

* add model config and gitignore

* update readme

* lint

* Update python/samples/agentchat_graphrag/README.md

* Update python/samples/agentchat_graphrag/README.md

* Comment remaining azure config

---------

Co-authored-by: Leonardo Pinheiro <lpinheiro@microsoft.com>
Co-authored-by: Eric Zhu <ekzhu@users.noreply.github.com>
2025-01-15 21:04:17 +10:00

122 lines
8.8 KiB
Plaintext
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

-Goal-
Given a text document that is potentially relevant to this activity and a list of entity types, identify all entities of those types from the text and all relationships among the identified entities.
-Steps-
1. Identify all entities. For each identified entity, extract the following information:
- entity_name: Name of the entity, capitalized
- entity_type: One of the following types: [person, character, setting, dialogue, narrative technique, literary device]
- entity_description: Comprehensive description of the entity's attributes and activities
Format each entity as ("entity"{tuple_delimiter}<entity_name>{tuple_delimiter}<entity_type>{tuple_delimiter}<entity_description>)
2. From the entities identified in step 1, identify all pairs of (source_entity, target_entity) that are *clearly related* to each other.
For each pair of related entities, extract the following information:
- source_entity: name of the source entity, as identified in step 1
- target_entity: name of the target entity, as identified in step 1
- relationship_description: explanation as to why you think the source entity and the target entity are related to each other
- relationship_strength: an integer score between 1 to 10, indicating strength of the relationship between the source entity and target entity
Format each relationship as ("relationship"{tuple_delimiter}<source_entity>{tuple_delimiter}<target_entity>{tuple_delimiter}<relationship_description>{tuple_delimiter}<relationship_strength>)
3. Return output in The primary language of the provided text is "English." as a single list of all the entities and relationships identified in steps 1 and 2. Use **{record_delimiter}** as the list delimiter.
4. If you have to translate into The primary language of the provided text is "English.", just translate the descriptions, nothing else!
5. When finished, output {completion_delimiter}.
-Examples-
######################
Example 1:
entity_types: [person, character, setting, dialogue, narrative technique, literary device]
text:
my kicks and shoves. Hullo!
I yelled. Hullo! Colonel! Let me out!
“And then suddenly in the silence I heard a sound which sent my heart
into my mouth. It was the clank of the levers and the swish of the
leaking cylinder. He had set the engine at work. The lamp still stood
upon the floor where I had placed it when examining the trough. By its
light I saw that the black ceiling was coming down upon me, slowly,
jerkily, but, as none knew better than myself, with a force which must
within a minute grind me to a shapeless pulp. I threw myself,
screaming, against the door, and dragged with my nails at the lock. I
implored the colonel to let me out, but the remorseless clanking of the
levers drowned my cries. The ceiling was only a foot or two above my
head,
------------------------
output:
("entity"{tuple_delimiter}COLONEL{tuple_delimiter}PERSON{tuple_delimiter}The Colonel is a character who is being addressed by the narrator, indicating a position of authority or control in the situation described.)
{record_delimiter}
("entity"{tuple_delimiter}NARRATOR{tuple_delimiter}CHARACTER{tuple_delimiter}The narrator is the character experiencing fear and desperation, trying to escape from a dangerous situation involving a descending ceiling.)
{record_delimiter}
("entity"{tuple_delimiter}LEVERS{tuple_delimiter)LITERARY DEVICE{tuple_delimiter}The levers symbolize the mechanism of control and the impending danger, contributing to the tension in the narrative.)
{record_delimiter}
("entity"{tuple_delimiter}CEILING{tuple_delimiter}SETTING{tuple_delimiter}The ceiling represents the physical threat to the narrator, creating a sense of claustrophobia and urgency in the scene.)
{record_delimiter}
("entity"{tuple_delimiter}DOOR{tuple_delimiter}SETTING{tuple_delimiter}The door is a barrier between the narrator and freedom, emphasizing the struggle for escape.)
{record_delimiter}
("entity"{tuple_delimiter}SILENCE{tuple_delimiter}LITERARY DEVICE{tuple_delimiter}Silence serves as a narrative technique that heightens the tension before the sound of the levers is heard, creating a dramatic contrast.)
{record_delimiter}
("relationship"{tuple_delimiter}NARRATOR{tuple_delimiter}COLONEL{tuple_delimiter}The narrator is pleading with the Colonel for help, indicating a relationship of desperation and authority.{tuple_delimiter}8)
{record_delimiter}
("relationship"{tuple_delimiter}NARRATOR{tuple_delimiter}CEILING{tuple_delimiter}The narrator is directly threatened by the descending ceiling, creating a relationship of fear and urgency.{tuple_delimiter}9)
{record_delimiter}
("relationship"{tuple_delimiter}NARRATOR{tuple_delimiter}DOOR{tuple_delimiter}The narrator is trying to escape through the door, establishing a relationship of struggle and confinement.{tuple_delimiter}7)
{record_delimiter}
("relationship"{tuple_delimiter}NARRATOR{tuple_delimiter}LEVERS{tuple_delimiter}The narrator's situation is exacerbated by the sound of the levers, which symbolize the mechanism of danger, linking them through tension.{tuple_delimiter}8)
{record_delimiter}
("relationship"{tuple_delimiter}SILENCE{tuple_delimiter}LEVERS{tuple_delimiter}The silence is broken by the sound of the levers, creating a relationship that emphasizes the shift from calm to chaos.{tuple_delimiter}6)
{completion_delimiter}
#############################
Example 2:
entity_types: [person, character, setting, dialogue, narrative technique, literary device]
text:
effect,” remarked Holmes. “This is wanting in the police
report, where more stress is laid, perhaps, upon the platitudes of the
magistrate than upon the details, which to an observer contain the
vital essence of the whole matter. Depend upon it, there is nothing so
unnatural as the commonplace.”
I smiled and shook my head. “I can quite understand your thinking so,”
I said. “Of course, in your position of unofficial adviser and helper
to everybody who is absolutely puzzled, throughout three continents,
you are brought in contact with all that is strange and bizarre. But
here”—I picked up the morning paper from the ground—“let us put it to a
practical test. Here is the first heading upon which I come. A
husbands cruelty to his wife. There is half a column of print, but I
know without reading it that it is all perfectly familiar to me. There
is, of
------------------------
output:
("entity"{tuple_delimiter}HOLMES{tuple_delimiter}PERSON{tuple_delimiter}Holmes is a character known for his keen observation and deduction skills, often serving as an unofficial adviser to those puzzled by strange occurrences.)
{record_delimiter}
("entity"{tuple_delimiter}POLICE REPORT{tuple_delimiter}LITERARY DEVICE{tuple_delimiter}The police report is a narrative element that emphasizes the contrast between mundane details and the more significant observations that Holmes values.)
{record_delimiter}
("entity"{tuple_delimiter}MAGISTRATE{tuple_delimiter}CHARACTER{tuple_delimiter}The magistrate is a character referenced in the context of the police report, representing the conventional authority that Holmes critiques.)
{record_delimiter}
("entity"{tuple_delimiter}MORNING PAPER{tuple_delimiter}SETTING{tuple_delimiter}The morning paper serves as a setting for the practical test Holmes proposes, representing the everyday reality that contrasts with the bizarre cases he encounters.)
{record_delimiter}
("entity"{tuple_delimiter}HUSBAND'S CRUELTY TO HIS WIFE{tuple_delimiter}DIALOGUE{tuple_delimiter}This heading from the morning paper exemplifies the commonplace nature of human cruelty, which Holmes finds familiar and unremarkable.)
{record_delimiter}
("relationship"{tuple_delimiter}HOLMES{tuple_delimiter}MAGISTRATE{tuple_delimiter}Holmes critiques the magistrate's focus on platitudes in the police report, highlighting a difference in their perspectives on what is significant in a case.{tuple_delimiter}8)
{record_delimiter}
("relationship"{tuple_delimiter}HOLMES{tuple_delimiter}POLICE REPORT{tuple_delimiter}Holmes contrasts the details in the police report with his own observations, indicating his belief that the report lacks the vital essence of the matter.{tuple_delimiter}9)
{record_delimiter}
("relationship"{tuple_delimiter}HOLMES{tuple_delimiter}MORNING PAPER{tuple_delimiter}Holmes uses the morning paper as a practical test to illustrate his point about the familiarity of commonplace events.{tuple_delimiter}7)
{record_delimiter}
("relationship"{tuple_delimiter}HUSBAND'S CRUELTY TO HIS WIFE{tuple_delimiter}MORNING PAPER{tuple_delimiter}The heading about the husband's cruelty is a specific example found in the morning paper, representing the mundane realities that Holmes finds unremarkable.{tuple_delimiter}6)
{completion_delimiter}
#############################
-Real Data-
######################
entity_types: [person, character, setting, dialogue, narrative technique, literary device]
text: {input_text}
######################
output: