diff --git a/config/template/index.html b/config/template/index.html index 1949bcf6..b2538c1e 100644 --- a/config/template/index.html +++ b/config/template/index.html @@ -1518,7 +1518,7 @@

Configuration Template

The following template can be used and stored as a .env in the the directory where you're are pointing the --root parameter on your Indexing Pipeline execution.

-

For details about how to run the Indexing Pipeline, refer to the Index CLI documentation.

+

For details about how to run the Indexing Pipeline, refer to the Index CLI documentation.

.env File Template

Required variables are uncommented. All the optional configuration can be turned on or off as needed.

Minimal Configuration

diff --git a/developing/index.html b/developing/index.html index b70dac8c..ffc30995 100644 --- a/developing/index.html +++ b/developing/index.html @@ -1479,7 +1479,7 @@

Make sure you have python3.10-dev installed or more generally python<version>-dev

sudo apt-get install python3.10-dev

LLM call constantly exceeds TPM, RPM or time limits

-

GRAPHRAG_LLM_THREAD_COUNT and GRAPHRAG_EMBEDDING_THREAD_COUNT are both set to 50 by default. You can modify this values +

GRAPHRAG_LLM_THREAD_COUNT and GRAPHRAG_EMBEDDING_THREAD_COUNT are both set to 50 by default. You can modify these values to reduce concurrency. Please refer to the Configuration Documents

diff --git a/examples_notebooks/drift_search/index.html b/examples_notebooks/drift_search/index.html index 175938d5..ddb8cff1 100644 --- a/examples_notebooks/drift_search/index.html +++ b/examples_notebooks/drift_search/index.html @@ -2040,7 +2040,7 @@ Text unit records: 50
-
[2024-11-14T22:57:02Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/default-entity-description.lance, it will be created
+
[2024-11-15T03:27:50Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/default-entity-description.lance, it will be created
@@ -2417,31 +2417,25 @@ search = DRIFTSearch(
-
 20%|██        | 1/5 [00:06<00:25,  6.50s/it]
+
 20%|██        | 1/5 [00:05<00:20,  5.22s/it]
-
 40%|████      | 2/5 [00:08<00:10,  3.61s/it]
+
 40%|████      | 2/5 [00:05<00:06,  2.30s/it]
-
 60%|██████    | 3/5 [00:10<00:05,  2.84s/it]
+
 60%|██████    | 3/5 [00:05<00:02,  1.35s/it]
-
 80%|████████  | 4/5 [00:11<00:02,  2.19s/it]
-
-
-
-
-
-
100%|██████████| 5/5 [00:11<00:00,  1.58s/it]
+
100%|██████████| 5/5 [00:05<00:00,  1.55it/s]
@@ -2465,85 +2459,73 @@ search = DRIFTSearch(
-
-
- - -
-
- -
@@ -2567,97 +2549,97 @@ search = DRIFTSearch( @@ -6182,7 +6148,7 @@ search = DRIFTSearch(
diff --git a/examples_notebooks/global_search/index.html b/examples_notebooks/global_search/index.html index 93344293..be824bab 100644 --- a/examples_notebooks/global_search/index.html +++ b/examples_notebooks/global_search/index.html @@ -2606,19 +2606,19 @@ print(result.response)
### Overview of Cosmic Vocalization
 
-Cosmic Vocalization is a phenomenon that has garnered significant attention within various communities, including military and civilian groups. It is perceived as a cosmic event that may have implications for security and interstellar communication. The phenomenon is central to discussions and strategic considerations, particularly due to its potential impact and the mystery surrounding its nature.
+Cosmic Vocalization is a phenomenon that has garnered significant attention within various communities, including military and civilian groups. It is perceived as a cosmic event that may have implications for security and interstellar communication.
 
 ### Involvement and Perspectives
 
-1. **Paranormal Military Squad**: This group is actively engaged with Cosmic Vocalization, indicating that it is considered a matter of strategic importance in terms of security measures. Their involvement suggests that the phenomenon is not only of scientific interest but also of potential national or global security concern [Data: Reports (6)].
+1. **Paranormal Military Squad**: This group is actively engaged with Cosmic Vocalization, indicating that it is considered a strategic element in their security measures. Their involvement underscores the potential significance of these cosmic phenomena in defense and security contexts [Data: Reports (6)].
 
-2. **Community Interest**: Cosmic Vocalization has captured the interest of various individuals and groups within the community. Alex Mercer, for instance, views it as part of an "interstellar duet," which implies a belief in its communicative or responsive nature. This perspective highlights the broader curiosity and speculative interpretations that surround the phenomenon [Data: Reports (6)].
+2. **Community Interest**: Cosmic Vocalization has become a focal point of interest for various individuals and groups. Alex Mercer, for instance, views it as part of an interstellar duet, suggesting a responsive and perhaps communicative aspect to these cosmic events [Data: Reports (6)].
 
-3. **Concerns and Speculations**: Taylor Cruz has expressed concerns about Cosmic Vocalization, fearing it might be a "homing tune." This adds a layer of urgency and potential threat to the discussions, as it suggests the possibility of the phenomenon being a signal or beacon with unknown intentions or consequences [Data: Reports (6)].
+3. **Concerns and Speculations**: Taylor Cruz has expressed concerns about the nature of Cosmic Vocalization, fearing it might function as a homing tune. This perspective adds a layer of urgency and potential threat to the phenomenon, highlighting the diverse interpretations and concerns surrounding it [Data: Reports (6)].
 
 ### Implications
 
-The involvement of diverse groups and the range of perspectives on Cosmic Vocalization underscore its complexity and the need for further investigation. The phenomenon's potential implications for security, communication, and interstellar relations make it a subject of both intrigue and caution. As such, it remains a focal point for ongoing research and strategic planning.
+The involvement of both military and civilian entities in Cosmic Vocalization suggests that it may have far-reaching implications. The strategic interest from the Paranormal Military Squad indicates potential security concerns, while the community's varied interpretations reflect a broader curiosity and apprehension about the phenomenon's nature and purpose.
@@ -2869,7 +2869,7 @@ print(
-
LLM calls: 2. Prompt tokens: 11237. Output tokens: 533.
+
LLM calls: 2. Prompt tokens: 11237. Output tokens: 474.
diff --git a/examples_notebooks/global_search_with_dynamic_community_selection/index.html b/examples_notebooks/global_search_with_dynamic_community_selection/index.html index 673964bd..b3188fa9 100644 --- a/examples_notebooks/global_search_with_dynamic_community_selection/index.html +++ b/examples_notebooks/global_search_with_dynamic_community_selection/index.html @@ -2566,19 +2566,19 @@ print(result.response) 
### Overview of Cosmic Vocalization
 
-Cosmic Vocalization is a phenomenon that has captured the attention of various individuals and groups within the community. It is perceived as a significant cosmic event, with different interpretations and implications depending on the observer's perspective.
+Cosmic Vocalization is a phenomenon that has captured the attention of various individuals and groups within the community. It is perceived as a significant event with potential implications for both cosmic and terrestrial activities. The concept of Cosmic Vocalization is central to the community's focus, suggesting its importance in ongoing discussions and actions [Data: Reports (6)].
 
-### Key Perspectives and Concerns
+### Perspectives and Concerns
 
-Alex Mercer views Cosmic Vocalization as part of an interstellar duet, suggesting that it may be a responsive or interactive cosmic event. This perspective highlights the potential for Cosmic Vocalization to be part of a larger cosmic communication or interaction [Data: Reports (6)].
+Alex Mercer views Cosmic Vocalization as part of an interstellar duet, indicating a belief that it may be a responsive or interactive event. This perspective suggests that Cosmic Vocalization could be a form of communication or interaction with cosmic entities or phenomena [Data: Reports (6)].
 
-On the other hand, Taylor Cruz expresses concerns that Cosmic Vocalization might be a homing tune. This interpretation adds a layer of urgency and potential threat, as it implies that the phenomenon could be signaling or attracting attention from unknown entities [Data: Reports (6)].
+On the other hand, Taylor Cruz raises concerns about the implications of Cosmic Vocalization, fearing it might serve as a homing tune. This perspective introduces a layer of urgency and potential threat, as it implies that the vocalization could attract attention or entities from beyond our planet [Data: Reports (6)].
 
-### Involvement of the Paranormal Military Squad
+### Strategic Engagement
 
-The Paranormal Military Squad is actively engaged with Cosmic Vocalization, indicating its importance in strategic and security measures. Their involvement suggests that Cosmic Vocalization is not only a subject of scientific or philosophical interest but also a matter of security concern. The squad metaphorically treats the Universe as a concert hall, which reflects a broader perspective on how cosmic events are interpreted and responded to by human entities [Data: Reports (6)].
+The Paranormal Military Squad's involvement with Cosmic Vocalization highlights its significance in terms of security and strategic response. Their engagement suggests that Cosmic Vocalization is not only a subject of scientific or philosophical interest but also a matter of national or global security. This involvement underscores the need for a coordinated and strategic approach to understanding and potentially mitigating any risks associated with the phenomenon [Data: Reports (6)].
 
-In summary, Cosmic Vocalization is a multifaceted phenomenon with varying interpretations and implications. It involves key figures like Alex Mercer and Taylor Cruz, as well as strategic entities like the Paranormal Military Squad, each bringing their unique perspectives and concerns to the table.
+In summary, Cosmic Vocalization is a multifaceted phenomenon involving various stakeholders, each with their own perspectives and concerns. Its implications range from potential communication with cosmic entities to security threats, necessitating a comprehensive and strategic response.
@@ -2672,10 +2672,10 @@ result.context_data["reports"] 3 -18 -Paranormal Military Squad Team and Dulce Base'... -0.04 -# Paranormal Military Squad Team and Dulce Bas... +17 +Dulce Team and Underground Command Center: Int... +0.02 +# Dulce Team and Underground Command Center: I... 8.5 @@ -2778,9 +2778,9 @@ print( 
Build context (gpt-4o-mini)
-LLM calls: 12. Prompt tokens: 8565. Output tokens: 1095.
+LLM calls: 12. Prompt tokens: 8565. Output tokens: 1098.
 Map-reduce (gpt-4o)
-LLM calls: 2. Prompt tokens: 5772. Output tokens: 549.
+LLM calls: 2. Prompt tokens: 5864. Output tokens: 506.
diff --git a/examples_notebooks/local_search/index.html b/examples_notebooks/local_search/index.html index db4cc686..5594f0ef 100644 --- a/examples_notebooks/local_search/index.html +++ b/examples_notebooks/local_search/index.html @@ -3376,23 +3376,21 @@ print(result.response) 
### Overview of Agent Mercer
 
-Agent Alex Mercer is a prominent figure within the Paranormal Military Squad, playing a crucial role in Operation: Dulce at the Dulce Base. He is recognized for his strategic and commanding presence, particularly in overseeing operations related to communication with extraterrestrial intelligence. His responsibilities include initiating broadcasts to communicate with extraterrestrial beings, decoding alien messages, and leading the team in understanding and responding to these communications [Data: Entities (0)].
+Agent Alex Mercer is a prominent figure within the Paranormal Military Squad, playing a crucial role in Operation: Dulce at the Dulce Base. He is recognized for his commanding and strategic presence, particularly in overseeing operations related to communication with extraterrestrial intelligence. His responsibilities include initiating broadcasts to communicate with extraterrestrial beings, decoding alien messages, and leading the team in understanding and responding to these communications [Data: Entities (0)].
 
 ### Role and Responsibilities
 
-Mercer is deeply involved in the philosophical and strategic aspects of interstellar communication, viewing these interactions as a form of cosmic dialogue. His leadership style is characterized by a blend of determination, compliance with mission protocols, and a protective approach towards his team. He collaborates with team members like Jordan Hayes, exploring secured areas and engaging in high-stakes, secretive operations. Mercer is also known for his intellectual curiosity and deep involvement in the philosophical and strategic aspects of interstellar communication [Data: Entities (0)].
+Mercer is deeply involved in the philosophical and strategic aspects of interstellar communication, viewing these interactions as a form of cosmic dialogue. His leadership style is characterized by a blend of determination, compliance with mission protocols, and a protective approach towards his team. He collaborates with team members like Jordan Hayes, exploring secured areas and engaging in high-stakes, secretive operations. Mercer is also known for his intellectual curiosity and deep involvement in the mission's objectives, which include analyzing cosmic signals and strategizing contact efforts [Data: Entities (0)].
 
-### Mentorship and Influence
+### Relationships and Influence
 
-Mercer is depicted as a thoughtful mentor, particularly to Sam Rivera, emphasizing the importance of intuition and trust beyond protocol. His experiences during encounters with alien signals have led to profound changes, reinforcing his role as a key decision-maker and guardian in missions that transcend traditional boundaries. His mentorship is evident in the influence he has on team members like Sam Rivera, whose actions and confidence are shaped by Mercer's guidance [Data: Entities (0); Relationships (167)].
+Agent Mercer is depicted as a thoughtful mentor, particularly to Sam Rivera, emphasizing the importance of intuition and trust beyond protocol. His mentorship has a significant influence on Rivera's actions and confidence [Data: Relationships (167)]. Additionally, Mercer shares a professional relationship with other key figures at Dulce Base, such as Taylor Cruz and Jordan Hayes, working closely with them to navigate the complexities of alien communication and technology [Data: Relationships (71, 116, 336)].
 
-### Interactions and Relationships
+### Challenges and Philosophical Approach
 
-Within the team, Mercer shares a complex dynamic with other members, such as Taylor Cruz and Jordan Hayes. He is involved in discussions and operations at Dulce Base, often navigating the tension between strict military discipline and the imperative to confront unknown challenges. His interactions with Jordan Hayes highlight a shared commitment to discovery and understanding, often in the face of authoritarian oversight from figures like Taylor Cruz [Data: Entities (0); Relationships (116, 336)].
+Despite his outward compliance, Mercer harbors a subtle form of determination that resists easy submission to authority, as observed during the briefing for Operation: Dulce [Data: Claims (1)]. He is involved in high-stakes operations that require a balance between strict military discipline and the imperative to confront unknown challenges. This duality underscores his complex role within the squad, as he navigates the tension between adhering to protocols and exploring the broader implications of their mission [Data: Entities (0, 8)].
 
-### Conclusion
-
-Agent Alex Mercer is a pivotal character in the narrative of Operation: Dulce, embodying the intersection of leadership, mentorship, and strategic communication with extraterrestrial intelligence. His role is essential in guiding the Paranormal Military Squad through the complexities of interstellar dialogue and the challenges posed by their mission at Dulce Base.
+In summary, Agent Alex Mercer is a pivotal figure in the Paranormal Military Squad, known for his strategic leadership, mentorship, and philosophical approach to interstellar communication. His role at Dulce Base is marked by a commitment to understanding and engaging with extraterrestrial intelligence, making him a key decision-maker in missions that transcend traditional boundaries.
@@ -3438,23 +3436,21 @@ print(result.response) 
### Overview of Dr. Jordan Hayes
 
-Dr. Jordan Hayes is a prominent scientist at Dulce Base, playing a crucial role in the Paranormal Military Squad's efforts to understand and communicate with extraterrestrial entities. Dr. Hayes's work is primarily focused on decoding and analyzing alien signals and codes, which are essential for interstellar communication and potentially interspecies interaction. This involves decrypting algorithms, interpreting cosmic signals, and analyzing the implications of alien society [Data: Entities (2, 17, 47); Relationships (201, 322, 283)].
+Dr. Jordan Hayes is a prominent scientist at Dulce Base, playing a crucial role in the Paranormal Military Squad's efforts to understand and communicate with extraterrestrial entities. Dr. Hayes's work is primarily focused on decoding and analyzing alien signals and codes, which are essential for interstellar communication and potentially interspecies interaction. This involves decrypting algorithms, interpreting cosmic signals, and analyzing the implications of alien society [Data: Entities (2, 17); Relationships (201, 322, 283)].
 
 ### Key Contributions and Expertise
 
-Dr. Hayes is noted for their adaptability and skepticism, qualities that are essential given the uncertainties and unknown challenges of their mission. They are deeply involved in the scientific exploration aspects of Operation: Dulce, where they focus on critical decryption tasks. Their expertise plays a vital role in understanding and interpreting alien messages, aiding the squad in their paranormal operations [Data: Entities (2, 17); Relationships (201, 246, 322)].
-
-Dr. Hayes's efforts are on the verge of a significant scientific breakthrough, as they lead efforts in isolating and understanding complex alien signals that resemble human cognition. Their work suggests that these signals are artificial and patterned, indicating a tandem evolution with humanity. This breakthrough is crucial for crafting humanity's responses to cosmic alignments with stars and responsive galactic signals [Data: Entities (2); Claims (60, 83, 91, 134)].
+Dr. Hayes is noted for their adaptability and skepticism, qualities that are essential given the uncertainties and unknown challenges of their mission. They are deeply involved in the scientific exploration aspects of Operation: Dulce, where their efforts are on the verge of a significant scientific breakthrough. Dr. Hayes leads efforts in isolating and understanding complex alien signals that resemble human cognition, suggesting that these signals are artificial and patterned, indicating a tandem evolution with humanity [Data: Entities (2); Claims (60, 91, 134)].
 
 ### Collaborative Efforts and Leadership
 
 In addition to their scientific endeavors, Dr. Hayes is involved in setting up lab stations, operating the mainframe, and playing a crucial role in the command center at Dulce Base. They work closely with colleagues like Alex Mercer, engaging in thoughtful dialogue and showing analytical thinking about the mission's uncertainties. Dr. Hayes's attention to detail is also evident in their discovery of significant panels among secured doorways and their contemplation of the mission's broader ramifications [Data: Entities (2); Relationships (26, 270, 254)].
 
-Dr. Hayes emphasizes the importance of adaptability in leadership, especially when facing unknown challenges. This perspective is crucial in navigating the unknown variables in their field of research, making them a vital asset to the team's efforts in decoding extraterrestrial messages [Data: Claims (2, 13); Entities (2, 17)].
+### Challenges and Philosophical Reflections
 
-### Conclusion
+Dr. Hayes's role is not without its challenges. They often reflect on their own skepticism and its potential as a blind spot, indicating a moment of self-awareness and growth. This introspection is crucial as they navigate the profound implications of their findings on physics and the possibilities of their mission. Dr. Hayes's work suggests that the signals they are deciphering are not just messages but structured, intentional interstellar communications, which could lead to a technological breakthrough [Data: Claims (2, 13, 83, 96)].
 
-Dr. Jordan Hayes is a central figure in the efforts to understand and communicate with extraterrestrial entities at Dulce Base. Their work in decoding alien signals and their leadership in scientific exploration are pivotal to the success of Operation: Dulce. Dr. Hayes's contributions are not only advancing the mission's objectives but also potentially reshaping humanity's understanding of its place in the universe [Data: Entities (2, 17); Relationships (201, 246, 322)].
+In summary, Dr. Jordan Hayes is a vital asset to the Paranormal Military Squad, contributing significantly to the understanding of extraterrestrial communications and the potential for interspecies interaction. Their work at Dulce Base is characterized by a blend of scientific rigor, adaptability, and philosophical reflection, making them a key figure in the ongoing exploration of the unknown.
diff --git a/get_started/index.html b/get_started/index.html index 5ca50aeb..ffd81d52 100644 --- a/get_started/index.html +++ b/get_started/index.html @@ -1538,7 +1538,7 @@ Since we have already configured a directory named ./ragtest in the

Running the Indexing pipeline

Finally we'll run the pipeline!

diff --git a/index/outputs/index.html b/index/outputs/index.html index 342c2146..eb1ef7e5 100644 --- a/index/outputs/index.html +++ b/index/outputs/index.html @@ -675,6 +675,42 @@ + + +
  • + + + create_final_entities + + + +
    • + +
  • + + + create_final_nodes + + + +
    • + +
  • + + + create_final_relationships + + + +
    • + +
  • + + + create_final_text_units + + +
    • @@ -1517,6 +1553,42 @@ + + +
  • + + + create_final_entities + + + +
    • + +
  • + + + create_final_nodes + + + +
    • + +
  • + + + create_final_relationships + + + +
    • + +
  • + + + create_final_text_units + + +
    • @@ -1540,81 +1612,408 @@

    Outputs

    The default pipeline produces a series of output tables that align with the conceptual knowledge model. This page describes the detailed output table schemas. By default we write these tables out as parquet files on disk.

    Shared fields

    -

    All tables have two identifier fields: -- id: str - Generated UUID, assuring global uniqueness -- human_readable_id: int - This is an incremented short ID created per-run. For example, we use this short ID with generated summaries that print citations so they are easy to cross-reference visually.

    +

    All tables have two identifier fields:

    + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    idstrGenerated UUID, assuring global uniqueness
    human_readable_idintThis is an incremented short ID created per-run. For example, we use this short ID with generated summaries that print citations so they are easy to cross-reference visually.

    create_final_communities

    -

    This is a list of the final communities generated by Leiden. Communities are strictly hierarchical, subdividing into children as the cluster affinity is narrowed. -- community: int - Leiden-generated cluster ID for the community. Note that these increment with depth, so they are unique through all levels of the community hierarchy. For this table, human_readable_id is a copy of the community ID rather than a plain increment. -- level: int - Depth of the community in the hierarchy. -- title: str - Friendly name of the community. -- entity_ids - List of entities that are members of the community. -- relationship_ids - List of relationships that are wholly within the community (source and target are both in the community). -- text_unit_ids - List of text units represented within the community. -- period - Date of ingest, used for incremental update merges. -- size - Size of the community (entity count), used for incremental update merges.

    +

    This is a list of the final communities generated by Leiden. Communities are strictly hierarchical, subdividing into children as the cluster affinity is narrowed.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    communityintLeiden-generated cluster ID for the community. Note that these increment with depth, so they are unique through all levels of the community hierarchy. For this table, human_readable_id is a copy of the community ID rather than a plain increment.
    levelintDepth of the community in the hierarchy.
    titlestrFriendly name of the community.
    entity_idsstr[]List of entities that are members of the community.
    relationship_idsstr[]List of relationships that are wholly within the community (source and target are both in the community).
    text_unit_idsstr[]List of text units represented within the community.
    periodstrDate of ingest, used for incremental update merges. ISO8601
    sizeintSize of the community (entity count), used for incremental update merges.

    create_final_community_reports

    -

    This is the list of summarized reports for each community. -- community: int - Short ID of the community this report applies to. -- level: int - Level of the community this report applies to. -- title: str - LM-generated title for the report. -- summary: str - LM-generated summary of the report. -- full_content: str - LM-generated full report. -- rank: float - LM-derived relevance ranking of the report based on member entity salience -- rank_explanation - LM-derived explanation of the rank. -- findings: dict - LM-derived list of the top 5-10 insights from the community. Contains summary and explanation values. -- full_content_json - Full JSON output as returned by the LM. Most fields are extracted into columns, but this JSON is sent for query summarization so we leave it to allow for prompt tuning to add fields/content by end users. -- period - Date of ingest, used for incremental update merges. -- size - Size of the community (entity count), used for incremental update merges.

    +

    This is the list of summarized reports for each community.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    communityintShort ID of the community this report applies to.
    levelintLevel of the community this report applies to.
    titlestrLM-generated title for the report.
    summarystrLM-generated summary of the report.
    full_contentstrLM-generated full report.
    rankfloatLM-derived relevance ranking of the report based on member entity salience
    rank_explanationstrLM-derived explanation of the rank.
    findingsdictLM-derived list of the top 5-10 insights from the community. Contains summary and explanation values.
    full_content_jsonjsonFull JSON output as returned by the LM. Most fields are extracted into columns, but this JSON is sent for query summarization so we leave it to allow for prompt tuning to add fields/content by end users.
    periodstrDate of ingest, used for incremental update merges. ISO8601
    sizeintSize of the community (entity count), used for incremental update merges.

    create_final_covariates

    -

    (Optional) If claim extraction is turned on, this is a list of the extracted covariates. Note that claims are typically oriented around identifying malicious behavior such as fraud, so they are not useful for all datasets. -- covariate_type: str - This is always "claim" with our default covariates. -- type: str - Nature of the claim type. -- description: str - LM-generated description of the behavior. -- subject_id: str - Name of the source entity (that is performing the claimed behavior). -- object_id: str - Name of the target entity (that the claimed behavior is performed on). -- status: str [TRUE, FALSE, SUSPECTED] - LM-derived assessment of the correctness of the claim. -- start_date: str (ISO8601) - LM-derived start of the claimed activity. -- end_date: str (ISO8601) - LM-derived end of the claimed activity. -- source_text: str - Short string of text containing the claimed behavior. -- text_unit_id: str - ID of the text unit the claim text was extracted from.

    +

    (Optional) If claim extraction is turned on, this is a list of the extracted covariates. Note that claims are typically oriented around identifying malicious behavior such as fraud, so they are not useful for all datasets.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    covariate_typestrThis is always "claim" with our default covariates.
    typestrNature of the claim type.
    descriptionstrLM-generated description of the behavior.
    subject_idstrName of the source entity (that is performing the claimed behavior).
    object_idstrName of the target entity (that the claimed behavior is performed on).
    statusstrLM-derived assessment of the correctness of the claim. One of [TRUE, FALSE, SUSPECTED]
    start_datestrLM-derived start of the claimed activity. ISO8601
    end_datestrLM-derived end of the claimed activity. ISO8601
    source_textstrShort string of text containing the claimed behavior.
    text_unit_idstrID of the text unit the claim text was extracted from.

    create_final_documents

    -

    List of document content after import. -- title: str - Filename, unless otherwise configured during CSV import. -- text: str - Full text of the document. -- text_unit_ids: str[] - List of text units (chunks) that were parsed from the document. -- attributes: dict (optional) - If specified during CSV import, this is a dict of attributes for the document.

    -

    create_final_entities

    -

    List of all entities found in the data by the LM. -- title: str - Name of the entity. -- type: str - Type of the entity. By default this will be "organization", "person", "geo", or "event" unless configured differently or auto-tuning is used. -- description: str - Textual description of the entity. Entities may be found in many text units, so this is an LM-derived summary of all descriptions. -- text_unit_ids: str[] - List of the text units containing the entity.

    -

    create_final_nodes

    +

    List of document content after import.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    titlestrFilename, unless otherwise configured during CSV import.
    textstrFull text of the document.
    text_unit_idsstr[]List of text units (chunks) that were parsed from the document.
    attributesdict(optional) If specified during CSV import, this is a dict of attributes for the document.
    +

    create_final_entities

    +

    List of all entities found in the data by the LM.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    titlestrName of the entity.
    typestrType of the entity. By default this will be "organization", "person", "geo", or "event" unless configured differently or auto-tuning is used.
    descriptionstrTextual description of the entity. Entities may be found in many text units, so this is an LM-derived summary of all descriptions.
    text_unit_idsstr[]List of the text units containing the entity.
    +

    create_final_nodes

    This is graph-related information for the entities. It contains only information relevant to the graph such as community. There is an entry for each entity at every community level it is found within, so you may see "duplicate" entities.

    -

    Note that the ID fields match those in create_final_entities and can be used for joining if additional information about a node is required. -- title: str - Name of the referenced entity. Duplicated from create_final_entities for convenient cross-referencing. -- community: int - Leiden community the node is found within. Entities are not always assigned a community (they may not be close enough to any), so they may have a ID of -1. -- level: int - Level of the community the entity is in. -- degree: int - Node degree (connectedness) in the graph. -- x: float - X position of the node for visual layouts. If graph embeddings and UMAP are not turned on, this will be 0. -- y: float - Y position of the node for visual layouts. If graph embeddings and UMAP are not turned on, this will be 0.

    +

    Note that the ID fields match those in create_final_entities and can be used for joining if additional information about a node is required.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    titlestrName of the referenced entity. Duplicated from create_final_entities for convenient cross-referencing.
    communityintLeiden community the node is found within. Entities are not always assigned a community (they may not be close enough to any), so they may have a ID of -1.
    levelintLevel of the community the entity is in.
    degreeintNode degree (connectedness) in the graph.
    xfloatX position of the node for visual layouts. If graph embeddings and UMAP are not turned on, this will be 0.
    yfloatY position of the node for visual layouts. If graph embeddings and UMAP are not turned on, this will be 0.

    create_final_relationships

    -

    List of all entity-to-entity relationships found in the data by the LM. This is also the edge list for the graph. -- source: str - Name of the source entity. -- target: str - Name of the target entity. -- description: str - LM-derived description of the relationship. Also see note for entity descriptions. -- weight: float - Weight of the edge in the graph. This is summed from an LM-derived "strength" measure for each relationship instance. -- combined_degree: int - Sum of source and target node degrees. -- text_unit_ids: str[] - List of text units the relationship was found within.

    +

    List of all entity-to-entity relationships found in the data by the LM. This is also the edge list for the graph.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    sourcestrName of the source entity.
    targetstrName of the target entity.
    descriptionstrLM-derived description of the relationship. Also see note for entity descriptions.
    weightfloatWeight of the edge in the graph. This is summed from an LM-derived "strength" measure for each relationship instance.
    combined_degreeintSum of source and target node degrees.
    text_unit_idsstr[]List of text units the relationship was found within.

    create_final_text_units

    -

    List of all text chunks parsed from the input documents. -- text: str - Raw full text of the chunk. -- n_tokens: int - Number of tokens in the chunk. This should normally match the chunk_size config parameter, except for the last chunk which is often shorter. -- document_ids: str[] - List of document IDs the chunk came from. This is normally only 1 due to our default groupby, but for very short text documents (e.g., microblogs) it can be configured so text units span multiple documents. -- entity_ids: str[] - List of entities found in the text unit. -- relationships_ids: str[] - List of relationships found in the text unit. -- covariate_ids: str[] - Optional list of covariates found in the text unit.

    +

    List of all text chunks parsed from the input documents.

    + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
    nametypedescription
    textstrRaw full text of the chunk.
    n_tokensintNumber of tokens in the chunk. This should normally match the chunk_size config parameter, except for the last chunk which is often shorter.
    document_idsstr[]List of document IDs the chunk came from. This is normally only 1 due to our default groupby, but for very short text documents (e.g., microblogs) it can be configured so text units span multiple documents.
    entity_idsstr[]List of entities found in the text unit.
    relationships_idsstr[]List of relationships found in the text unit.
    covariate_idsstr[]Optional list of covariates found in the text unit.
    diff --git a/index/overview/index.html b/index/overview/index.html index 0dd9f8b0..51a8e718 100644 --- a/index/overview/index.html +++ b/index/overview/index.html @@ -1607,34 +1607,7 @@ After you have a config file you can run the pipeline using the CLI or the Pytho yarn run:index --config your_pipeline.yml # custom config mode

    Python API

    -
    from graphrag.index import run_pipeline
-from graphrag.index.config import PipelineWorkflowReference
-
-workflows: list[PipelineWorkflowReference] = [
-    PipelineWorkflowReference(
-        steps=[
-            {
-                # built-in verb
-                "verb": "derive",  # https://github.com/microsoft/datashaper/blob/main/python/datashaper/datashaper/verbs/derive.py
-                "args": {
-                    "column1": "col1",  # from above
-                    "column2": "col2",  # from above
-                    "to": "col_multiplied",  # new column name
-                    "operator": "*",  # multiply the two columns
-                },
-                # Since we're trying to act on the default input, we don't need explicitly to specify an input
-            }
-        ]
-    ),
-]
-
-dataset = pd.DataFrame([{"col1": 2, "col2": 4}, {"col1": 5, "col2": 10}])
-outputs = []
-async for output in await run_pipeline(dataset=dataset, workflows=workflows):
-    outputs.append(output)
-pipeline_result = outputs[-1]
-print(pipeline_result)
-
    +

    Please see the examples folder for a handful of functional pipelines illustrating how to create and run via a custom settings.yml or through custom python scripts.

    Further Reading