diff --git a/examples_notebooks/drift_search/index.html b/examples_notebooks/drift_search/index.html index 41e8139e..175938d5 100644 --- a/examples_notebooks/drift_search/index.html +++ b/examples_notebooks/drift_search/index.html @@ -2040,7 +2040,7 @@ Text unit records: 50
-
[2024-11-14T22:55:15Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/default-entity-description.lance, it will be created
+
[2024-11-14T22:57:02Z WARN  lance::dataset] No existing dataset at /home/runner/work/graphrag/graphrag/docs/examples_notebooks/inputs/operation dulce/lancedb/default-entity-description.lance, it will be created
@@ -2417,25 +2417,31 @@ search = DRIFTSearch(
-
 20%|██        | 1/5 [00:05<00:20,  5.12s/it]
+
 20%|██        | 1/5 [00:06<00:25,  6.50s/it]
-
 40%|████      | 2/5 [00:05<00:07,  2.57s/it]
+
 40%|████      | 2/5 [00:08<00:10,  3.61s/it]
-
 80%|████████  | 4/5 [00:06<00:01,  1.05s/it]
+
 60%|██████    | 3/5 [00:10<00:05,  2.84s/it]
-
100%|██████████| 5/5 [00:10<00:00,  2.18s/it]
+
 80%|████████  | 4/5 [00:11<00:02,  2.19s/it]
+
+
+
+
+
+
100%|██████████| 5/5 [00:11<00:00,  1.58s/it]
@@ -2459,85 +2465,85 @@ search = DRIFTSearch(
@@ -2561,85 +2567,97 @@ search = DRIFTSearch( + + @@ -6157,7 +6182,7 @@ search = DRIFTSearch(
diff --git a/examples_notebooks/global_search/index.html b/examples_notebooks/global_search/index.html index 144b9545..93344293 100644 --- a/examples_notebooks/global_search/index.html +++ b/examples_notebooks/global_search/index.html @@ -2606,19 +2606,19 @@ print(result.response)
### Overview of Cosmic Vocalization
 
-Cosmic Vocalization is a phenomenon that has garnered significant attention within various communities. It is perceived as a cosmic event with potential implications for security and interstellar communication. The phenomenon is central to discussions and has been described metaphorically as an "interstellar duet," suggesting a responsive and interactive nature [Data: Reports (6)].
+Cosmic Vocalization is a phenomenon that has garnered significant attention within various communities, including military and civilian groups. It is perceived as a cosmic event that may have implications for security and interstellar communication. The phenomenon is central to discussions and strategic considerations, particularly due to its potential impact and the mystery surrounding its nature.
 
-### Key Stakeholders
+### Involvement and Perspectives
 
-1. **Paranormal Military Squad**: This group is actively engaged with Cosmic Vocalization, indicating that it is part of their strategic response to cosmic phenomena. Their involvement underscores the importance of Cosmic Vocalization in security measures, suggesting that it may have implications for national or global safety [Data: Reports (6)].
+1. **Paranormal Military Squad**: This group is actively engaged with Cosmic Vocalization, indicating that it is considered a matter of strategic importance in terms of security measures. Their involvement suggests that the phenomenon is not only of scientific interest but also of potential national or global security concern [Data: Reports (6)].
 
-2. **Alex Mercer**: Alex Mercer is one of the individuals who has shown interest in Cosmic Vocalization. He perceives it as part of an interstellar duet, which implies a belief in its communicative or interactive potential with cosmic entities or phenomena [Data: Reports (6)].
+2. **Community Interest**: Cosmic Vocalization has captured the interest of various individuals and groups within the community. Alex Mercer, for instance, views it as part of an "interstellar duet," which implies a belief in its communicative or responsive nature. This perspective highlights the broader curiosity and speculative interpretations that surround the phenomenon [Data: Reports (6)].
 
-3. **Taylor Cruz**: Taylor Cruz has expressed concerns about Cosmic Vocalization, fearing it might be a "homing tune." This perspective adds a layer of urgency and potential threat, as it suggests that the phenomenon could be attracting or signaling to unknown entities [Data: Reports (6)].
+3. **Concerns and Speculations**: Taylor Cruz has expressed concerns about Cosmic Vocalization, fearing it might be a "homing tune." This adds a layer of urgency and potential threat to the discussions, as it suggests the possibility of the phenomenon being a signal or beacon with unknown intentions or consequences [Data: Reports (6)].
 
 ### Implications
 
-The involvement of these stakeholders highlights the multifaceted nature of Cosmic Vocalization. The Paranormal Military Squad's engagement suggests a need for strategic responses, while the perspectives of Alex Mercer and Taylor Cruz reflect both curiosity and concern about the phenomenon's potential implications. The notion of it being a "homing tune" raises questions about the safety and security of engaging with such cosmic events.
+The involvement of diverse groups and the range of perspectives on Cosmic Vocalization underscore its complexity and the need for further investigation. The phenomenon's potential implications for security, communication, and interstellar relations make it a subject of both intrigue and caution. As such, it remains a focal point for ongoing research and strategic planning.
@@ -2869,7 +2869,7 @@ print(
-
LLM calls: 2. Prompt tokens: 11238. Output tokens: 524.
+
LLM calls: 2. Prompt tokens: 11237. Output tokens: 533.
diff --git a/examples_notebooks/global_search_with_dynamic_community_selection/index.html b/examples_notebooks/global_search_with_dynamic_community_selection/index.html index 0c7b4197..673964bd 100644 --- a/examples_notebooks/global_search_with_dynamic_community_selection/index.html +++ b/examples_notebooks/global_search_with_dynamic_community_selection/index.html @@ -2566,19 +2566,19 @@ print(result.response) 
### Overview of Cosmic Vocalization
 
-Cosmic Vocalization is a phenomenon that has captured the attention of various individuals and groups within the community. It is perceived as a significant cosmic event, with implications that extend beyond mere observation. The concept is central to the community's focus, suggesting its importance in both cultural and scientific contexts [Data: Reports (6)].
+Cosmic Vocalization is a phenomenon that has captured the attention of various individuals and groups within the community. It is perceived as a significant cosmic event, with different interpretations and implications depending on the observer's perspective.
 
 ### Key Perspectives and Concerns
 
-Alex Mercer views Cosmic Vocalization as part of an interstellar duet, indicating a responsive and perhaps interactive approach to these cosmic events. This perspective suggests that Cosmic Vocalization may not be a solitary occurrence but part of a larger cosmic interaction [Data: Reports (6)].
+Alex Mercer views Cosmic Vocalization as part of an interstellar duet, suggesting that it may be a responsive or interactive cosmic event. This perspective highlights the potential for Cosmic Vocalization to be part of a larger cosmic communication or interaction [Data: Reports (6)].
 
-On the other hand, Taylor Cruz raises concerns about the potential implications of Cosmic Vocalization, fearing it might serve as a homing tune. This adds a layer of urgency and potential threat, suggesting that the phenomenon could have unintended consequences or attract unwanted attention [Data: Reports (6)].
+On the other hand, Taylor Cruz expresses concerns that Cosmic Vocalization might be a homing tune. This interpretation adds a layer of urgency and potential threat, as it implies that the phenomenon could be signaling or attracting attention from unknown entities [Data: Reports (6)].
 
 ### Involvement of the Paranormal Military Squad
 
-The Paranormal Military Squad is actively engaged with Cosmic Vocalization, indicating its strategic importance in security measures. Their involvement highlights the significance of these cosmic phenomena in terms of national or global security. The squad's approach treats the Universe metaphorically as a concert hall, suggesting a broader perspective on how cosmic events are interpreted and responded to by human entities [Data: Reports (6)].
+The Paranormal Military Squad is actively engaged with Cosmic Vocalization, indicating its importance in strategic and security measures. Their involvement suggests that Cosmic Vocalization is not only a subject of scientific or philosophical interest but also a matter of security concern. The squad metaphorically treats the Universe as a concert hall, which reflects a broader perspective on how cosmic events are interpreted and responded to by human entities [Data: Reports (6)].
 
-In summary, Cosmic Vocalization is a multifaceted phenomenon involving various stakeholders, each with their own perspectives and concerns. Its implications are both cultural and strategic, with potential risks and opportunities that warrant careful consideration.
+In summary, Cosmic Vocalization is a multifaceted phenomenon with varying interpretations and implications. It involves key figures like Alex Mercer and Taylor Cruz, as well as strategic entities like the Paranormal Military Squad, each bringing their unique perspectives and concerns to the table.
@@ -2656,6 +2656,14 @@ result.context_data["reports"] 1 +1 +Earth's Interstellar Communication Initiative +0.16 +# Earth's Interstellar Communication Initiativ... +8.5 + + +2 16 Dulce Military Base and Alien Intelligence Com... 0.08 @@ -2663,15 +2671,15 @@ result.context_data["reports"] 8.5 -2 -19 -Central Terminal and Viewing Monitors at Dulce... -0.02 -# Central Terminal and Viewing Monitors at Dul... +3 +18 +Paranormal Military Squad Team and Dulce Base'... +0.04 +# Paranormal Military Squad Team and Dulce Bas... 8.5 -3 +4 4 Dulce Facility and Control Room of Dulce: Extr... 0.02 @@ -2679,7 +2687,15 @@ result.context_data["reports"] 8.5 -4 +5 +19 +Central Terminal and Viewing Monitors at Dulce... +0.02 +# Central Terminal and Viewing Monitors at Dul... +8.5 + + +6 6 Cosmic Vocalization and Universe Interactions 0.02 @@ -2762,9 +2778,9 @@ print( 
Build context (gpt-4o-mini)
-LLM calls: 12. Prompt tokens: 8565. Output tokens: 1059.
+LLM calls: 12. Prompt tokens: 8565. Output tokens: 1095.
 Map-reduce (gpt-4o)
-LLM calls: 2. Prompt tokens: 4822. Output tokens: 558.
+LLM calls: 2. Prompt tokens: 5772. Output tokens: 549.
diff --git a/examples_notebooks/local_search/index.html b/examples_notebooks/local_search/index.html index c33dc038..db4cc686 100644 --- a/examples_notebooks/local_search/index.html +++ b/examples_notebooks/local_search/index.html @@ -3382,15 +3382,17 @@ Agent Alex Mercer is a prominent figure within the Paranormal Military Squad, pl Mercer is deeply involved in the philosophical and strategic aspects of interstellar communication, viewing these interactions as a form of cosmic dialogue. His leadership style is characterized by a blend of determination, compliance with mission protocols, and a protective approach towards his team. He collaborates with team members like Jordan Hayes, exploring secured areas and engaging in high-stakes, secretive operations. Mercer is also known for his intellectual curiosity and deep involvement in the philosophical and strategic aspects of interstellar communication [Data: Entities (0)]. -### Relationships and Influence +### Mentorship and Influence -Agent Mercer is depicted as a thoughtful mentor, particularly to Sam Rivera, emphasizing the importance of intuition and trust beyond protocol. His experiences during encounters with alien signals have led to profound changes, reinforcing his role as a key decision-maker and guardian in missions that transcend traditional boundaries. He is also recognized as a mentor to the team, providing belief and guidance, which highlights his influential role in both leadership and operational contexts [Data: Entities (0, 30); Relationships (167)]. +Mercer is depicted as a thoughtful mentor, particularly to Sam Rivera, emphasizing the importance of intuition and trust beyond protocol. His experiences during encounters with alien signals have led to profound changes, reinforcing his role as a key decision-maker and guardian in missions that transcend traditional boundaries. His mentorship is evident in the influence he has on team members like Sam Rivera, whose actions and confidence are shaped by Mercer's guidance [Data: Entities (0); Relationships (167)]. -### Involvement in Operation: Dulce +### Interactions and Relationships -Operation: Dulce was a significant mission undertaken by the Paranormal Military Squad at the Dulce military base, involving team members like Alex Mercer, Sam Rivera, and Taylor Cruz. The operation was characterized by uncertainty, anticipation, and a shift from mere observation to active interaction and preparation. Mercer's involvement in this operation was pivotal as it led to the first contact with an alien race, initiating a dialogue that would expand the boundaries of human understanding of the universe [Data: Entities (5); Relationships (245)]. +Within the team, Mercer shares a complex dynamic with other members, such as Taylor Cruz and Jordan Hayes. He is involved in discussions and operations at Dulce Base, often navigating the tension between strict military discipline and the imperative to confront unknown challenges. His interactions with Jordan Hayes highlight a shared commitment to discovery and understanding, often in the face of authoritarian oversight from figures like Taylor Cruz [Data: Entities (0); Relationships (116, 336)]. -In summary, Agent Alex Mercer is a key figure in the Paranormal Military Squad, known for his strategic leadership and deep involvement in extraterrestrial communications. His role in Operation: Dulce and his mentorship within the team underscore his importance in navigating the complexities of interstellar interactions. +### Conclusion + +Agent Alex Mercer is a pivotal character in the narrative of Operation: Dulce, embodying the intersection of leadership, mentorship, and strategic communication with extraterrestrial intelligence. His role is essential in guiding the Paranormal Military Squad through the complexities of interstellar dialogue and the challenges posed by their mission at Dulce Base. @@ -3440,17 +3442,19 @@ Dr. Jordan Hayes is a prominent scientist at Dulce Base, playing a crucial role ### Key Contributions and Expertise -Dr. Hayes is noted for their adaptability and skepticism, qualities that are essential given the uncertainties and unknown challenges of their mission. They are deeply involved in the scientific exploration aspects of Operation: Dulce, where their efforts are on the verge of a significant scientific breakthrough. Dr. Hayes leads efforts in isolating and understanding complex alien signals that resemble human cognition, suggesting that these signals are artificial and patterned, indicating a tandem evolution with humanity [Data: Entities (2, 17); Claims (60, 91, 134)]. +Dr. Hayes is noted for their adaptability and skepticism, qualities that are essential given the uncertainties and unknown challenges of their mission. They are deeply involved in the scientific exploration aspects of Operation: Dulce, where they focus on critical decryption tasks. Their expertise plays a vital role in understanding and interpreting alien messages, aiding the squad in their paranormal operations [Data: Entities (2, 17); Relationships (201, 246, 322)]. + +Dr. Hayes's efforts are on the verge of a significant scientific breakthrough, as they lead efforts in isolating and understanding complex alien signals that resemble human cognition. Their work suggests that these signals are artificial and patterned, indicating a tandem evolution with humanity. This breakthrough is crucial for crafting humanity's responses to cosmic alignments with stars and responsive galactic signals [Data: Entities (2); Claims (60, 83, 91, 134)]. ### Collaborative Efforts and Leadership In addition to their scientific endeavors, Dr. Hayes is involved in setting up lab stations, operating the mainframe, and playing a crucial role in the command center at Dulce Base. They work closely with colleagues like Alex Mercer, engaging in thoughtful dialogue and showing analytical thinking about the mission's uncertainties. Dr. Hayes's attention to detail is also evident in their discovery of significant panels among secured doorways and their contemplation of the mission's broader ramifications [Data: Entities (2); Relationships (26, 270, 254)]. -### Challenges and Philosophical Reflections +Dr. Hayes emphasizes the importance of adaptability in leadership, especially when facing unknown challenges. This perspective is crucial in navigating the unknown variables in their field of research, making them a vital asset to the team's efforts in decoding extraterrestrial messages [Data: Claims (2, 13); Entities (2, 17)]. -Dr. Hayes's work is not without its challenges. They have reflected on their own skepticism and its potential as a blind spot, indicating a moment of self-awareness and growth. This introspection is crucial as they navigate the complexities of alien communication and the profound implications of their findings on physics and the possibilities of their mission [Data: Claims (13, 2); Sources (4, 42)]. +### Conclusion -In summary, Dr. Jordan Hayes is a central figure in the efforts to bridge the gap between human and extraterrestrial understanding, contributing significantly to the scientific and philosophical dimensions of their mission at Dulce Base. Their work not only advances the field of extraterrestrial communication but also challenges the boundaries of human knowledge and adaptability. +Dr. Jordan Hayes is a central figure in the efforts to understand and communicate with extraterrestrial entities at Dulce Base. Their work in decoding alien signals and their leadership in scientific exploration are pivotal to the success of Operation: Dulce. Dr. Hayes's contributions are not only advancing the mission's objectives but also potentially reshaping humanity's understanding of its place in the universe [Data: Entities (2, 17); Relationships (201, 246, 322)]. diff --git a/query/drift_search/index.html b/query/drift_search/index.html index 7a56e6fe..371095eb 100644 --- a/query/drift_search/index.html +++ b/query/drift_search/index.html @@ -1559,7 +1559,7 @@ Figure 1. An entire DRIFT search hierarchy highlighting the three core phases of
  • query_state: a state object as defined in Query State that allows to track execution of a DRIFT Search instance, alongside follow ups and DRIFT actions.

    • How to Use

    -

    An example of a global search scenario can be found in the following notebook.

    +

    An example of a drift search scenario can be found in the following notebook.

    Learn More

    For a more in-depth look at the DRIFT search method, please refer to our DRIFT Search blog post

    diff --git a/search/search_index.json b/search/search_index.json index 90040cfa..1a770d82 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config": {"lang": ["en"], "separator": "[\\s\\-]+", "pipeline": ["stopWordFilter"]}, "docs": [{"location": "", "title": "Welcome to GraphRAG", "text": "

    \ud83d\udc49 Microsoft Research Blog Post \ud83d\udc49 GraphRAG Accelerator \ud83d\udc49 GraphRAG Arxiv

    Figure 1: An LLM-generated knowledge graph built using GPT-4 Turbo.

    GraphRAG is a structured, hierarchical approach to Retrieval Augmented Generation (RAG), as opposed to naive semantic-search approaches using plain text snippets. The GraphRAG process involves extracting a knowledge graph out of raw text, building a community hierarchy, generating summaries for these communities, and then leveraging these structures when perform RAG-based tasks.

    To learn more about GraphRAG and how it can be used to enhance your LLMs ability to reason about your private data, please visit the Microsoft Research Blog Post.

    "}, {"location": "#solution-accelerator", "title": "Solution Accelerator \ud83d\ude80", "text": "

    To quickstart the GraphRAG system we recommend trying the Solution Accelerator package. This provides a user-friendly end-to-end experience with Azure resources.

    "}, {"location": "#get-started-with-graphrag", "title": "Get Started with GraphRAG \ud83d\ude80", "text": "

    To start using GraphRAG, check out the Get Started guide. For a deeper dive into the main sub-systems, please visit the docpages for the Indexer and Query packages.

    "}, {"location": "#graphrag-vs-baseline-rag", "title": "GraphRAG vs Baseline RAG \ud83d\udd0d", "text": "

    Retrieval-Augmented Generation (RAG) is a technique to improve LLM outputs using real-world information. This technique is an important part of most LLM-based tools and the majority of RAG approaches use vector similarity as the search technique, which we call Baseline RAG. GraphRAG uses knowledge graphs to provide substantial improvements in question-and-answer performance when reasoning about complex information. RAG techniques have shown promise in helping LLMs to reason about private datasets - data that the LLM is not trained on and has never seen before, such as an enterprise\u2019s proprietary research, business documents, or communications. Baseline RAG was created to help solve this problem, but we observe situations where baseline RAG performs very poorly. For example:

    To address this, the tech community is working to develop methods that extend and enhance RAG. Microsoft Research\u2019s new approach, GraphRAG, uses LLMs to create a knowledge graph based on an input corpus. This graph, along with community summaries and graph machine learning outputs, are used to augment prompts at query time. GraphRAG shows substantial improvement in answering the two classes of questions described above, demonstrating intelligence or mastery that outperforms other approaches previously applied to private datasets.

    "}, {"location": "#the-graphrag-process", "title": "The GraphRAG Process \ud83e\udd16", "text": "

    GraphRAG builds upon our prior research and tooling using graph machine learning. The basic steps of the GraphRAG process are as follows:

    "}, {"location": "#index", "title": "Index", "text": ""}, {"location": "#query", "title": "Query", "text": "

    At query time, these structures are used to provide materials for the LLM context window when answering a question. The primary query modes are:

    "}, {"location": "#prompt-tuning", "title": "Prompt Tuning", "text": "

    Using GraphRAG with your data out of the box may not yield the best possible results. We strongly recommend to fine-tune your prompts following the Prompt Tuning Guide in our documentation.

    "}, {"location": "blog_posts/", "title": "Microsoft Research Blog", "text": "