From 0e7d22bfb0cbd08c8d66b34da763d38dfe4a719c Mon Sep 17 00:00:00 2001 From: Nathan Evans Date: Fri, 10 Jan 2025 11:36:27 -0800 Subject: [PATCH] Jan documentation updates (#1612) * Update workflow docs * Docs cleanup --- docs/developing.md | 1 + docs/index/default_dataflow.md | 96 ++++++++++++++++++---------------- docs/index/overview.md | 9 +--- 3 files changed, 53 insertions(+), 53 deletions(-) diff --git a/docs/developing.md b/docs/developing.md index 83165313..b143b593 100644 --- a/docs/developing.md +++ b/docs/developing.md @@ -51,6 +51,7 @@ Available scripts are: - `poetry run poe test_unit` - This will execute unit tests. - `poetry run poe test_integration` - This will execute integration tests. - `poetry run poe test_smoke` - This will execute smoke tests. +- `poetry run poe test_verbs` - This will execute tests of the basic workflows. - `poetry run poe check` - This will perform a suite of static checks across the package, including: - formatting - documentation formatting diff --git a/docs/index/default_dataflow.md b/docs/index/default_dataflow.md index 6f559565..1683d162 100644 --- a/docs/index/default_dataflow.md +++ b/docs/index/default_dataflow.md @@ -7,7 +7,7 @@ The knowledge model is a specification for data outputs that conform to our data - `Document` - An input document into the system. These either represent individual rows in a CSV or individual .txt file. - `TextUnit` - A chunk of text to analyze. The size of these chunks, their overlap, and whether they adhere to any data boundaries may be configured below. A common use case is to set `CHUNK_BY_COLUMNS` to `id` so that there is a 1-to-many relationship between documents and TextUnits instead of a many-to-many. - `Entity` - An entity extracted from a TextUnit. These represent people, places, events, or some other entity-model that you provide. -- `Relationship` - A relationship between two entities. These are generated from the covariates. +- `Relationship` - A relationship between two entities. - `Covariate` - Extracted claim information, which contains statements about entities which may be time-bound. - `Community` - Once the graph of entities and relationships is built, we perform hierarchical community detection on them to create a clustering structure. - `Community Report` - The contents of each community are summarized into a generated report, useful for human reading and downstream search. @@ -24,7 +24,7 @@ title: Dataflow Overview flowchart TB subgraph phase1[Phase 1: Compose TextUnits] documents[Documents] --> chunk[Chunk] - chunk --> embed[Embed] --> textUnits[Text Units] + chunk --> textUnits[Text Units] end subgraph phase2[Phase 2: Graph Extraction] textUnits --> graph_extract[Entity & Relationship Extraction] @@ -34,39 +34,37 @@ flowchart TB end subgraph phase3[Phase 3: Graph Augmentation] graph_outputs --> community_detect[Community Detection] - community_detect --> graph_embed[Graph Embedding] - graph_embed --> augmented_graph[Augmented Graph Tables] + community_detect --> community_outputs[Communities Table] end subgraph phase4[Phase 4: Community Summarization] - augmented_graph --> summarized_communities[Community Summarization] - summarized_communities --> embed_communities[Community Embedding] - embed_communities --> community_outputs[Community Tables] + community_outputs --> summarized_communities[Community Summarization] + summarized_communities --> community_report_outputs[Community Reports Table] end subgraph phase5[Phase 5: Document Processing] documents --> link_to_text_units[Link to TextUnits] textUnits --> link_to_text_units - link_to_text_units --> embed_documents[Document Embedding] - embed_documents --> document_graph[Document Graph Creation] - document_graph --> document_outputs[Document Tables] + link_to_text_units --> document_outputs[Documents Table] end subgraph phase6[Phase 6: Network Visualization] - document_outputs --> umap_docs[Umap Documents] - augmented_graph --> umap_entities[Umap Entities] - umap_docs --> combine_nodes[Nodes Table] - umap_entities --> combine_nodes + graph_outputs --> graph_embed[Graph Embedding] + graph_embed --> umap_entities[Umap Entities] + umap_entities --> combine_nodes[Final Nodes] + end + subgraph phase7[Phase 7: Text Embeddings] + textUnits --> text_embed[Text Embedding] + graph_outputs --> description_embed[Description Embedding] + community_report_outputs --> content_embed[Content Embedding] end ``` ## Phase 1: Compose TextUnits -The first phase of the default-configuration workflow is to transform input documents into _TextUnits_. A _TextUnit_ is a chunk of text that is used for our graph extraction techniques. They are also used as source-references by extracted knowledge items in order to empower breadcrumbs and provenance by concepts back to their original source tex. +The first phase of the default-configuration workflow is to transform input documents into _TextUnits_. A _TextUnit_ is a chunk of text that is used for our graph extraction techniques. They are also used as source-references by extracted knowledge items in order to empower breadcrumbs and provenance by concepts back to their original source text. The chunk size (counted in tokens), is user-configurable. By default this is set to 300 tokens, although we've had positive experience with 1200-token chunks using a single "glean" step. (A "glean" step is a follow-on extraction). Larger chunks result in lower-fidelity output and less meaningful reference texts; however, using larger chunks can result in much faster processing time. The group-by configuration is also user-configurable. By default, we align our chunks to document boundaries, meaning that there is a strict 1-to-many relationship between Documents and TextUnits. In rare cases, this can be turned into a many-to-many relationship. This is useful when the documents are very short and we need several of them to compose a meaningful analysis unit (e.g. Tweets or a chat log) -Each of these text-units are text-embedded and passed into the next phase of the pipeline. - ```mermaid --- title: Documents into Text Chunks @@ -95,15 +93,15 @@ flowchart LR ### Entity & Relationship Extraction -In this first step of graph extraction, we process each text-unit in order to extract entities and relationships out of the raw text using the LLM. The output of this step is a subgraph-per-TextUnit containing a list of **entities** with a _name_, _type_, and _description_, and a list of **relationships** with a _source_, _target_, and _description_. +In this first step of graph extraction, we process each text-unit in order to extract entities and relationships out of the raw text using the LLM. The output of this step is a subgraph-per-TextUnit containing a list of **entities** with a _title_, _type_, and _description_, and a list of **relationships** with a _source_, _target_, and _description_. -These subgraphs are merged together - any entities with the same _name_ and _type_ are merged by creating an array of their descriptions. Similarly, any relationships with the same _source_ and _target_ are merged by creating an array of their descriptions. +These subgraphs are merged together - any entities with the same _title_ and _type_ are merged by creating an array of their descriptions. Similarly, any relationships with the same _source_ and _target_ are merged by creating an array of their descriptions. ### Entity & Relationship Summarization Now that we have a graph of entities and relationships, each with a list of descriptions, we can summarize these lists into a single description per entity and relationship. This is done by asking the LLM for a short summary that captures all of the distinct information from each description. This allows all of our entities and relationships to have a single concise description. -### Claim Extraction & Emission +### Claim Extraction (optional) Finally, as an independent workflow, we extract claims from the source TextUnits. These claims represent positive factual statements with an evaluated status and time-bounds. These get exported as a primary artifact called **Covariates**. @@ -111,27 +109,23 @@ Note: claim extraction is _optional_ and turned off by default. This is because ## Phase 3: Graph Augmentation -Now that we have a usable graph of entities and relationships, we want to understand their community structure and augment the graph with additional information. This is done in two steps: _Community Detection_ and _Graph Embedding_. These give us explicit (communities) and implicit (embeddings) ways of understanding the topological structure of our graph. +Now that we have a usable graph of entities and relationships, we want to understand their community structure. These give us explicit ways of understanding the topological structure of our graph. ```mermaid --- title: Graph Augmentation --- flowchart LR - cd[Leiden Hierarchical Community Detection] --> ge[Node2Vec Graph Embedding] --> ag[Graph Table Emission] + cd[Leiden Hierarchical Community Detection] --> ag[Graph Tables] ``` ### Community Detection In this step, we generate a hierarchy of entity communities using the Hierarchical Leiden Algorithm. This method will apply a recursive community-clustering to our graph until we reach a community-size threshold. This will allow us to understand the community structure of our graph and provide a way to navigate and summarize the graph at different levels of granularity. -### Graph Embedding +### Graph Tables -In this step, we generate a vector representation of our graph using the Node2Vec algorithm. This will allow us to understand the implicit structure of our graph and provide an additional vector-space in which to search for related concepts during our query phase. - -### Graph Tables Emission - -Once our graph augmentation steps are complete, the final **Entities** and **Relationships** tables are exported after their text fields are text-embedded. +Once our graph augmentation steps are complete, the final **Entities**, **Relationships**, and **Communities** tables are exported. ## Phase 4: Community Summarization @@ -140,10 +134,10 @@ Once our graph augmentation steps are complete, the final **Entities** and **Rel title: Community Summarization --- flowchart LR - sc[Generate Community Reports] --> ss[Summarize Community Reports] --> ce[Community Embedding] --> co[Community Tables Emission] + sc[Generate Community Reports] --> ss[Summarize Community Reports] --> co[Community Reports Table] ``` -At this point, we have a functional graph of entities and relationships, a hierarchy of communities for the entities, as well as node2vec embeddings. +At this point, we have a functional graph of entities and relationships and a hierarchy of communities for the entities. Now we want to build on the communities data and generate reports for each community. This gives us a high-level understanding of the graph at several points of graph granularity. For example, if community A is the top-level community, we'll get a report about the entire graph. If the community is lower-level, we'll get a report about a local cluster. @@ -155,13 +149,9 @@ In this step, we generate a summary of each community using the LLM. This will a In this step, each _community report_ is then summarized via the LLM for shorthand use. -### Community Embedding +### Community Reports Table -In this step, we generate a vector representation of our communities by generating text embeddings of the community report, the community report summary, and the title of the community report. - -### Community Tables Emission - -At this point, some bookkeeping work is performed and we export the **Communities** and **CommunityReports** tables. +At this point, some bookkeeping work is performed and we export the **Community Reports** tables. ## Phase 5: Document Processing @@ -172,7 +162,7 @@ In this phase of the workflow, we create the _Documents_ table for the knowledge title: Document Processing --- flowchart LR - aug[Augment] --> dp[Link to TextUnits] --> de[Avg. Embedding] --> dg[Document Table Emission] + aug[Augment] --> dp[Link to TextUnits] --> dg[Documents Table] ``` ### Augment with Columns (CSV Only) @@ -183,15 +173,11 @@ If the workflow is operating on CSV data, you may configure your workflow to add In this step, we link each document to the text-units that were created in the first phase. This allows us to understand which documents are related to which text-units and vice-versa. -### Document Embedding - -In this step, we generate a vector representation of our documents using an average embedding of document slices. We re-chunk documents without overlapping chunks, and then generate an embedding for each chunk. We create an average of these chunks weighted by token-count and use this as the document embedding. This will allow us to understand the implicit relationship between documents, and will help us generate a network representation of our documents. - -### Documents Table Emission +### Documents Table At this point, we can export the **Documents** table into the knowledge Model. -## Phase 6: Network Visualization +## Phase 6: Network Visualization (optional) In this phase of the workflow, we perform some steps to support network visualization of our high-dimensional vector spaces within our existing graphs. At this point there are two logical graphs at play: the _Entity-Relationship_ graph and the _Document_ graph. @@ -200,7 +186,27 @@ In this phase of the workflow, we perform some steps to support network visualiz title: Network Visualization Workflows --- flowchart LR - nv[Umap Documents] --> ne[Umap Entities] --> ng[Nodes Table Emission] + ag[Graph Table] --> ge[Node2Vec Graph Embedding] --> ne[Umap Entities] --> ng[Nodes Table] ``` -For each of the logical graphs, we perform a UMAP dimensionality reduction to generate a 2D representation of the graph. This will allow us to visualize the graph in a 2D space and understand the relationships between the nodes in the graph. The UMAP embeddings are then exported as a table of _Nodes_. The rows of this table include a discriminator indicating whether the node is a document or an entity, and the UMAP coordinates. +### Graph Embedding + +In this step, we generate a vector representation of our graph using the Node2Vec algorithm. This will allow us to understand the implicit structure of our graph and provide an additional vector-space in which to search for related concepts during our query phase. + +### Dimensionality Reduction + +For each of the logical graphs, we perform a UMAP dimensionality reduction to generate a 2D representation of the graph. This will allow us to visualize the graph in a 2D space and understand the relationships between the nodes in the graph. The UMAP embeddings are then exported as a table of _Nodes_. The rows of this table include the UMAP dimensions as x/y coordinates. + +## Phase 7: Text Embedding + +For all artifacts that require downstream vector search, we generate text embeddings as a final step. These embeddings are written directly to a configured vector store. By default we embed entity descriptions, text unit text, and community report text. + +```mermaid +--- +title: Text Embedding Workflows +--- +flowchart LR + textUnits[Text Units] --> text_embed[Text Embedding] + graph_outputs[Graph Tables] --> description_embed[Description Embedding] + community_report_outputs[Community Reports] --> content_embed[Content Embedding] +``` diff --git a/docs/index/overview.md b/docs/index/overview.md index 32179e12..1bc572f4 100644 --- a/docs/index/overview.md +++ b/docs/index/overview.md @@ -10,7 +10,7 @@ Indexing Pipelines are configurable. They are composed of workflows, standard an - embed entities into a graph vector space - embed text chunks into a textual vector space -The outputs of the pipeline can be stored in a variety of formats, including JSON and Parquet - or they can be handled manually via the Python API. +The outputs of the pipeline are stored as Parquet tables by default, and embeddings are written to your configured vector store. ## Getting Started @@ -18,7 +18,6 @@ The outputs of the pipeline can be stored in a variety of formats, including JSO See the [requirements](../developing.md#requirements) section in [Get Started](../get_started.md) for details on setting up a development environment. -The Indexing Engine can be used in either a default configuration mode or with a custom pipeline. To configure GraphRAG, see the [configuration](../config/overview.md) documentation. After you have a config file you can run the pipeline using the CLI or the Python API. @@ -29,12 +28,6 @@ After you have a config file you can run the pipeline using the CLI or the Pytho ```bash # Via Poetry poetry run poe cli --root # default config mode -poetry run poe cli --config your_pipeline.yml # custom config mode - -# Via Node -yarn run:index --root # default config mode -yarn run:index --config your_pipeline.yml # custom config mode - ``` ### Python API