<p>The primary configuration sections for Indexing Engine pipelines are described below. Each configuration section can be expressed in Python (for use in Python API mode) as well as YAML, but YAML is show here for brevity.</p>
<p>Using custom configuration is an advanced use-case. Most users will want to use the <ahref="/graphrag/posts/config/overview">Default Configuration</a> instead.</p>
<p>The <ahref="https://github.com/microsoft/graphrag/blob/main/examples/">examples</a> directory contains several examples of how to use the indexing engine with <em>custom configuration</em>.</p>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
</button>
</div>
<h1>> root_dir</h1>
<p>This configuration allows you to set the root directory for the pipeline. All data inputs and outputs are assumed to be relative to this path.</p>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
</button>
</div>
<h1>> storage</h1>
<p>This configuration allows you define the output strategy for the pipeline.</p>
<ul>
<li><code>type</code>: The type of storage to use. Options are <code>file</code>, <code>memory</code>, and <code>blob</code></li>
<li><code>base_dir</code> (<code>type: file</code> only): The base directory to store the data in. This is relative to the config root.</li>
<li><code>connection_string</code> (<code>type: blob</code> only): The connection string to use for blob storage.</li>
<li><code>container_name</code> (<code>type: blob</code> only): The container to use for blob storage.</li>
</ul>
<h1>> cache</h1>
<p>This configuration allows you define the cache strategy for the pipeline.</p>
<ul>
<li><code>type</code>: The type of cache to use. Options are <code>file</code> and <code>memory</code>, and <code>blob</code>.</li>
<li><code>base_dir</code> (<code>type: file</code> only): The base directory to store the cache in. This is relative to the config root.</li>
<li><code>connection_string</code> (<code>type: blob</code> only): The connection string to use for blob storage.</li>
<li><code>container_name</code> (<code>type: blob</code> only): The container to use for blob storage.</li>
</ul>
<h1>> reporting</h1>
<p>This configuration allows you define the reporting strategy for the pipeline. Report files are generated artifacts that summarize the performance metrics of the pipeline and emit any error messages.</p>
<ul>
<li><code>type</code>: The type of reporting to use. Options are <code>file</code>, <code>memory</code>, and <code>blob</code></li>
<li><code>base_dir</code> (<code>type: file</code> only): The base directory to store the reports in. This is relative to the config root.</li>
<li><code>connection_string</code> (<code>type: blob</code> only): The connection string to use for blob storage.</li>
<li><code>container_name</code> (<code>type: blob</code> only): The container to use for blob storage.</li>
</ul>
<h1>> workflows</h1>
<p>This configuration section defines the workflow DAG for the pipeline. Here we define an array of workflows and express their inter-dependencies in steps:</p>
<ul>
<li><code>name</code>: The name of the workflow. This is used to reference the workflow in other parts of the config.</li>
<li><code>steps</code>: The DataShaper steps that this workflow comprises. If a step defines an input in the form of <code>workflow:<workflow_name></code>, then it is assumed to have a dependency on the output of that workflow.</li>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
</button>
</div>
<h1>> input</h1>
<ul>
<li><code>type</code>: The input type field discriminates between the different input types. Options are <code>csv</code> and <code>text</code>.</li>
<li><code>base_dir</code>: The base directory to read the input files from. This is relative to the config file.</li>
<li><code>file_pattern</code>: A regex to match the input files. The regex must have named groups for each of the fields in the file_filter.</li>
<li><code>post_process</code>: A DataShaper workflow definition to apply to the input before executing the primary workflow.</li>
<li><code>source_column</code> (<code>type: csv</code> only): The column containing the source/author of the data</li>
<li><code>text_column</code> (<code>type: csv</code> only): The column containing the text of the data</li>
<li><code>timestamp_column</code> (<code>type: csv</code> only): The column containing the timestamp of the data</li>
<li><code>timestamp_format</code> (<code>type: csv</code> only): The format of the timestamp</li>
<spanclass="token key atrule">base_dir</span><spanclass="token punctuation">:</span> ../data/csv <spanclass="token comment"># the directory containing the CSV files, this is relative to the config file</span>
<spanclass="token key atrule">file_pattern</span><spanclass="token punctuation">:</span><spanclass="token string">'.*[\/](?P<source>[^\/]+)[\/](?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})_(?P<author>[^_]+)_\d+\.csv$'</span><spanclass="token comment"># a regex to match the CSV files</span>
<spanclass="token comment"># An additional file filter which uses the named groups from the file_pattern to further filter the files</span>
<spanclass="token key atrule">source_column</span><spanclass="token punctuation">:</span><spanclass="token string">"author"</span><spanclass="token comment"># the column containing the source/author of the data</span>
<spanclass="token key atrule">text_column</span><spanclass="token punctuation">:</span><spanclass="token string">"message"</span><spanclass="token comment"># the column containing the text of the data</span>
<spanclass="token key atrule">timestamp_column</span><spanclass="token punctuation">:</span><spanclass="token string">"date(yyyyMMddHHmmss)"</span><spanclass="token comment"># optional, the column containing the timestamp of the data</span>
<spanclass="token key atrule">timestamp_format</span><spanclass="token punctuation">:</span><spanclass="token string">"%Y%m%d%H%M%S"</span><spanclass="token comment"># optional, the format of the timestamp</span>
<spanclass="token key atrule">post_process</span><spanclass="token punctuation">:</span><spanclass="token comment"># Optional, set of steps to process the data before going into the workflow</span>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
<spanclass="token key atrule">type</span><spanclass="token punctuation">:</span> text
<spanclass="token key atrule">base_dir</span><spanclass="token punctuation">:</span> ../data/csv <spanclass="token comment"># the directory containing the CSV files, this is relative to the config file</span>
<spanclass="token key atrule">file_pattern</span><spanclass="token punctuation">:</span><spanclass="token string">'.*[\/](?P<source>[^\/]+)[\/](?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})_(?P<author>[^_]+)_\d+\.csv$'</span><spanclass="token comment"># a regex to match the CSV files</span>
<spanclass="token comment"># An additional file filter which uses the named groups from the file_pattern to further filter the files</span>
<spanclass="token key atrule">post_process</span><spanclass="token punctuation">:</span><spanclass="token comment"># Optional, set of steps to process the data before going into the workflow</span>
<spanstyle="display:inline-block;background:url(https://api.iconify.design/mdi/content-copy.svg) no-repeat center center / contain;width: 16px; height: 16px;"class=""></span>
