mirror of
https://github.com/datahub-project/datahub.git
synced 2025-06-27 05:03:31 +00:00
docs: use sphinx-markdown-builder for sdk doc generation (#13721)
This commit is contained in:
parent
f3c49c3174
commit
e24ef15e77
@ -746,11 +746,26 @@ module.exports = {
|
|||||||
items: [
|
items: [
|
||||||
"metadata-ingestion/as-a-library",
|
"metadata-ingestion/as-a-library",
|
||||||
{
|
{
|
||||||
"Python SDK Reference": [
|
type: "category",
|
||||||
|
label: "SDK Reference",
|
||||||
|
items: [
|
||||||
{
|
{
|
||||||
type: "autogenerated",
|
type: "category",
|
||||||
dirName: "python-sdk",
|
label: "Builder",
|
||||||
|
items: [{ type: "autogenerated", dirName: "python-sdk/builder" }],
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
type: "category",
|
||||||
|
label: "Clients",
|
||||||
|
items: [{ type: "autogenerated", dirName: "python-sdk/clients" }],
|
||||||
|
},
|
||||||
|
{
|
||||||
|
type: "category",
|
||||||
|
label: "SDK V2",
|
||||||
|
items: [{ type: "autogenerated", dirName: "python-sdk/sdk-v2" }],
|
||||||
|
},
|
||||||
|
"python-sdk/models",
|
||||||
|
"python-sdk/urns",
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
|
@ -26,10 +26,10 @@ $(VENV_SENTINEL): requirements.txt
|
|||||||
|
|
||||||
# Not using Python's http.server because it enables caching headers.
|
# Not using Python's http.server because it enables caching headers.
|
||||||
serve:
|
serve:
|
||||||
serve -p 3001 _build/html/
|
serve -p 3001 _build/markdown/
|
||||||
|
|
||||||
md: html
|
md: venv
|
||||||
$(VENV_DIR)/bin/python3 convert_sphinx_to_docusaurus.py
|
@$(SPHINXBUILD) -M markdown "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) && $(VENV_DIR)/bin/python3 convert_sphinx_to_docusaurus.py
|
||||||
|
|
||||||
# Route other targets to Sphinx using the new
|
# Route other targets to Sphinx using the new
|
||||||
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
|
||||||
|
@ -1,10 +0,0 @@
|
|||||||
Builder
|
|
||||||
=======
|
|
||||||
|
|
||||||
These classes and methods make it easier to construct MetadataChangeProposals and MetadataChangeEvents.
|
|
||||||
|
|
||||||
.. automodule:: datahub.emitter.mcp
|
|
||||||
|
|
||||||
.. automodule:: datahub.emitter.mce_builder
|
|
||||||
|
|
||||||
.. automodule:: datahub.emitter.mcp_builder
|
|
7
docs-website/sphinx/apidocs/builder/mce-builder.rst
Normal file
7
docs-website/sphinx/apidocs/builder/mce-builder.rst
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
MCE Builder
|
||||||
|
=======
|
||||||
|
|
||||||
|
These classes and methods make it easier to construct MetadataChangeEvents.
|
||||||
|
|
||||||
|
.. automodule:: datahub.emitter.mce_builder
|
||||||
|
:member-order: alphabetical
|
10
docs-website/sphinx/apidocs/builder/mcp-builder.rst
Normal file
10
docs-website/sphinx/apidocs/builder/mcp-builder.rst
Normal file
@ -0,0 +1,10 @@
|
|||||||
|
MCP Builder
|
||||||
|
=======
|
||||||
|
|
||||||
|
These classes and methods make it easier to construct MetadataChangeProposals.
|
||||||
|
|
||||||
|
.. automodule:: datahub.emitter.mcp
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.emitter.mcp_builder
|
||||||
|
:member-order: alphabetical
|
@ -1,11 +0,0 @@
|
|||||||
Client
|
|
||||||
======
|
|
||||||
|
|
||||||
The Kafka emitter or Rest emitter can be used to push metadata to DataHub.
|
|
||||||
The DataHub graph client extends the Rest emitter with additional functionality.
|
|
||||||
|
|
||||||
.. automodule:: datahub.emitter.rest_emitter
|
|
||||||
|
|
||||||
.. automodule:: datahub.emitter.kafka_emitter
|
|
||||||
|
|
||||||
.. automodule:: datahub.ingestion.graph.client
|
|
8
docs-website/sphinx/apidocs/clients/graph-client.rst
Normal file
8
docs-website/sphinx/apidocs/clients/graph-client.rst
Normal file
@ -0,0 +1,8 @@
|
|||||||
|
Graph Client
|
||||||
|
======
|
||||||
|
|
||||||
|
The DataHub graph client extends the Rest emitter with additional functionality.
|
||||||
|
|
||||||
|
.. automodule:: datahub.ingestion.graph.client
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
8
docs-website/sphinx/apidocs/clients/kafka-emitter.rst
Normal file
8
docs-website/sphinx/apidocs/clients/kafka-emitter.rst
Normal file
@ -0,0 +1,8 @@
|
|||||||
|
Kafka Emitter
|
||||||
|
======
|
||||||
|
|
||||||
|
The Kafka emitter can be used to push metadata to DataHub.
|
||||||
|
|
||||||
|
.. automodule:: datahub.emitter.kafka_emitter
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
8
docs-website/sphinx/apidocs/clients/rest-emitter.rst
Normal file
8
docs-website/sphinx/apidocs/clients/rest-emitter.rst
Normal file
@ -0,0 +1,8 @@
|
|||||||
|
Rest Emitter
|
||||||
|
======
|
||||||
|
|
||||||
|
The Rest emitter can be used to push metadata to DataHub.
|
||||||
|
|
||||||
|
.. automodule:: datahub.emitter.rest_emitter
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
29
docs-website/sphinx/apidocs/sdk-v2/entities.rst
Normal file
29
docs-website/sphinx/apidocs/sdk-v2/entities.rst
Normal file
@ -0,0 +1,29 @@
|
|||||||
|
Entities
|
||||||
|
=======
|
||||||
|
|
||||||
|
The DataHub SDK provides a set of entities that can be used to interact with DataHub's metadata.
|
||||||
|
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.dataset
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.container
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.mlmodel
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.mlmodelgroup
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.dashboard
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.chart
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.datajob
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.dataflow
|
||||||
|
:member-order: alphabetical
|
7
docs-website/sphinx/apidocs/sdk-v2/entity-client.rst
Normal file
7
docs-website/sphinx/apidocs/sdk-v2/entity-client.rst
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
Entity Client
|
||||||
|
=======
|
||||||
|
|
||||||
|
The DataHub Entity Client provides a client for interacting with DataHub entities.
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.entity_client
|
||||||
|
:member-order: alphabetical
|
7
docs-website/sphinx/apidocs/sdk-v2/lineage-client.rst
Normal file
7
docs-website/sphinx/apidocs/sdk-v2/lineage-client.rst
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
Lineage Client
|
||||||
|
=======
|
||||||
|
|
||||||
|
The DataHub Lineage Client provides a client for searching and retrieving lineage metadata from DataHub.
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.lineage_client
|
||||||
|
:member-order: alphabetical
|
7
docs-website/sphinx/apidocs/sdk-v2/main-client.rst
Normal file
7
docs-website/sphinx/apidocs/sdk-v2/main-client.rst
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
Main Client
|
||||||
|
=======
|
||||||
|
|
||||||
|
The DataHub Main Client provides a client for interacting with DataHub.
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.main_client
|
||||||
|
:member-order: alphabetical
|
7
docs-website/sphinx/apidocs/sdk-v2/resolver-client.rst
Normal file
7
docs-website/sphinx/apidocs/sdk-v2/resolver-client.rst
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
Resolver Client
|
||||||
|
=======
|
||||||
|
|
||||||
|
The DataHub Resolver Client provides a client for resolving entities by their URN.
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.resolver_client
|
||||||
|
:member-order: alphabetical
|
10
docs-website/sphinx/apidocs/sdk-v2/search-client.rst
Normal file
10
docs-website/sphinx/apidocs/sdk-v2/search-client.rst
Normal file
@ -0,0 +1,10 @@
|
|||||||
|
Search Client
|
||||||
|
=======
|
||||||
|
|
||||||
|
The DataHub Search Client provides a client for searching and retrieving metadata from DataHub.
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.search_client
|
||||||
|
:member-order: alphabetical
|
||||||
|
|
||||||
|
.. automodule:: datahub.sdk.search_filters
|
||||||
|
:member-order: alphabetical
|
@ -1,6 +0,0 @@
|
|||||||
DataHub CLI
|
|
||||||
===========
|
|
||||||
|
|
||||||
.. click:: datahub.entrypoints:datahub
|
|
||||||
:prog: datahub
|
|
||||||
:nested: full
|
|
@ -26,8 +26,10 @@ extensions = [
|
|||||||
"sphinx_autodoc_typehints",
|
"sphinx_autodoc_typehints",
|
||||||
# This enables us to autogenerate docs for our CLI.
|
# This enables us to autogenerate docs for our CLI.
|
||||||
"sphinx_click",
|
"sphinx_click",
|
||||||
|
"sphinx_markdown_builder",
|
||||||
]
|
]
|
||||||
|
|
||||||
|
markdown_anchor_sections = True
|
||||||
napoleon_use_param = True
|
napoleon_use_param = True
|
||||||
|
|
||||||
# Move type hint info to function description instead of signature
|
# Move type hint info to function description instead of signature
|
||||||
|
@ -1,78 +1,166 @@
|
|||||||
import pathlib
|
import pathlib
|
||||||
import json
|
import re
|
||||||
from bs4 import BeautifulSoup
|
|
||||||
|
|
||||||
|
|
||||||
SPHINX_ROOT_DIR = pathlib.Path(".")
|
SPHINX_ROOT_DIR = pathlib.Path(".")
|
||||||
SPHINX_BUILD_DIR = SPHINX_ROOT_DIR / pathlib.Path("_build/html/apidocs")
|
SPHINX_BUILD_DIR = SPHINX_ROOT_DIR / "_build/markdown"
|
||||||
DOCS_OUTPUT_DIR = pathlib.Path("../docs/python-sdk")
|
DOCS_OUTPUT_DIR = pathlib.Path("../docs/python-sdk")
|
||||||
|
|
||||||
|
HTML_TAGS = {
|
||||||
|
"html", "head", "title", "base", "link", "meta", "style", "script", "noscript",
|
||||||
|
"body", "section", "nav", "article", "aside", "h1", "h2", "h3", "h4", "h5", "h6",
|
||||||
|
"header", "footer", "address", "p", "hr", "pre", "blockquote", "ol", "ul", "li",
|
||||||
|
"dl", "dt", "dd", "figure", "figcaption", "div", "main", "a", "em", "strong",
|
||||||
|
"small", "s", "cite", "q", "dfn", "abbr", "data", "time", "code", "var", "samp",
|
||||||
|
"kbd", "sub", "sup", "i", "b", "u", "mark", "ruby", "rt", "rp", "bdi", "bdo",
|
||||||
|
"span", "br", "wbr", "ins", "del", "img", "iframe", "embed", "object", "param",
|
||||||
|
"video", "audio", "track", "canvas", "map", "area", "svg", "math",
|
||||||
|
"table", "caption", "colgroup", "col", "tbody", "thead", "tfoot", "tr", "td", "th",
|
||||||
|
"form", "fieldset", "legend", "label", "button", "select", "datalist",
|
||||||
|
"optgroup", "option", "textarea", "output", "progress", "meter", "details",
|
||||||
|
"summary", "dialog", "template", "slot", "portal"
|
||||||
|
}
|
||||||
|
|
||||||
def html_to_mdx(html: str) -> str:
|
REPLACEMENTS = [
|
||||||
# Because the HTML uses `class` and has `{}` in it, it isn't valid
|
("<function ", "<function "),
|
||||||
# MDX. As such, we use React's dangerouslySetInnerHTML.
|
("<disabled ", "<disabled "),
|
||||||
return f"""
|
("MDXContent.isMDXComponent = true", ""),
|
||||||
|
(".md#", ".mdx#"),
|
||||||
|
]
|
||||||
|
|
||||||
<div dangerouslySetInnerHTML={{{{__html: {json.dumps(html)}}}}}></div>
|
# ---- CLEAN HTML DANGEROUS TAGS ----
|
||||||
|
def sanitize_mdx_unsafe_tags(content: str) -> str:
|
||||||
|
return re.sub(
|
||||||
|
r"<([a-zA-Z0-9_-]+)>",
|
||||||
|
lambda m: f"<{m.group(1)}>" if m.group(1).lower() in HTML_TAGS else f"<{m.group(1)}>",
|
||||||
|
content
|
||||||
|
)
|
||||||
|
|
||||||
"""
|
# ---- REPAIR BROKEN MARKDOWN BOLD ----
|
||||||
|
def repair_broken_emphasis(content: str) -> str:
|
||||||
|
content = re.sub(r'\[\*\*([^\*]+)\*\s+\*', r'[**\1**', content)
|
||||||
|
content = re.sub(r'\*\*([^\*]+)\*\s+\*\]', r'**\1**]', content)
|
||||||
|
content = re.sub(r'\*\*([^\*]+)\*\s*,\s+\*\s*([^\*]+)\s*\*', r'**\1**, **\2**', content)
|
||||||
|
content = re.sub(r'\*\s*\*([^\*]+)\*\s+\*\]', r'**\1**]', content)
|
||||||
|
return content
|
||||||
|
|
||||||
|
# ---- ONLY USED INSIDE SECTION HEADINGS ----
|
||||||
|
def convert_md_link_to_html(arg_str: str) -> str:
|
||||||
|
# convert markdown links inside argument types into plain text fallback
|
||||||
|
return re.sub(
|
||||||
|
r'\[([^\]]+)\]\([^)]+\)',
|
||||||
|
r'<code>\1</code>',
|
||||||
|
arg_str
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def bs4_to_mdx(soup: BeautifulSoup) -> str:
|
# ---- ARGUMENT PARSER ----
|
||||||
# TODO: Eventually we should do something smarter here to
|
def parse_args(arg_str: str) -> str:
|
||||||
# generate something that's closer to real Markdown. This would
|
if not arg_str.strip():
|
||||||
# be helpful, for example, for enabling Docusaurus to generate
|
return ""
|
||||||
# a table of contents for the page.
|
|
||||||
return html_to_mdx(str(soup))
|
|
||||||
|
|
||||||
|
parts = []
|
||||||
|
for arg in arg_str.split(","):
|
||||||
|
arg = arg.strip().replace("\\", "")
|
||||||
|
if arg == "*":
|
||||||
|
parts.append("*")
|
||||||
|
continue
|
||||||
|
|
||||||
def convert_html_to_md(html_file: pathlib.Path) -> str:
|
for pattern, template in [
|
||||||
html = html_file.read_text()
|
(r"([\w_]+)\s*:\s*([^=]+)\s*=\s*(.+)", r'<span class="arg-name">\1</span>: <span class="arg-type">\2</span> = <span class="arg-default">\3</span>'),
|
||||||
soup = BeautifulSoup(html, "html.parser")
|
(r"([\w_]+)\s*=\s*(.+)", r'<span class="arg-name">\1</span> = <span class="arg-default">\2</span>'),
|
||||||
|
(r"([\w_]+)\s*:\s*(.+)", r'<span class="arg-name">\1</span>: <span class="arg-type">\2</span>')
|
||||||
|
]:
|
||||||
|
m = re.match(pattern, arg)
|
||||||
|
if m:
|
||||||
|
parts.append(m.expand(template))
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
parts.append(f'<span class="arg-name">{arg}</span>')
|
||||||
|
|
||||||
body = soup.find("main").find("div", {"class": "bd-article-container"})
|
parsed = ", ".join(parts)
|
||||||
article = body.find("article")
|
parsed = convert_md_link_to_html(parsed)
|
||||||
|
return parsed
|
||||||
|
|
||||||
# Remove all the "permalink to this heading" links.
|
# ---- HEADING PARSER ----
|
||||||
for link in article.find_all("a", {"class": "headerlink"}):
|
def parse_heading(text: str):
|
||||||
link.decompose()
|
match = re.match(r"(?:\*class\*\s+)?([\w\.]+)\.([\w]+)(?:\((.*)\))?", text)
|
||||||
|
if match:
|
||||||
|
owner, name, args = match.groups()
|
||||||
|
parsed_args = parse_args(args or "")
|
||||||
|
prefix = '<span class="class-text">class</span> ' if "*class*" in text else ""
|
||||||
|
heading = f'{prefix}<span class="class-owner">{owner}.</span><span class="class-name">{name}</span>'
|
||||||
|
heading += f"({parsed_args})" if parsed_args else "()"
|
||||||
|
slug = f"{owner}.{name}"
|
||||||
|
return name, heading, slug
|
||||||
|
|
||||||
# Remove the trailing " – " from arguments that are missing
|
match = re.match(r"([\w]+)(?:\((.*)\))?", text)
|
||||||
# a description.
|
if match:
|
||||||
for item in article.select("dl.field-list dd p"):
|
name, args = match.groups()
|
||||||
# Note - that's U+2013, not a normal hyphen.
|
parsed_args = parse_args(args or "")
|
||||||
if str(item).endswith(" – </p>"):
|
heading = f'<span class="class-name">{name}</span>'
|
||||||
parent = item.parent
|
heading += f"({parsed_args})" if parsed_args else "()"
|
||||||
# print("orig item", item)
|
return name, heading, name
|
||||||
new_item = BeautifulSoup(str(item)[:-7] + "</p>", "html.parser")
|
|
||||||
# print("new-item", str(new_item))
|
|
||||||
parent.p.replace_with(new_item)
|
|
||||||
# print("item post replace", parent)
|
|
||||||
|
|
||||||
# Extract title from the h1.
|
return text, text, text
|
||||||
title_element = article.find("h1")
|
|
||||||
title = title_element.text
|
|
||||||
title_element.decompose()
|
|
||||||
|
|
||||||
# TODO - generate nicer slugs for these pages
|
# ---- SECTION WRAPPER ----
|
||||||
md_meta = f"""---
|
def wrap_section_blocks(content: str, class_name: str) -> str:
|
||||||
title: {title}
|
lines = content.splitlines()
|
||||||
---\n\n"""
|
out = []
|
||||||
|
inside = False
|
||||||
|
|
||||||
return md_meta + bs4_to_mdx(article)
|
for line in lines:
|
||||||
|
m = re.match(r"^### (.+)$", line)
|
||||||
|
if m:
|
||||||
|
if inside:
|
||||||
|
out.append("\n\n</div>\n\n")
|
||||||
|
|
||||||
|
name, heading, slug = parse_heading(m.group(1))
|
||||||
|
out.append(f'\n\n### <span className="visually-hidden">{name}</span> {{#{slug}}}\n\n')
|
||||||
|
out.append(f'<div className="{class_name}">\n')
|
||||||
|
out.append(f'<div className="section-heading">{heading}<a href="#{slug}" class="hash-link"></a></div>\n')
|
||||||
|
inside = True
|
||||||
|
else:
|
||||||
|
out.append(line)
|
||||||
|
|
||||||
|
if inside:
|
||||||
|
out.append("\n\n</div>\n\n")
|
||||||
|
|
||||||
|
return "\n".join(out)
|
||||||
|
|
||||||
|
# ---- PARAMETER DASH FIX ----
|
||||||
|
def fix_parameter_dash(content: str) -> str:
|
||||||
|
return re.sub(r'(\*\s+\*\*[\w]+?\*\*\s+\([^\)]*\))\s+–\s*(?=\n|\r|\Z)', r'\1', content)
|
||||||
|
|
||||||
|
# ---- FILE CONVERTER ----
|
||||||
|
def convert_file(doc: pathlib.Path, outfile: pathlib.Path):
|
||||||
|
content = doc.read_text()
|
||||||
|
|
||||||
|
for old, new in REPLACEMENTS:
|
||||||
|
content = content.replace(old, new)
|
||||||
|
|
||||||
|
content = sanitize_mdx_unsafe_tags(content)
|
||||||
|
content = repair_broken_emphasis(content)
|
||||||
|
content = wrap_section_blocks(content, "h3-block")
|
||||||
|
content = fix_parameter_dash(content)
|
||||||
|
|
||||||
|
title_match = re.search(r"^# (.+)$", content, re.MULTILINE)
|
||||||
|
title = title_match.group(1).strip() if title_match else doc.stem
|
||||||
|
content = re.sub(r"^# .+\n?", "", content, count=1, flags=re.MULTILINE)
|
||||||
|
|
||||||
|
final = f"---\ntitle: {title}\n---\n<div className=\"sphinx-api-docs\">\n{content.strip()}\n</div>\n"
|
||||||
|
|
||||||
|
outfile.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
outfile.write_text(final)
|
||||||
|
print(f"Generated {outfile}")
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
DOCS_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
DOCS_OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
for doc in SPHINX_BUILD_DIR.glob("**/*.md"):
|
||||||
for doc in SPHINX_BUILD_DIR.glob("**/*.html"):
|
if doc.stem == "index":
|
||||||
md = convert_html_to_md(doc)
|
continue
|
||||||
|
outfile = DOCS_OUTPUT_DIR / doc.relative_to(SPHINX_BUILD_DIR / "apidocs").with_suffix(".mdx")
|
||||||
outfile = DOCS_OUTPUT_DIR / doc.relative_to(SPHINX_BUILD_DIR).with_suffix(".md")
|
convert_file(doc, outfile)
|
||||||
outfile.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
outfile.write_text(md)
|
|
||||||
|
|
||||||
print(f"Generated {outfile}")
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
||||||
|
@ -5,6 +5,7 @@ sphinx-click==4.4.0
|
|||||||
sphinx_autodoc_typehints==1.22
|
sphinx_autodoc_typehints==1.22
|
||||||
pydata-sphinx-theme==0.13.1
|
pydata-sphinx-theme==0.13.1
|
||||||
snowballstemmer>=2.2,<3 # Fixes https://github.com/sphinx-doc/sphinx/issues/13533
|
snowballstemmer>=2.2,<3 # Fixes https://github.com/sphinx-doc/sphinx/issues/13533
|
||||||
|
sphinx-markdown-builder==0.6.8
|
||||||
|
|
||||||
# Because of https://github.com/pydata/pydata-sphinx-theme/issues/108
|
# Because of https://github.com/pydata/pydata-sphinx-theme/issues/108
|
||||||
accessible-pygments
|
accessible-pygments
|
||||||
|
@ -1,124 +1,108 @@
|
|||||||
// Styles for Sphinx Python SDK generated docs
|
.sphinx-api-docs {
|
||||||
$borderRadius: 5px;
|
font-size: 15px;
|
||||||
|
color: var(--ifm-font-color-base);
|
||||||
|
|
||||||
dl.py {
|
.anchor {
|
||||||
margin-bottom: calc(var(--ifm-spacing-vertical) * 2);
|
.visually-hidden {
|
||||||
font-size: 14px;
|
clip: rect(0 0 0 0);
|
||||||
border: 1px solid var(--ifm-hr-border-color);
|
clip-path: inset(50%);
|
||||||
border-radius: $borderRadius;
|
height: 10px;
|
||||||
|
width: 3rem;
|
||||||
code {
|
overflow: hidden;
|
||||||
border: none;
|
white-space: nowrap;
|
||||||
background: none;
|
padding: 0;
|
||||||
}
|
margin: 0;
|
||||||
|
|
||||||
p {
|
|
||||||
margin-bottom: 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
dl {
|
|
||||||
margin-bottom: var(--ifm-spacing-vertical);
|
|
||||||
}
|
|
||||||
|
|
||||||
// The parameter name:
|
|
||||||
em.sig-param > span:first-child {
|
|
||||||
font-weight: bold;
|
|
||||||
}
|
|
||||||
|
|
||||||
> dd:not(:empty) {
|
|
||||||
padding-bottom: var(--ifm-spacing-vertical);
|
|
||||||
}
|
|
||||||
|
|
||||||
dt.sig {
|
|
||||||
box-sizing: border-box;
|
|
||||||
font-size: 0.9rem;
|
|
||||||
padding: var(--ifm-spacing-vertical);
|
|
||||||
border-radius: $borderRadius;
|
|
||||||
font-family: var(--ifm-font-family-monospace);
|
|
||||||
background-color: var(--ifm-background-surface-color);
|
|
||||||
}
|
|
||||||
|
|
||||||
> dd {
|
|
||||||
&:not(:empty) {
|
|
||||||
padding-top: calc(var(--ifm-spacing-vertical) / 2);
|
|
||||||
margin-top: 0;
|
|
||||||
margin-left: var(--ifm-spacing-horizontal);
|
|
||||||
margin-right: var(--ifm-spacing-horizontal);
|
|
||||||
}
|
}
|
||||||
|
.hash-link { display: none; }
|
||||||
}
|
}
|
||||||
|
|
||||||
// e.g. `class`, or `def`
|
.h3-block {
|
||||||
em.property {
|
border: 1px solid var(--ifm-hr-border-color);
|
||||||
color: var(--ifm-font-color-base);
|
border-radius: 8px;
|
||||||
font-weight: bold;
|
overflow: hidden;
|
||||||
}
|
|
||||||
|
|
||||||
// e.g. `MyClass`
|
.section-heading {
|
||||||
span.sig-name {
|
background-color: var(--ifm-color-primary-opaque);
|
||||||
color: #2774b3;
|
|
||||||
font-weight: bold;
|
|
||||||
}
|
|
||||||
|
|
||||||
// e.g classmethod
|
|
||||||
em.property {
|
|
||||||
color: #66b327;
|
|
||||||
}
|
|
||||||
|
|
||||||
em.sig-param {
|
|
||||||
span.default_value {
|
|
||||||
color: #66b327;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
span.sig-return {
|
|
||||||
span.sig-return-typehint {
|
|
||||||
color: var(--ifm-font-color-base);
|
|
||||||
|
|
||||||
pre {
|
|
||||||
color: var(--ifm-font-color-base);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
dl.field-list {
|
|
||||||
padding-top: calc(var(--ifm-spacing-vertical) / 2);
|
|
||||||
display: grid;
|
|
||||||
grid-template-columns: fit-content(30%) auto;
|
|
||||||
&:not(:first-child) {
|
|
||||||
border-top: 1px solid var(--ifm-hr-border-color);
|
|
||||||
}
|
|
||||||
|
|
||||||
dt {
|
|
||||||
margin-right: 0.5em;
|
|
||||||
}
|
|
||||||
|
|
||||||
dd {
|
|
||||||
font-family: var(--ifm-font-family-monospace);
|
font-family: var(--ifm-font-family-monospace);
|
||||||
|
padding: 1rem 1.5rem;
|
||||||
|
font-weight: 700;
|
||||||
|
font-size: 1rem;
|
||||||
|
border-bottom: 1px solid var(--ifm-hr-border-color);
|
||||||
|
|
||||||
|
.class-text { color: #aaa; margin-right: 0.3rem; }
|
||||||
|
.class-owner { color: var(--ifm-font-color-secondary); }
|
||||||
|
.class-name { color: var(--ifm-color-primary); font-weight: 700; }
|
||||||
|
.arg-name { color: var(--ifm-font-color-base); font-weight: 600; }
|
||||||
|
.arg-type { color: gray; }
|
||||||
|
.arg-default { color: #999; font-weight: 600; }
|
||||||
}
|
}
|
||||||
|
|
||||||
dt,
|
p {
|
||||||
dd {
|
padding: 1rem 1.5rem;
|
||||||
margin-left: 0;
|
}
|
||||||
padding-left: 0;
|
|
||||||
|
|
||||||
&:not(:first-of-type) {
|
h4 {
|
||||||
border-top: 1px solid var(--ifm-hr-border-color);
|
background: #FAFAFA;
|
||||||
padding-top: var(--ifm-spacing-vertical);
|
font-family: var(--ifm-font-family-monospace);
|
||||||
}
|
padding: 1rem 1.5rem;
|
||||||
&:not(:last-of-type) {
|
margin: 0;
|
||||||
padding-bottom: var(--ifm-spacing-vertical);
|
font-weight: 600;
|
||||||
|
border-top: 1px solid var(--ifm-hr-border-color);
|
||||||
|
border-bottom: 1px solid var(--ifm-hr-border-color);
|
||||||
|
color: #444;
|
||||||
|
|
||||||
|
em {
|
||||||
|
color: gray;
|
||||||
|
font-style: normal;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
ul {
|
ul {
|
||||||
list-style-type: none;
|
margin: 0;
|
||||||
padding-left: 0;
|
padding: 1rem 2rem;
|
||||||
li {
|
background: #FDFDFD;
|
||||||
p {
|
|
||||||
margin: 0;
|
li {
|
||||||
padding: 0;
|
display: flex;
|
||||||
}
|
flex-wrap: wrap;
|
||||||
|
margin-bottom: 0.75rem;
|
||||||
|
|
||||||
|
.param-name {
|
||||||
|
min-width: 160px;
|
||||||
|
font-weight: 600;
|
||||||
|
color: var(--ifm-color-primary-darker);
|
||||||
|
}
|
||||||
|
|
||||||
|
.param-type {
|
||||||
|
font-family: var(--ifm-font-family-monospace);
|
||||||
|
font-weight: 400;
|
||||||
|
color: #444;
|
||||||
|
background: #F5F5F5;
|
||||||
|
padding: 0.15rem 0.4rem;
|
||||||
|
border-radius: 4px;
|
||||||
|
margin-left: 0.5rem;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// pre code block (full examples)
|
||||||
|
pre code, pre code * {
|
||||||
|
color: var(--ifm-font-color-secondary);
|
||||||
|
padding: 1rem;
|
||||||
|
margin: 0;
|
||||||
|
font-size: 0.9rem;
|
||||||
|
overflow-x: auto;
|
||||||
|
}
|
||||||
|
|
||||||
|
code {
|
||||||
|
background-color: #F5F5F5;
|
||||||
|
padding: 0.1rem 0.4rem;
|
||||||
|
margin: auto 0.2rem;
|
||||||
|
font-size: 0.9em;
|
||||||
|
border: none;
|
||||||
|
line-height: 1.3;
|
||||||
|
border-radius: 6px;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
18
docs-website/src/theme/Root.js
Normal file
18
docs-website/src/theme/Root.js
Normal file
@ -0,0 +1,18 @@
|
|||||||
|
import React, { useEffect } from 'react';
|
||||||
|
import { useLocation } from '@docusaurus/router';
|
||||||
|
|
||||||
|
export default function Root({ children }) {
|
||||||
|
const location = useLocation();
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
setTimeout(() => {
|
||||||
|
if (location.hash) {
|
||||||
|
const id = decodeURIComponent(location.hash.substring(1));
|
||||||
|
const el = document.getElementById(id);
|
||||||
|
if (el) el.scrollIntoView();
|
||||||
|
}
|
||||||
|
}, 0);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
return <>{children}</>;
|
||||||
|
}
|
@ -225,7 +225,7 @@ datahub delete --platform snowflake --only-soft-deleted --hard
|
|||||||
|
|
||||||
## Deletes using the SDK and APIs
|
## Deletes using the SDK and APIs
|
||||||
|
|
||||||
The Python SDK's [DataHubGraph](../../python-sdk/clients.md) client supports deletes via the following methods:
|
The Python SDK's [DataHubGraph](../../python-sdk/clients/graph-client.mdx) client supports deletes via the following methods:
|
||||||
|
|
||||||
- `soft_delete_entity`
|
- `soft_delete_entity`
|
||||||
- `hard_delete_entity`
|
- `hard_delete_entity`
|
||||||
|
@ -21,7 +21,7 @@ If you’re using a different database system for which we don’t support colum
|
|||||||
|
|
||||||
## SDK Support
|
## SDK Support
|
||||||
|
|
||||||
Our SDK provides a [`DataHubGraph.parse_sql_lineage()`](../../python-sdk/clients.md#datahub.ingestion.graph.client.DataHubGraph.parse_sql_lineage) method for programmatically parsing SQL queries.
|
Our SDK provides a [`DataHubGraph.parse_sql_lineage()`](../../python-sdk/clients/graph-client.mdx#datahub.ingestion.graph.client.DataHubGraph.parse_sql_lineage) method for programmatically parsing SQL queries.
|
||||||
|
|
||||||
The resulting object contains a `sql_parsing_result.debug_info.confidence_score` field, which is a 0-1 value indicating the confidence of the parser.
|
The resulting object contains a `sql_parsing_result.debug_info.confidence_score` field, which is a 0-1 value indicating the confidence of the parser.
|
||||||
|
|
||||||
|
@ -669,7 +669,7 @@ This is a summary of automatic lineage extraction support in our data source. Pl
|
|||||||
If you’re using a different database system for which we don’t support column-level lineage out of the box, but you do have a database query log available,
|
If you’re using a different database system for which we don’t support column-level lineage out of the box, but you do have a database query log available,
|
||||||
we have a SQL queries connector that generates column-level lineage and detailed table usage statistics from the query log.
|
we have a SQL queries connector that generates column-level lineage and detailed table usage statistics from the query log.
|
||||||
|
|
||||||
If these does not suit your needs, you can use the new `DataHubGraph.parse_sql_lineage()` method in our SDK. (See the source code [here](https://docs.datahub.com/docs/python-sdk/clients/))
|
If these does not suit your needs, you can use the new `DataHubGraph.parse_sql_lineage()` method in our SDK. (See the source code [here](https://docs.datahub.com/docs/python-sdk/clients/graph-client))
|
||||||
|
|
||||||
For more information, refer to the [Extracting Column-Level Lineage from SQL](https://blog.datahubproject.io/extracting-column-level-lineage-from-sql-779b8ce17567)
|
For more information, refer to the [Extracting Column-Level Lineage from SQL](https://blog.datahubproject.io/extracting-column-level-lineage-from-sql-779b8ce17567)
|
||||||
|
|
||||||
|
Loading…
x
Reference in New Issue
Block a user