Real-world corpora#

A documentation indexer that only works on the toy project in its own test suite is worth nothing. This chapter records what happens on documentation nobody wrote with Sphinx Lens in mind: this project’s own MyST docs, Django, and CPython. Between them they cover MyST and RST, the Python and C domains, custom domains, autosectionlabel, intersphinx, and project-specific extensions.

Treat these as evidence about extraction quality and retrieval behavior. They are no kind of benchmark: the timings come from one machine and mean nothing across machines.

Recorded corpora#

Corpus	Commit	Documents	Sections	Objects	Links	Resolved	Index size
Sphinx Lens	`0a9a8bd` plus this change	10	50	6	43	100%	60.7 KB
Django	`c9eb16a87e60c305fb3651459639f647cce498db`	672	6,128	7,547	22,367	96.9%	18.2 MB
CPython	`998b89020456db591be41e6529b04f4bc8c8181f`	553	5,121	19,623	56,303	98.2%	34.6 MB

“Resolved” combines internal and external links. Django produced 17,773 internal, 3,911 external, and 683 unresolved links. CPython produced 48,198 internal, 7,103 external, and 1,002 unresolved links.

The runs used Python 3.14.4, Sphinx 9.1.0, and cached Sphinx doctrees when available. The observed build times and peak resident memory were 39.9 seconds and 287 MB for Django, and 104.6 seconds and 497 MB for CPython.

Sphinx Lens: MyST and glossary precision#

The project builds its own index without listing sphinx_lens in conf.py:

make lens

All 43 links were classified as internal or external. This specifically checks that MyST document links pass through Sphinx’s resolver instead of remaining raw pending_xref nodes. Reading the glossary object also returns the precise definition rather than the complete configuration page:

$ sphinx-lens read std:term:PYTHONPATH -i docs/_build/lens
PYTHONPATH

Python import search path.
In this project docs, it is used for module execution from source (for example PYTHONPATH=src uv run -m ...).

Django: narrative and ORM API#

Build from Django’s own documentation environment so its package imports and extensions are available:

PYTHONPATH=/tmp/django sphinx-lens build /tmp/django/docs \
  --output /tmp/django-lens

A concept query puts the exact label and document first, followed by the most specific subsection and API objects:

$ sphinx-lens locate "database transactions" -i /tmp/django-lens --limit 5
00  std:label:topics/db/transactions:database transactions
00  topics/db/transactions
90  std:label:topics/db/transactions:managing database transactions
70  py:exception:django.db.transaction.TransactionManagementError
70  py:function:django.db.transaction.atomic

Regex plus domain filtering locates exact Python objects without relying on body ranking:

$ sphinx-lens locate '^django\.db\.models\.(Model|query\.QuerySet)$' \
    --regex --kind object --domain py --json -i /tmp/django-lens
py:class:django.db.models.Model              ref/models/instances#django.db.models.Model
py:class:django.db.models.query.QuerySet     ref/models/querysets#django.db.models.query.QuerySet

The graph reports 17 incoming and 13 outgoing links for Model, and 41 incoming and 3 outgoing links for QuerySet. A cold CLI process loaded the JSON and ran the first query in 1.08 seconds with 97 MB peak resident memory.

CPython: mixed Python and C domains#

PYTHONPATH=/tmp/cpython sphinx-lens build /tmp/cpython/Doc \
  --output /tmp/cpython-lens

The natural-language query returns the canonical glossary term first and then relevant narrative scopes:

$ sphinx-lens locate "global interpreter lock" -i /tmp/cpython-lens --limit 5
00  std:term:global interpreter lock
90  c-api/threads
90  faq/library#can-t-we-get-rid-of-the-global-interpreter-lock
90  howto/free-threading-python#the-global-interpreter-lock-in-free-threaded-python
90  std:label:threads

Two structural queries demonstrate that the same interface spans domains:

$ sphinx-lens locate '^asyncio\.(Task|TaskGroup)$' --regex --kind object --domain py --json -i /tmp/cpython-lens
py:class:asyncio.Task        library/asyncio-task#asyncio.Task
py:class:asyncio.TaskGroup   library/asyncio-task#asyncio.TaskGroup

$ sphinx-lens locate '^PyObject_(Call|GetAttrString)$' --regex --kind object --domain c --json -i /tmp/cpython-lens
c:function:PyObject_Call            c-api/call#c.PyObject_Call
c:function:PyObject_GetAttrString   c-api/object#c.PyObject_GetAttrString

std:term:global interpreter lock has 25 incoming links. asyncio.Task has 45 incoming and 45 outgoing links. The GIL query took 1.63 seconds and 151 MB peak resident memory in a new CLI process.

Findings and limits#

Sphinx-resolved doctrees handle MyST, contextual domain lookup, and local cross-references more accurately than a Lens-specific resolver. Taking the unresolved set from Sphinx’s own missing-reference event, instead of reconstructing it by matching labels, removed a further 104 false negatives on Django and 1,233 on CPython.
Storing only each entry’s own text reduced the large-corpus artifacts while keeping read able to compose complete scopes.
JSON is parsed in full on every open, so a cold process pays roughly a second on these corpora before answering anything.
Normalized astext() output does not distinguish prose, code, tables, and admonitions.
Remaining unresolved links include intersphinx and extension-specific targets that do not resolve into a local physical location.

Real-world corpora

Contents