Technical Reference

Technical Reference#

Current Pipeline#

extract_main_content returns (title, markdown, fallback_text, intro).

URL acquisition can use the existing GitHub, arXiv and YouTube integrations or the shared oEmbed client. Otherwise it downloads HTML once.

Downloaded HTML, local files and supplied HTML share this pipeline:

  1. Read metadata and the source URL.

  2. Extract embedded Arc Fusion article data when present, on any hostname.

  3. Select a body using a declarative domain rule when one matches.

  4. Use Readability, with schema.org article text as a fallback for short or missing content. Nested nodes and multiple declared types are supported.

  5. Normalize images, figures and relative links and remove structural chrome.

  6. Convert to Markdown, add front matter and derive preview text.

Pass source_url= with supplied HTML to resolve its relative links. The web service forwards the URL from request metadata for bookmarklet extraction.

Rules in markdown_this.rules declare hostnames, ordered body selectors and selectors to strip inside the body. The first nonempty body wins; nested selectors do not duplicate content. Each default rule needs an offline fixture. Unmarked aside elements are preserved because they may contain article notes.

Captured Post Collections#

A rule can also declare an item_selector, permalink_selector and boundary_selector. The shared collector deduplicates post IDs and selects the contiguous author sequence containing the requested post. A boundary outside the posts stops collection before recommendations. X and Twitter use this rule mechanism; no X-specific fetcher or DOM parser is registered.

Supply rendered HTML with source_url= (the bookmarklet does this already). On an X/Twitter status page it progressively scrolls and retains same-author post cards before the virtualized page removes them. This requires neither an X API token nor a server-side browser. It is best-effort: the capture can only include posts delivered to that browser and author identity cannot prove a reply relationship. Images, video/audio URLs and video posters pass through common normalization. Browser-local blob: URLs are not portable; their posters and post permalinks remain available instead.

The output declares extraction_scope: captured-posts. It contains only the posts available in that HTML snapshot, not a verified complete thread. The collector does not scroll the browser, fetch missing replies or infer reply relationships from author identity alone. Consecutive author posts are a presentation heuristic, not proof of a conversation graph.

Public Rich oEmbed#

Public X/Twitter URLs use the shared oEmbed client without API credentials. Rich responses pass through the same HTML cleanup and media conversion as documents; widget scripts are removed. Failed requests fall back to HTML acquisition through the existing extractor registry.

The result declares extraction_scope: oembed: it contains the post content returned by the provider, not a complete thread. Supplied HTML bypasses URL acquisition, so an oEmbed response cannot replace a captured collection. For a saved capture, retain the original post URL:

uvx markdown-this thread.html --source-url https://x.com/alice/status/1002

Main Modules#

  • markdown_this.extractor: public pipeline orchestration.

  • markdown_this.fetchers: HTTP fetchers and special URL handlers.

  • markdown_this.html: HTML cleanup before Markdown conversion.

  • markdown_this.markdown: Markdown normalization and preview text helpers.

  • markdown_this.metadata: YAML front matter and HTML metadata extraction.

  • markdown_this.structured: embedded article formats, independent of hostnames.

  • markdown_this.rules: declarative domain selectors for known high-value sites.

Testing Notes#

The root pytest configuration enforces 100% coverage for markdown-this and md-to-telegraph. New extraction behavior should add narrow offline fixtures before using live URLs.