Overview

Contents

Overview#

markdown-this extracts readable web content and supported special URLs as Markdown. It is the workspace package that should own content fetching, metadata extraction, HTML cleanup, and Markdown normalization.

Goals#

  • Produce clean Markdown for articles, media pages, source-code pages, and local HTML.

  • Preserve source metadata such as title, author, canonical URL, publication date, image, and page type.

  • Recover article text from schema.org JSON-LD when the generic extractor selects too little content.

  • Apply small declarative domain rules for high-value sites with verified fixtures.

  • Keep extraction deterministic and testable with offline fixtures.

  • Prefer small, verified provider/rule support over a large unowned rule dump.

Functionality#

  • Generic HTML extraction with readability-lxml.

  • GitHub repository README and Markdown blob extraction through the GitHub API.

  • arXiv abstract extraction.

  • YouTube metadata, description, and optional transcript extraction.

  • Vimeo and Dailymotion media-page extraction through oEmbed.

  • Public X/Twitter posts through rich oEmbed, and captured post collections through declarative selectors. Both expose their limited extraction scope.

  • Substack body selection through a declarative rule.

  • Image and figure normalization before Markdown conversion.

Contents