# Extractor Roadmap

This work covers issues #83, #88, #91, #92, #93, #97 and related issues
#84, #89, #90, #94 and #95.

## Extraction Contract

URL downloads, local files and browser-supplied HTML must share content
extraction and Markdown normalization. A website using a supported format
should work without adding a new Python function for that website.

1. Fetch HTML, or accept supplied HTML with its source URL.
2. Read document metadata and embedded structured content.
3. Select content using a supported format (such as Arc Fusion or schema.org)
   or a small declarative CSS rule backed by a failing fixture.
4. Use Readability when no more reliable content is available.
5. Normalize images, links, figures and structural chrome in one HTML path.
6. Convert to Markdown and derive front matter and preview text.

External APIs belong at the acquisition boundary. Reuse protocols such as
oEmbed across providers; keep API-specific operations such as transcripts
isolated. Do not grow the pipeline with domain conditionals or hide the same
conditionals inside a registry of per-site functions.

## Work Order

1. **Offline benchmark (#91):** assert required and forbidden phrases in the
   publishable Markdown body, measure body length without front matter, and
   check metadata separately. Include ordinary articles, embedded data,
   media, social pages and noisy layouts.
2. **Shared pipeline (#93):** keep acquisition separate from parsing. Use
   plain functions and data tables where they suffice. Supplied HTML must
   exercise the same parsers as downloaded HTML.
3. **Embedded article formats (#88, #97, #94):** parse Fusion JSON and
   schema.org using the standard JSON decoder. Test both source paths and
   more than one hostname to prove format-based extraction.
4. **Media protocols (#92):** share oEmbed requests and rendering across
   Vimeo, Dailymotion and YouTube while retaining transcripts.
5. **Declarative rules (#90, #89):** add selectors only when a fixture shows
   generic extraction losing content. Readeck/FiveFilters are references,
   not rule sets to vendor wholesale.
6. **HTML normalization (#95):** preserve code, tables, footnotes and
   content-bearing callouts; remove known chrome conservatively.
7. **Social content (#83, #93):** use available structured data and shared
   protocols first. Add API-specific logic only for demonstrated gaps.

## Ponytail Criteria For Every PR

- Prefer the standard library and already-installed parsers.
- Keep domain differences in data when they are selectors or endpoints.
- Share normalization across every extraction path.
- Keep readable code and meaningful offline checks in the introducing PR.
- Avoid plugin frameworks, speculative configuration and custom async code.
- Compare alternatives on the same corpus before adding another extractor
  dependency. Browser rendering stays outside this synchronous package.

Reusable extraction stays in `markdown-this`; web and Telegram orchestration
stay in their application packages.