Agent scraping HTML to Agent reads llms.txt

Agent scraping HTML to Agent reads llms.txt An architecture diagram generated by Archify. Agent scraping HTML · → 642 pages to crawl → parse DOM, extract te… · Architecture component Agent scraping HTML → 642 pages to crawl → parse DOM, extract te… Agent reads llms.txt · → 1 request, 5.5 KB → structured index → pic… · Architecture component Agent reads llms.txt → 1 request, 5.5 KB → structured index → pic… Legend Frontend Backend

Chain

  • • 2 steps, one path
  • • Edge labels omitted for clarity