About our crawler

The Smoko News crawler is a lightweight RSS reader that polls public news feeds to keep the Smoko News front page current. This page exists so publishers can understand what the bot does and how to contact us.

Identification

Every request the crawler makes includes this User-Agent header:

NewsAggBot/1.0 (+https://smokonews.com.au/about/bot; contact=support@smokonews.com.au)

Behaviour

• Conditional requests. The crawler sends ETag and If-Modified-Since headers so that unchanged feeds return HTTP 304 Not Modified, minimising bandwidth on your end.
• Polite intervals. Feeds are polled no more frequently than once every 15 minutes. Sources that return errors are backed off exponentially — up to six hours between retries — so a struggling server never gets hammered.
• RSS/Atom only — no full-page scraping. The crawler reads syndication feeds only. It never fetches full article pages or follows links within articles. Full article text is never stored or reproduced.

What is stored

• Article headline (title)
• Short summary — truncated, sourced from the feed's own <description> field
• Article URL, linking back to your site
• Publication date and source attribution

Nothing beyond what appears in your public RSS feed is accessed or stored.

Requesting exclusion

If you publish an RSS feed and would prefer Smoko News not index it, email support@smokonews.com.au and we will remove your feed promptly — usually within one business day.