The SurveiLN crawler

What it is

SurveiLN is a content-change alerting service. People ask it to watch a specific page — a manga chapter list, a YouTube channel, a job board — and we tell them when that page changes in the way they described.

Every fetch we make is user-initiated: someone explicitly asked us to watch that exact URL. We don't crawl link-to-link, we don't index sites, and we never discover pages on our own.

How to identify us

Every request we make carries this User-Agent:

SurveiLN/1.0 (+https://surveiln.com/bot)

We don't disguise ourselves as a browser. If you see a fake browser User-Agent, that isn't us.

How to control or block us

We read and obey robots.txt before fetching any page. Our token is SurveiLN. To stop us entirely:

User-agent: SurveiLN
Disallow: /

Or to keep us out of one area only:

User-agent: SurveiLN
Disallow: /members/

Changes take effect within an hour (we cache robots.txt for that long). When a page is disallowed we stop fetching it and tell the user their tracker can't run — we don't retry around the block or route requests elsewhere to evade it. If someone tries to add a page your rules disallow, we refuse it outright rather than creating a tracker that quietly fails.

One exception, stated plainly:if you publish an RSS or Atom feed and a user subscribes to it, we fetch that feed directly without applying robots rules — the same behaviour as any feed reader, since a feed exists to be read by machines. Several large sites disallow their feed paths to keep them out of search results, which isn't aimed at subscribers. If you'd rather we didn't fetch yours, email us and we'll stop.

How often we visit

Rarely, by design. SurveiLN is built for content that changes a few times a day at most, so the fastest any page is checked is once every three hours, and most are checked every six or once a day.

• We send If-None-Match and If-Modified-Since, so an unchanged page costs you a 304 and no body.
• Pages that haven't changed in a while are checked progressively less often.
• If your site publishes a feed, we use it instead of fetching pages.
• A user can request one immediate check, at most once per 15 minutes per tracked page.
• After three consecutive failures we pause that tracker rather than retrying indefinitely.

What we don't do

• We don't log in, submit forms, or access anything a signed-out visitor couldn't.
• We don't bypass paywalls, CAPTCHAs, or bot challenges. If you challenge us, we stop.
• We don't execute JavaScript — we read the HTML you serve.
• We don't republish or redistribute your content.
• We don't store copies of your pages. We keep a cryptographic hash for change detection, plus a one-sentence summary of the latest state to show the person who asked.

Getting in touch

If we're causing you trouble, or you'd rather we fetched differently — a slower rate, a specific endpoint, an API you'd prefer we used — email hello@surveiln.com and a human will read it. It's a small operation and we'd rather fix it than be a nuisance.