Crawler policy

What we collect

For each casino we cover, once per week, we load the publicly accessible lobby pages from an IP address in a country that casino serves. From those pages we record only the ordering: which games are shown, in which section, at which position, viewed from which country, in which week.

We do not collect player data, account data, odds, RNG behaviour, or anything behind a login — because we never log in.

What we never do

• Create accounts, log in, or authenticate in any form.
• Launch games, place bets, or interact with wallets, cashiers or bonuses.
• Submit forms or send anything other than ordinary page requests.
• Attempt to reach non-public endpoints, or anything requiring credentials.
• Probe for vulnerabilities, or test rate limits to find where they break.
• Resell the raw collected data as a feed.

How we behave

• Frequency: one lobby visit per casino per region per week. Not hourly, not daily.
• Rate: no more than one request per second within a visit, and we cache aggressively so we do not re-request what we already have.
• robots.txt: we read it before we read anything else on a host, and we do not request a path it disallows. We follow RFC 9309 for what an unreadable one means, which is the same reading Google and the other major crawlers use:
  • A 4xx response other than 429 — including 403 — means no robots.txt exists, so there are no rules to honour and we may crawl (RFC 9309 §2.3.1.3). We previously treated a 403 as a refusal. That was stricter than the standard and, it turned out, usually wrong about intent: the hosts it stopped us reading were image CDNs whose 403 is a firewall default, serving sites whose own robots.txt says yes.
  • A 429, a 5xx, or no response at all means the file is undefined, and we crawl nothing on that host (RFC 9309 §2.3.1.4). A host under load telling us to slow down is not a host with no rules.
  Where robots.txt sets a Crawl-delay longer than our own one-second floor, we use yours.
• 403 or 429 on a page or a file is always a refusal. None of the above changes that. What an unreadable robots.txt means is a question about whether a policy exists; being refused a page is the policy, and we treat it as an answer. We do not retry around it, do not change our User-Agent, and do not work around bot protection.
• Backing off: repeated 403 or 429 responses abort the run. We treat those as an answer, not an obstacle, and we do not retry harder or rotate around them.
• Rendering: we load pages as a normal browser does, which is how the lobby ordering becomes visible at all. We are not attempting to disguise ourselves as a specific person, and we do not defeat access controls.

Asking us to stop

If you operate a casino and would rather we did not include your lobby, email hello@lobby-rank.com from a domain or role address associated with the operator. We will disable collection for your properties within 48 hours of confirming the request, and confirm in writing when it is done.

You do not need to give a reason, and we will not argue with the request. We may still cover the casino editorially using publicly available facts and manually maintained game lists — but we will stop crawling.

Corrections

If we have your lobby data wrong — a section mislabelled, a game misattributed, a region mapped incorrectly — tell us at hello@lobby-rank.com and we will correct it and note the correction.

Standing

This policy is a commitment we intend to be held to. If our behaviour ever fails to match what is written here, that is a bug and we want to hear about it.