Our crawler (AdsupBot)

Who we are

Adsup.ai is an ad-intelligence service. Our crawler, AdsupBot, visits public websites and records the ads that those pages show to every visitor: the creative, its size and position, the ad network that served it, and where a click would lead. This page explains what it does and how to make it stop.

Site owner? Opt out of crawling here. It takes a few minutes and works immediately.

What it visits

• Public pages only. The home page and at most a handful of article or section pages that are linked from it on the same domain. We never visit pages that need a login, a payment or a form submission.
• Small files. robots.txt and ads.txt / app-ads.txt, read once per check.
• Nothing personal. We do not create accounts, submit forms, store cookies or collect visitors' personal data.

How often and how hard

• One session per site at a time, with at least 5 seconds between page loads.
• The same site, country and device combination is visited at most every 6 hours. Most sites are revisited every 1 to 180 days depending on how much advertising they carry. Sites without ads are rarely visited again.
• We never click ads, and we do not request pay-per-click redirect links. Our visits do not generate paid clicks for you or your advertisers.

How it browses

AdsupBot loads pages in a real Chrome browser so that it sees the same ads as a person. It sends a standard Chrome user-agent, not a bot name, because many ad networks serve nothing to self-declared bots. This has two consequences for site owners:

• A rule that targets a bot by name (for example User-agent: AdsupBot in robots.txt) does not reach it, and user-agent filters won't reliably identify it.
• The opt-out is the reliable way to stop us. Once your domain is verified, every subdomain is excluded and no further visits are scheduled.

robots.txt

When AdsupBot picks which inner pages to visit, it skips any URL that your robots.txt disallows for all crawlers (User-agent: *). The home page itself may still be loaded, because it is the page every visitor sees. To stop all visits, use the opt-out rather than robots.txt.

How the opt-out works

1. Enter your domain on the opt-out page. We work on the registrable domain, so blog.example.co.uk becomes example.co.uk and covers all its subdomains.
2. Prove that you control it, either with a DNS TXT record adsup-verify=<token> on the domain, or with a file at https://<domain>/.well-known/adsup-verify.txt that contains the token.
3. Click verify. On success, crawling stops at once: queued visits are canceled, a visit already in progress is discarded, and the domain gets no public profile pages.

You can remove the DNS record or the file afterwards; the opt-out stays in place.

Contact

For anything the form does not cover (removing ads already collected, a data-deletion request, or undoing an opt-out), sign in and open a support ticket from your account, or use the support button on any page. We answer takedown requests within 7 working days and personal-data deletion requests within 30 days.