An online store might be happy to let an AI shopping assistant read a product listing. It may feel differently when that assistant keeps refreshing inventory or tries to place an order without clear approval from the shopper. The server log could identify all three activities under the same bot name.
Calling a visitor a good bot or a bad bot doesn’t offer much help in that situation. AI agents can move from reading public information to taking actions for a person during a single visit. What they’re allowed to do matters more than the category they fall into.
A Familiar Bot Can Still Cause Problems
Search engines use crawlers to find pages, and monitoring services make automated requests to check that websites are working. Site owners generally have reasons to welcome both. Spam scripts and programs that try stolen passwords have a much less welcome purpose.
Scraping is trickier. An approved price-comparison service might collect a retailer’s public listings. Another scraper could pull the same catalog despite the retailer’s restrictions. The requests may look similar at first, even though the store has agreed to one use and objects to the other.
Even an ordinary search crawler can slow a small site down if it requests pages too aggressively. Whoever pays the hosting bill may have to restrict a visitor the site would otherwise welcome.
We’ve covered an obvious case of abuse in our reporting on fake followers and bot-driven manipulation. Automated accounts can make people or posts appear more popular than they are. With everyday crawlers and shopping tools, deciding what to permit often takes a closer look.
AI Agents Don’t Stick to One Job
Traditional crawlers tend to have a defined task. A search bot fetches pages for indexing; an uptime monitor checks whether a service responds. An AI assistant might start by reading a product description, then check shipping options and move toward checkout because that’s what its user asked it to do.
An assistant doesn’t need account permissions to read a public listing. It does need the customer’s approval to check an order. Changing a delivery address or placing an order should require permission for that specific action.
DataDome’s good bot vs bad bots guide is a handful one as it looks at bots’ purposes and whether they respect a website’s rules. An AI assistant’s assignment can change halfway through a visit, so its general category tells the owner little about its next request.
A shopping assistant may be welcome when it helps someone find a product. That doesn’t entitle it to keep requesting inventory updates at a rate that slows down the store. Nor should an agent that’s allowed to check an order automatically be allowed to cancel it. Permission has to match the action, even when the agent itself is legitimate.
A Recognizable Name Doesn’t Grant Access
Bots often announce their identities through an HTTP user-agent string. It’s a helpful starting point when reading logs, but the client making a request controls that field. An unwanted scraper can claim to be a well-known search crawler.
Identity can be checked more carefully for providers that publish verification methods. Google’s documentation on verifying crawler requests, for instance, explains how to use published IP ranges and DNS information to confirm certain Google crawlers and fetchers.
Verification establishes where a request came from. A genuine crawler can read public articles without getting access to customer accounts. A signed-in assistant still needs permission for the particular information or action it requests.
Say a customer asks an assistant to check delivery progress. The store can allow that through an authenticated session with limited permissions. If the assistant later requests a payment-method change, the application needs separate authorization for that action. Recognition of the agent alone wouldn’t be enough.
Read the Requests, Not Just the Label
A site’s logs can show which URLs an automated visitor requests, how quickly it moves between pages and what happens when it tries to sign in. Failed authentication attempts and repeated form submissions tell a more complete story than the name in the request header.
A crawler reading public help pages at a manageable pace may need no intervention. One repeatedly hitting a login endpoint deserves a closer look, even if the name in its header seems legitimate.
Website owners can publish crawling preferences in `robots.txt`. Many standard crawlers follow those instructions, although some legitimate user-triggered fetchers operate differently. In any case, the file can’t protect private pages or enforce permissions against a client that ignores it. Login checks and other access controls still have to do that work.
Sometimes the logs won’t settle why an agent is visiting. Rate limits can still protect expensive endpoints, and authentication rules can prevent it from acting on private accounts without approval.
Allow, Limit or Block According to the Action
A crawler reading an article and an assistant trying to update a payment method shouldn’t face the same access rules. Sites can decide what to allow based on the page or action being requested.
| What the visitor is doing | What the site should check | Possible response |
| Verified crawler reading public articles | Origin, crawling rules and request rate | Allow within reasonable limits |
| Shopping assistant checking stock | Public access and frequency of requests | Allow or rate-limit |
| Assistant viewing an order | Customer authentication and permission to view it | Require scoped account access |
| Script repeatedly attempting logins | Failed attempts and suspicious request patterns | Challenge, throttle or block |
| Scraper ignoring content-access restrictions | Site policy and the effect on resources | Restrict or block |
For a publisher, search indexing may be welcome while bulk copying isn’t. Retailers may want shopping assistants to read product listings but require authentication before those assistants do anything with an order. A small community site might need tighter request limits than a larger platform simply because it has fewer server resources.
Traffic patterns change, so restrictions need occasional review. An agent that previously caused no trouble might begin making requests the server can’t handle. Blocking everything associated with its provider could also affect customers who use that service. It’s usually worth looking at the affected pages and request rates before extending a block across the site.
Where Website Owners Should Draw the Line
A store can welcome an assistant searching for products and still require a fresh permission check before it changes an order. A publisher can permit search indexing and place limits on heavy data collection.
Neither decision requires the operator to guess what’s inside an agent’s instructions.
Bot names can help identify who is sending traffic, but access rules have to be enforced where the requests happen. Public content, customer accounts and checkout each need their own boundaries. An agent should be able to do the work it’s authorized to do without acquiring access to everything else along the way.
