AI Crawler Access in Singapore: 22 Sites Tested
Measured 26 September 2026; every fetch completed by 08:17 SGT. Every status code below was fetched in this test.
The answer: On 26 September 2026 we fetched 22 well-known Singapore websites twice in the same minute: once as a browser, once as GPTBot. Eighteen answered a browser with HTTP 200. Three of those 18 returned something different to an AI crawler: sgCarMart answered 403 to GPTBot, Shopee Singapore 403 to both GPTBot and PerplexityBot, and Singapore Airlines 404 to both.
This test matters because a page can rank perfectly in Google and still be invisible to AI answers. If your server answers 403 to GPTBot, ChatGPT cannot read the page it is being asked about — it reads a competitor's instead. The pages that get cited are the ones an AI crawler was allowed to fetch and parse.
How we tested
Each of the 22 domains was fetched twice inside the same minute, from the same connection, with the same TLS stack, changing only the user-agent string:
- Browser user-agent — Chrome 128 on Windows.
- GPTBot user-agent —
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot. - PerplexityBot user-agent —
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot.
Because the browser fetch is the control, a difference between the two responses is caused by the user-agent and not by the network, the time of day or the requesting IP. Any domain whose bot response differed from its browser response was fetched a second time, two seconds later; every difference below reproduced.
What we found
- 22 domains tested; 18 answered a browser with HTTP 200.
- 15 of 18 returned the same status to GPTBot and PerplexityBot as to a browser.
- 3 of 18 served an AI crawler something different — two answered 403 and one answered 404, while all three served the page to a browser in the same minute.
- 4 domains answered 403 to every user-agent, including a browser; those are a general non-browser block, not an AI-specific one, and we cannot place them from this test. They are listed separately below.
- Median homepage HTML weight across the 18 sites that answered: 306.5 KB.
- 1 of 22 homepages carried FAQPage markup in the served HTML.
- 2 of 22 said Disallow to GPTBot in robots.txt (www.todayonline.com, www.foodpanda.sg). One of them served the page to GPTBot anyway, so a robots rule and the server response are separate controls — check both.
| Site | Browser | GPTBot | PerplexityBot | AI crawler verdict |
|---|---|---|---|---|
| The Straits Times (www.straitstimes.com) | 200 | 200 | 200 | same response to both |
| CNA (www.channelnewsasia.com) | 200 | 200 | 200 | same response to both |
| TODAY (www.todayonline.com) | 200 | 200 | 200 | same response to both |
| The Business Times (www.businesstimes.com.sg) | 200 | 200 | 200 | same response to both |
| Lianhe Zaobao (www.zaobao.com.sg) | 200 | 200 | 200 | same response to both |
| sgCarMart (www.sgcarmart.com) | 200 | 403 | 200 | differs — reproduced on a second pass |
| PropertyGuru | 403 | 403 | 200 | blocked every user-agent, including a browser — unresolved |
| 99.co | 403 | 403 | 403 | blocked every user-agent, including a browser — unresolved |
| Carousell | 403 | 403 | 403 | blocked every user-agent, including a browser — unresolved |
| Lazada Singapore (www.lazada.sg) | 200 | 200 | 200 | same response to both |
| Shopee Singapore (shopee.sg) | 200 | 403 | 403 | differs — reproduced on a second pass |
| Grab Singapore (www.grab.com) | 200 | 200 | 200 | same response to both |
| foodpanda Singapore | 403 | 403 | 403 | blocked every user-agent, including a browser — unresolved |
| Singapore Airlines (www.singaporeair.com) | 200 | 404 | 404 | differs — reproduced on a second pass |
| DBS (www.dbs.com.sg) | 200 | 200 | 200 | same response to both |
| OCBC (www.ocbc.com) | 200 | 200 | 200 | same response to both |
| UOB (www.uob.com.sg) | 200 | 200 | 200 | same response to both |
| SingSaver (www.singsaver.com.sg) | 200 | 200 | 200 | same response to both |
| Ministry of Manpower (www.mom.gov.sg) | 200 | 200 | 200 | same response to both |
| CPF Board (www.cpf.gov.sg) | 200 | 200 | 200 | same response to both |
| IRAS (www.iras.gov.sg) | 200 | 200 | 200 | same response to both |
| gov.sg (www.gov.sg) | 200 | 200 | 200 | same response to both |
Status codes only. A 403 to a crawler can also come from CDN bot rules, rate limiting or a WAF challenge; this test shows the outcome a crawler sees, not the cause. No per-site cause is claimed.
What to do if your site blocks a crawler
- Test your own homepage the same way. Ten minutes, no account, no tool install: two fetches, one user-agent changed.
- Check the layer that answered. A 403 usually comes from the CDN or WAF (Cloudflare, Vercel, AWS WAF), not from your page. Look for a bot rule, a managed challenge, or a rate limit applied to unknown user-agents.
- Allow the search crawlers by name — GPTBot and OAI-SearchBot (OpenAI), PerplexityBot, ClaudeBot (Anthropic), Google-Extended — before switching on any blanket "block bots" setting.
- Re-test after the change. A rule that is saved is not a rule that is live; re-fetch with the bot user-agent and compare the status code and the byte count.
Your free AISight audit fetches your pages the way an AI crawler does, and scores what came back.
Run Your Free AI Visibility Audit →Questions
How do I know if an AI crawler is blocked on my site?
Fetch your own homepage twice with the same tool, changing only the user-agent string: once as a normal browser, once as GPTBot (Mozilla/5.0 AppleWebKit/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot). If the browser fetch returns 200 and the GPTBot fetch returns 403, 404 or a challenge page, an AI crawler cannot read your page. Our 22-site test on 26 September 2026 found this pattern on 3 sites.
Does blocking GPTBot affect Google rankings?
No. Googlebot and GPTBot are separate crawlers, and blocking GPTBot does not change how Googlebot indexes your site. The cost is different: a page that answers 403 to GPTBot cannot be quoted in ChatGPT's answers, even while it ranks normally in Google. Of the 18 Singapore sites that answered a browser in our test, 15 returned the same status to GPTBot and PerplexityBot and 3 did not.
Which AI crawlers should a Singapore business allow?
At minimum GPTBot and OAI-SearchBot (OpenAI), PerplexityBot (Perplexity), ClaudeBot (Anthropic) and Google-Extended (Google AI). Two of the 22 sites we tested disallowed GPTBot in robots.txt on 26 September 2026, and one of those served the page to GPTBot anyway, so a robots rule and the server response do not always agree.
Next in this series: what the 10 top-ranking AI visibility tool pages actually serve, which sources AI models actually cite, how ChatGPT search picks pages, and how to read your AISight score. For the platform layer, see AEO ranking and GEO ranking, or see the plans.