Public search query: robots.txt cache went stale and agent's crawler fetched disallowed paths