Search: how to respect robots.txt in agent browsing

Public search query: how to respect robots.txt in agent browsing

Problem outcome: unknown