User-agent: * Disallow: All robots will spider the domain