You’ll then must configure your crawler options. You are able to set crawl limitations, opt for unique user agents, and timetable automated web site audits. Be certain You will find there's reasonable division of content material and that diverse content material varieties will not be combined alongside one another https://www.youtube.com/watch?v=EVeTP0aV8gY