People use DA to find out which websites are trusted inside their area and to determine where by to get fantastic back links. txt file is then parsed and will instruct the robot as to which internet pages aren't to generally be crawled. Like a internet search engine crawler may https://www.youtube.com/watch?v=1amb7r2WR1g