The Robots.txt is a file which you will be adding in the root directory of the Web Project. The search engines will take a look at the robot.txt file before it crawls into your website.

    Example :You may have frontend and backend in your project. In that case You need not want the crawlers(Search … Read more