https://apix-drive.com/en/blog/useful/what-is-a-robots-txt-file
What is robots.txt and Why Do You Need This File at All