https://www.imintweb.com/2015/11/robotstxt-explanation.html