https://analyticsindiamag.com/google-open-sources-robot-txt-to-help-standardise-robots-exclusion-protocol/