https://tech365.info/perplexity-defensive-over-ignoring-robots-txt-and-stealing-information/