matbot.co.uk

Hi. I'm Mat. I have a few projects that use functions that programatically request content from websites. These requests should identify themselves as 'MatBot (https://matbot.co.uk)' or similar.

robots.txt - exclusion

I normally only use MatBot to fetch content from online services that I know won't have any objections., so do not check for rules in robots.txt files. Please email mat at matbot.co.uk with any comments, questions or block requests.

Searchable index

Most of the results of MatBot's activities are used for specific purposes. There is a half-baked-idea to partially integrate with the XSolver words database to create an index based on dictionary words that are also active websites using .co.uk domains. This is partly driven by the idea of creating a small search engine that only lists websites that are not plugged into distributed ad networks and don't block content with repetitive boring crap about cookies. Cookies aren't the issue - 'trackers and privacy' are the issue.