HTML Posts

Recursive – Explore the endless web

Recursive – Explore the endless web

Wow, well that took longer than expected! 44 days ago I blogged that I had started work on a second version of my Chrome Crawler extension and have only just managed to get it to a state I was happy with enough to release it. To be fair I had been on a trip to New York during that period so perhaps I can be excused. Having said that however I think the time has been well spent and am fairly proud of the result.


Recursive is an experimental tool for visualising the world wide web. Given a URL it downloads the page search for links and then recursively downloads those. The information is then displayed in a node-based graph.

Printomi Maps

Printomi Maps

Well since we have made the decision to discontinue Printomi I have been backing up the databases and downloading the 90GB+ of images that users have uploaded.

Well it wouldn’t be like me if I didn’t start thinking about what cool things I do with all those pixels. I remember seeing those cool Mincraft maps that use the Google Maps API to explore the Minecraft servers and it got me thinking if it could be possible to do something like that but for the Printomi images.

Chrome Crawler – A web-crawler written in Javascript

EDIT: I now have a newer, better version of this called “Recursive

Depending on your level of geekness you may or may not enjoy this one.

I proudly present Chrome Crawler, my latest Google Chrome extension:

The idea is simple really. You just give it a URL, it then goes off and finds all the links on that page then follows them to more pages then gets all the links and follows them and so on and so on.

Along the way it checks each page to see if there are any ‘interesting’ files linked there, if it finds an interesting link it will flag it for you so you can check it out.

Theres an options page that lets you customise the way it all works:

If you are still confused check out the video below:

So why did I make this? Well to be frank, I made it mostly “just ’cause I can”!

Also having learned from my last Chrome Extension project PostToTumblr I realised the Chrome API allowed you to do some things that you wouldn’t normally be allowed to do on a website (nameley the Cross-Origin XHR) and I wanted to do something to take advantage of it.

It didnt take me long to knock out this project, one lazy Saturday for the majority of the code and today for a quick fix or two and to write this post and make the video. As such I expect there to be many bugs and problems so if you encounter one drop me an email (my address is found in the options page).

Oh finally, I wouldnt try using this on a google page as you will likely end up seeing this quite often:

Anyways you can grab it over on the Chrome extensions gallery here. If you enjoy it please leave me a review / comment, much love!