Commit Graph
1 Commits
Author SHA1 Message Date
Christopher Giffard 238313fbb1 Add simplecrawler (#489)
Simplecrawler is a simple but powerful event driven web crawler.

Rationale:

* Simplecrawler is well documented and tested.
* Simplecrawler has more than thirty individual contributors and a core team of three.
* Simplecrawler has some notoriety: with nearly 1000 stars on Github it is the most popular still-maintained web crawler library for node.
* Articles and blog entries have been written about simplecrawler, (most notably [this one from WIRED](http://www.wired.com/2015/10/cyphon-wired-archive-migration/)) and it has seen continued use in many large organisations.

Closes #489
2016-04-15 15:33:49 +07:00