Larbin is a web crawler (also called (web) robot, spider, scooter, etc). It is intended to fetch a large number of web pages to fill the database of a search engine. With a network fast enough, Larbin should be able to fetch more than 100 millions pages on a standard PC. /
Similar scripts
webbase
(Popularity: ) : webbase is an internet web crawler written in C and later ported to C++. It uses a MySQL database to ... harvest
(Popularity: ) : Harvest is a system to collect information and make them searchable using a web interface. Harvest can collect information on ...
Script Library
) : webbase is an internet web crawler written in C and later ported to C++. It uses a MySQL database to ...