Archive for August, 2009
Open-Search-Server, experiences? (open source web crawler) — Question
Was wondering what other’s experiences have been using the open source web crawler, Open Search Server.
http://www.open-search-server.com/
Does anyone know of any documentation that exists that details the GUI and what all the configurations, variables, and query parameters mean?
Also, has anyone used the xml api and had success getting data out of the index?
Web Craweler Design — Article
Good article on web crawler design.
Writing a Web Crawler in Java — Article
Here’s a great article on how web crawlers work, their benefits, and the different reasons for writing one.
http://java.sun.com/developer/technicalArticles/ThirdParty/WebCrawler/