SpiderServer - Design and Implementation of a MetaSearch Engine

Abstract

Search engines on the Web are valuable tools for searching information according to a user’s interests whether an individual or a software agent. In the present article we describe the design and the operation mode of SpiderServer, a metasearch engine used for the submission of a query followed by the retrieving of results from few popular search engines. We have divided the entire operation of this metaserch engine in two phases: the formation of the query and the parsing of the results.
There are two files for every search engine describing the two phases respectively, that is, the way in which the user’s query will have to be modified in order to be accepted by the specific search engine and then, the methodology that SpiderServer must follow in order to parse the results from the specific search engine. From the moment these two files exist, it is relatively easy to add a new search engine to the existing system. The ultimate goal is to show that we are able to construct systems that can retrieve information from sources for which they have not been programmed or from sources whose query interface and presentation of results changes quite often.