基於網路拓撲的聚焦爬蟲研究

Abstract

聚焦爬蟲面向主題，過濾無關鏈結，只抓取相關的網頁資訊。通用的聚焦爬蟲，無法處理冗餘鏈結，因此本文提出了一種基於網路拓撲的聚焦爬蟲。從搜索引擎獲取初始網頁集，使用向量空間模型計算文本相似性。對抽取出的URL先進行鏈結分析，再根據無標度網路特徵，修正URL的權值。同時回饋不相關的主題區域，並通過URL與種子集合的距離設置不相關URL的緩衝區長度。仿真結果表明基於網路拓撲的爬蟲比通用爬蟲具有更高的查準率。Subject-oriented focued crawler, skips irrelevant links, and receives only relevant information. However, general fouced crawler couldn't deal with redundant links. This paper presents a kind of focused crawler based upon network topology. The crawler gets original URL sets from search engine, then calculates content similarity by the model of vector space. It analyzes link structure of websites, moreover modifies weight of URL according to the characteristic of scale-free network. Relevance feedback is used to disengage irrelevant regions, and the length of buffer is set for irrelevant URL by the distance between URL and seed pages. Experiments results prove that the precision of this focused crawler is higher than general crawler.