A Novel Parallel Method for Denoising and Deduplicating Mass Web Documents

Abstract

With rapid growth of web applications, search engines accumulate lots of duplicate data in their cloud archiving system which affects its data processing efficiency. Noise reduction and de-duplication technology can optimize storage to meet the growing demand for the mass data, especially for web search engines. This paper proposes a novel parallel noise reduction and deduplication method for web documents (PDNR). This novel method employs Map-Reduce computing model, the frequency statistics and a method of deleting duplicate web pages with hash table and inverted index. PDNR uses four Map-Reduce tasks to conduct the jobs of distributing data, noise reduction and deduplication. An experimental system is developed on Hadoop platform, and real world web pages are fed to the system to conduct the experiments. Result shows that PDNR can remove duplicate web pages efficiently and accurately.