文章基本信息

标题：Efficient and Effective Filtering of Duplication Detection in Large Database Applications
本地全文：下载
作者：Zhang, Ji
期刊名称：Journal of Software
印刷版ISSN：1796-217X
出版年度：2012
卷号：7
期号：11
页码：2424-2436
DOI：10.4304/jsw.7.11.2424-2436
语种：English
出版社：Academy Publisher
摘要：In this paper, a robust filtering technique, calledPC-Filter (PC stands for partition comparison), is proposedfor effective and efficient duplicate record detection in largedatabases. PC-Filter distinguishes itself from all of existingmethods by using record partitions in duplicate detection.PC-Filter operates in three steps. It first sorts the wholedatabase and splits the sorted database into a number ofrecord partitions. The Partition Comparison Graph (PCG) isthen generated by performing fast partition pruning. Finally,duplicate records are effectively detected through internaland external partition comparison based on PCG. Fourclosure properties, used as heuristics, have been devisedto achieve a remarkable efficiency of the filter based ontriangle inequity of record similarity. The partition size iswell specified such that the time complexity of PC-Filter canbe optimized. By equipping existing detection methods withPC-Filter, we are able to well solve the major problems thatthe existing methods suffer.