Web数据挖掘系统的设计及关键技术研究

Web数据挖掘是一种新兴的边缘科学技术,它涉及到机器学习、数据挖掘、信息检索、自然语言处理、数据库以及人工智能等技术,可用于网络检索、网站建设以及电子商务等方面。本文在对Web数据挖掘技术详细研究的基础上,提出了一个Web数据挖掘的通用系统框架,并对信息收集、信息选择和预处理、模式的提取和用户接口等各个组成部分所使用的技术和存在的问题及解决的方法进行了讨论。本文结合Web自身的特点,提出了一个智能网页收集器WebCrawler,它除具有一般WebRobot的基本功能外,还采用了一种既考虑文本重要性又考虑链接结构的URL排序方法,从而确保收集的Web页面是Web比较优秀的部分。

Research on the Design and Key Techniques of Web Mining System

Web mining is a new research area and involves several research communities, such as ML, DM, IR, NLP, DB and AI. It can be used in Web retrieval, web sites construction and e-commerce etc. Based on the whole study of Web mining techniques, a general architecture of Web mining system has been presented. The techniques, research issues and resolution of each component, resource finding, information selection and preprocessing, pattern discovery and pattern analysis, are also discussed in details. This paper introduced a smart web page gather-WebCrawler which takes both the text and structure information into account and gets the excellent pages of the web.