Construction of Community Web Directories based on Web usage Data

This paper support the concept of a community Web directory, as a Web directory that is constructed according to the needs and interests of particular user communities. Furthermore, it presents the complete method for the construction of such directories by using web usage data. User community models take the form of thematic hierarchies and are constructed by employing clustering approach. We applied our methodology to the ODP directory and also to an artificial Web directory, which was generated by clustering Web pages that appear in the access log of an Internet Service Provider. For the discovery of the community models, we introduced a new criterion that combines a priori thematic informativeness of the Web directory categories with the level of interest observed in the usage data. In this context, we introduced and evaluated new clustering method. We have tested the methodology using access log files which are collected from the proxy servers of an Internet Service Provider and provided results that indicates the usability of the community Web directories. The proposed clustering methodology is evaluated both on a specialized artificial and a community Web directory, indicating its value to the user of the web.


INTRODUCTION
The objective of the paper is building the Web directories according to the preferences of user communities. A Web directory, such as Microsoft (www.microsoft.com) and the Open Directory Project (ODP) (dmoz.org), users can find any topic that they are interested by searching the websites, starting with large categories and gradually moving down and choosing the category related to their interests. Searching for a particular topic ,user has to check deep inside the directory. Hence, the size and the complexity of the Web directory has increased and information overload problem will occur. To overcome the difficulties of web directories, we focus on the construction of community web directories. The users of a community can use the community directory as a starting point for navigating the Web, based on the topics that they are interested in, instead of accessing vast Web directories. The construction of community web directories with usage data collected from the Internet Service Provider raises a number of research issues, which are addressed and solved in this paper. User communities are formed using data collected from the log files of Internet Service Provider. The goal is constructing community web directories based on web usage data. The usage data that form the basis for the construction of community Web directories are collected by the proxy servers of Internet service provider.

Previous work
A Web directory, such as Microsoft (www.microsoft.com) and the Open Directory Project (ODP) (dmoz.org), users can find any topic that they are interested ,by searching the websites starting with large categories and gradually moving down and choosing the category related to their interests. However, the information for the particular topic ,user has to check deep inside the directory. Hence, the size and the complexity of the Web directory has increased and information overload problem will occur, i.e., it is often difficult to navigate to the information of interest to a particular user. On the other hand, Web Personalization, i.e., the task of making Web-based information systems changed to the needs and interests of individual users, or groups of users, to tackle information overload. However, in achieving personalization, we need to create the accurate and operational user models, but creating and maintaining these models are difficult for various reasons.

Present Work
In the concept of a community Web directory, a Web directory is created according to the needs and interests of particular user communities. Furthermore, it presents the complete method for the creation of such directories by using web usage data. User community models take the form of thematic hierarchies and are constructed by using clustering approach. We applied our methodology to the ODP directory, and also to an artificial Web directory, which was generated by clustering Web pages that appear in the log file of a proxy server. For the discovery of the community models, we introduced a new criterion that combines a priori thematic informativeness of the Web directory categories with the level of interest observed in the usage data. We have tested the methodology using access logs from the proxy servers of an Internet Service Provider(ISP) and provided results that indicates the usability of the community Web directories. The results presented here provide an initial measure of the benefits that we can obtain by constructing Web directories to the needs and interests of user communities. However, we have only approximated the gain of the end user and have not bothered about the cost of "losses" that could be encountered in the case of personalized web directory. This issue requires the evaluation of community Web directories according to the user preferences.

ISP Log Format
The Common Log Format is a standardized text file format used by web servers when generating access log files. Because the format is standardized, the files may be analyzed by a number of analysis programs. Each line in a file stored in the Common Log Format has the following syntax: host ident authuser date request status bytes  "GET /apache_pb.gif HTTP/1.0" is the request line from the client. The method GET, /apache_pb.gif the resource requested, and HTTP/1.0 the HTTP protocol.
200 is the HTTP status code returned to the client. 2xx indicates successful response, 3xx indicates redirection, 4xx indicates client error, and 5xx indicates server error.
2326 is the size of the object returned to the client, which is measured in bytes.

Community Web Directory Discovery Algorithm
In this section, we describe the new clustering methodology that we propose for the construction of the community Web directories. This method is used for the selection of categories of the Web directory .
Step2: Extract information from the log file.
Step3: Extract websites from log file.
Step4: Extract directories of the websites.
Step5: Cluster directories based on training set into various clusters. If cluster is not found then cluster is "unspecified"

Evaluation of Community Web Directories
The methodology introduced in this paper for the construction of community Web directories has been tested for analyzing the data collected from the proxy servers. Evaluation of community web directories involves the following steps.

Experimental Setup
Step1: Click on load file.
Step3: Click Get Site.

Conclusions and Future Work
We conclude that the concept of community Web directory specializes to the needs and interests of particular user communities. Furthermore, it presents the complete method for the creation of such directories by using web usage data. User community models are constructed by employing clustering approach. We applied our methodology to the ODP directory, as well as to an artificial Web directory, which was generated by clustering Web pages that appear in the access log files of a Web proxy. For the discovery of the community models, we introduced a new criterion that combines a priori thematic informativeness of the Web directory categories with the level of interest observed in the usage data. We have tested the methodology using access logs from the proxy servers of an Internet Service Provider(ISP) and provided results that indicates the usability of the community Web directories. The proposed methodology addresses the issues of existing method by reducing the dimensionality of the problem, through the classification of individual Web pages into the different categories of the web directory. The results presented here provide an initial measure of the benefits that we can obtain by creating Web directories to the needs and interests of user communities. However, we have only approximated the gain of the end user and have not bothered about the cost of "losses" that could be encountered in the case of personalized web directory. This issue requires the evaluation of community Web directories according to user preferences. Various components of the methodology could be replaced by a number of alternatives. Most importantly, more sophisticated methods for extracting the categories from usage data, in addition to the use of an existing Web directory, would make the mapping of pages to domains and then to categories more accurate and complete.