Evaluation of E-Learning Web Sites Using Fuzzy Axiomatic Design Based Approach

High quality web site has been generally recognized as a critical enabler to conduct online business. Numerous studies exist in the literature to measure the business performance in relation to web site quality. In this paper, an axiomatic design based approach for fuzzy group decision making is adopted to evaluate the quality of e-learning web sites. Another multi-criteria decision making technique, namely fuzzy TOPSIS, is applied in order to validate the outcome. The methodology proposed in this paper has the advantage of incorporating requirements and enabling reductions in the problem size, as compared to fuzzy TOPSIS. A case study focusing on Turkish e-learning websites is presented, and based on the empirical findings, managerial implications and recommendations for future research are offered.


Introduction
E-Learning, one of the e-service applications, is a wide set of applications and processes that manage diverse types of electronic media to deliver vocational education and training (Aladwani and Palvia, 2002). For e-learning service providers, the Internet serves as the primary interface with the e-learners, since the e-learning web site has a much more extended function, compared to conventional web sites, which only disseminate information about services and products. Consequently, the web site quality should be considered as a critical success factor for e-learning service providers. Especially, in the case of vital education or training services, the web site quality and its evaluation should be studied in a more detailed manner from e-learners' perspective (Colette, 2001).
In a number of publications, quantitative methods are adopted for the evaluation of web site quality, with statistical methods ranking as the most widely used assessment tools In another study, the authors integrate fuzzy AD and fuzzy AHP into QFD (Quality Function Deployment) principles for routing of shipping investment decisions in crude oil tanker market (Celik et al., 2009c). Celik (2009)  The aim of this paper is to attain a group consensus on functional requirements of an ideal e-learning web site. A case study is then conducted in order to evaluate several elearning web sites according to these functional requirements with group fuzzy AD. Fuzzy AD methodology is based on the conventional AD; however, crisp ranges are replaced by fuzzy numbers that represent linguistic terms. For measuring intangible criteria such as reliability, responsiveness, etc., fuzzy AD is applied to translate linguistic terms into performance measures. Also, group consensus is sought throughout the study and therefore, fuzzy AD model is enhanced with a group decision making tool.
The paper is organized as follows. In next section, e-learning web site evaluation criteria are defined. Section 3 briefly describes the proposed fuzzy AD based evaluation methodology. A case study is conducted in e-learning web sites evaluation and the outcomes are explained in Section 4. The concluding remarks are given in the last section.

Evaluation criteria for e-learning web sites
Internet-oriented applications aim at satisfying current educational needs by closing the gap between traditional educational techniques and future trends in technology-blended education (Tzouveli et al., 2008), enabling a new type of education on online platforms. E-Learning refers to Internet technologies used to deliver a broad array of solutions that support the instructional process in a networked environment through the establishment of an interactive virtual classroom (Poon et al., 2004). The expected outcomes of online teaching and learning are largely dependent on the quality of the teaching processes and the effectiveness of online access. To this end, e-learning systems must be designed and constructed cautiously, especially while applying a scientific approach with well-designed procedures and techniques. The ultimate goal is to accomplish an effective and high quality learning system, comparable with the traditional educational systems (Colette, 2001). Web sites appear as the primary interface to the end user (e-learner) and user satisfaction vis-àvis human-computer interaction determines the quality of the e-learning provider. An organization with a poor web site or ineffective services may project weaken the organization's image and position. Hence, determining evaluation criteria for e-learning web sites is important in order to determine user needs (Ahn et al., 2007). In this context, an e-learning web site quality has to be analyzed in a more detailed manner.
Literature offers numerous studies investigating e-service and e-learning web site evaluation criteria. Webb and Webb (2004) states that a business to customer (e-learning provider to e-learner, in our context) web site quality is directly affected by service quality and information quality. According to Ahn et al. (2007), even though web site evaluation criteria may vary, the main categories include system, information, and service quality.
System quality (such as interface design and functionality), is an engineering oriented performance characteristic while information quality (such as completeness and timeliness) has both engineering and operational characteristicsService quality refers to availability of communication, mechanisms for accepting consumer complaints and their timely resolution with responsiveness, assurance, and follow-up services. According to the survey conducted by Poon et al. (2004), five main factors influence the effectiveness of e-learning process: students' behavior, characteristics of lecturers, interactive application, technology or system, and the institutions. On the other hand, Mahdavi et al. (2008) state that elearner satisfaction can be classified into four dimensions: content, personalization, learning community, and learner interface. Kim and Lee (2008) Ahn et al. (2007) state that technology-focused approach considers the web site as an information system, while service-focused approach sees a web site as a service provider.
Following criteria were determined with a point of view combining the two approaches:  Right and Understandable Content (C1): This criterion includes credibility, clearness and succinctness. While using educational web sites, authority is a particular concern, as high quality content must be assured. Instructional objectives should also be assured. In addition, the content should be easily understood, unambiguous and succinct.
 Complete Content (C2): This criterion includes accuracy and coverage. The purpose of this assessment is to guarantee that the content is actually correct: up to date, factual, detailed, exact and comprehensive. This criterion also assesses the existence of tests, quizzes and exams for adequate evaluation procedures.    User Interface (C7): This criterion includes the design appearance, consistency, the information structure and the organization of the web site. Applications of the right design principles are essential. A consistent interface allows the e-learners to follow the required tasks easily. The information structure and organization of the web site should also be easy to follow and to be understood by the e-learners.

Fuzzy Axiomatic Design based Group Decision-Making
In line with the multi-dimensional characteristics of web site quality, MCDM methodology is a powerful tool widely used for evaluating and ranking problems containing multiple, usually conflicting criteria. Over the years, several behavioral scientists, operational researchers and decision theorists have proposed a variety of methods describing how an evaluator might arrive at a preference judgment while choosing among the multiple alternatives. Hence, this work attempts to model the e-learning web site evaluation in an MCDM framework. In addition, the subjectivity and vagueness in the assessment process is dealt with fuzzy logic (Zadeh, 1975). Multiple decision makers (DMs) are often preferred rather than a single DM to avoid the bias and to minimize the partiality in the decision process (Herrera et al., 2001). Therefore, fuzzy MCDM with group decision is increasingly employed in literature, as evaluation criteria become more intangible and the decision making becomes more complex to make for single DM. For example, Chen and Cheng (2005) apply fuzzy MCDM with group decision to information systems personnel selection. Wang and Parkan (2008) consider fuzzy preference aggregation problem in group decision and they apply it to the broadband internet service selection. Recently, Yeh and Chang (2009) develop a hierarchical weighting method in order to assess the weights of a large number of evaluation criteria by pairwise comparisons. This paper proposes a set of evaluation criteria for e-learning web sites, as well as a methodology to evaluate these web sites. Main steps of the proposed methodology are recapitulated in Figure 1. The first step in the methodology is determining e-learning web site evaluation criteria. In this study, criteria described in Section 2 are employed. These Techniques employed in the study, namely Fuzzy AD, fuzzy AHP, Chen's aggregation methodology and fuzzy TOPSIS are now described.

Fuzzy Axiomatic Design
AD, a systematic method offering a scientific base for design, was introduced by Suh (1990) and its application areas include software design, quality system design, general system design, manufacturing system design, ergonomics, engineering systems, office cell design, and e-commerce strategies. AD is based on two axioms. The independence axiom states that the independence of functional requirements should be maintained and information axiom states that among the designs that satisfy the functional requirements, the design with the minimum information content is the best design. Information content, on which MCDM technique is based, represents a function of probability of satisfying a functional requirement . Therefore, the design with the highest probability to meet these requirements is the best design. Information content of a design with probability of success for a given is defined as follows: Fig. 1. This is the caption for the figure. If the caption is less than one line then it is centered. Long captions are justified manually.

Literature survey Expert experiences Web search
Step 1. Determining e-learning website evaluation criteria Step 3. Determining e-learning web site alternatives Step 4.a. Evaluating of web site alternatives Step 4.b. Evaluating of functional requirements Step 5. Fuzzification and aggregation of experts' opinions Step 2. Determining criteria weights with fuzzy AHP

Expert opinions Expert opinions
Step 6. Calculating information contents Step 7. Calculating weighted information contents Step 8. Ranking of the e-learning web site alternatives Step 9. Comparing the obtained results with fuzzy TOPSIS

Expert opinions FAD implementation
According to Suh (2001), logarithm is employed in calculating the information contents, so as to attain additivity.
On the other hand, the probability of success is given by the design range (the requirements for the design) and the system range (the system capacity). Figure 2 illustrates the design and system ranges as well as the common area. The intersection of the ranges offers the feasible solution. Therefore, the probability of success can be expressed as: where and represent the lower and upper limits of the design range and where represents the probability distribution function of the system for a given .
The probability of success is equal to the common area . Consequently, the information content can be expressed as follows: Also, if the probability distribution function is uniform, the probability of success becomes: Therefore, the information content can also be written as: Fuzzy AD methodology is based on the conventional AD. However, crisp ranges are replaced by fuzzy numbers that represent linguistic terms ( Figure 3). In this study, triangular fuzzy numbers (TFNs) are employed. Intersection of TFNs representing design and system ranges presents the common area (Kulak and Kahraman, 2005a). Firstly, the information content is calculated as in a non-fuzzy environment. Then information content in a fuzzy environment is calculated as follows: In this study, the calculation of the weighted information content is adapted from Kahraman and Cebi (2008). This model requires determination of weights of criteria and sub-criteria. Total weighted information content for first level criteria is calculated as follows: where n is the number of first level criteria and ∑ = 1 . Likewise, information content for second level criteria (sub-criteria for criterion ) is calculated as follows: where is the number of sub-criteria for criterion and ∑ = 1 for = 1, … .
Finally, according to information axiom, alternatives are ranked with increasing order of information content.

Fuzzy AHP
It is not possible to assume that e-learning website evaluation criteria are of equal importance. There are many methods that can be employed to determine weights, such as eigenvector, weighted least square, entropy methods and diverse MCDM methods. In this study, the fuzzy extension of the one of the most outstanding MCDM approaches, AHP (Saaty, 1980) is used to determine the decision criteria weights. Despite its wide range of applications, the conventional AHP approach may not fully reflect a style of human thinking. One reason is that decision makers usually feel more confident to give interval judgments rather than expressing single numeric values. Furthermore, knowledge concepts and models contain tacit values and uncertainty, causing assessment and evaluation to be more difficult. This difficulty is handled by applying AHP in fuzzy environment to solve prioritisation and evaluation problems.
Firstly advocated by Zadeh (1965), fuzzy set theory has become important to deal with the ambiguity in a system. In this study, firstly linguistic terms are used to represent the expert assessments, then triangular fuzzy numbers, 1 to 9 as given in Table 1, are used to represent subjective pair-wise comparisons of evaluation processes in order to capture the vagueness.
The four step computational procedure is given as follows: 1. Compare the performance score. Triangular fuzzy numbers (1 , 3 , 5 , 7 , 9 ) are used to indicate the relative strength of each pair of elements in the same hierarchy.
3. Solve the fuzzy eigenvalue. A fuzzy eigenvalue, λ, is a fuzzy number solution to: where is a × fuzzy matrix containing fuzzy numbers and is a non-zero × 1 fuzzy vector containing fuzzy number . To perform fuzzy multiplications and additions by using the interval arithmetic and α-cut, the equation = is equivalent to: where: for 0 < ≤ 1 and all , , with = 1, 2, … , , = 1, 2, … , .
The α-cut is known to incorporate the experts or decision-maker(s) confidence over his/her preference or the judgments. The degree of satisfaction for the judgment matrix is estimated by the index of optimism . A larger value of the index indicates a higher degree of optimism. The index of optimism is a linear convex combination (Lee, 1999) defined as: When is fixed, the following matrix can be obtained after setting the index of optimism, , in order to estimate the degree of satisfaction: The eigenvector is calculated by fixing the value and identifying the maximal If is less than 0.10, the comparisons are acceptable, otherwise not. is the average index for randomly generated weights (Saaty, 1980).

4.
The priority weight of each criterion can be obtained by multiplying the matrix of evaluation ratings by the vector of attribute weights and summing over all attributes.
Then the degree of similarity between the standardized triangular fuzzy numbers and can be measured by the similarity function: where ( , ) ∈ [0,1]. The larger the value of ( , ), the greater is similarity between the standardized triangular fuzzy numbers and . The equation ( , ) = ( , ) is valid for the degree of similarity.

The aggregation result of the fuzzy opinions is
where operators ⨂ and ⨁ are the fuzzy multiplication operator and the fuzzy addition operator, respectively. The method is independent of the type of membership functions being used (Chen, 1998;Celik et al., 2009).

Fuzzy TOPSIS
In our methodology, another MCDM method, namely TOPSIS is applied in order to compare with fuzzy AD outcome. TOPSIS is proposed by Chen and Hwang (1992)    added-value of online access.
However, as the number of available online programs increases, the decision process becomes complex. Therefore, the need arises to evaluate the quality of e-learning providers as a pre-requisite of achieving high quality of e-learning in Turkey. In this study web sites of e-learning providers are considered given that the interface greatly influences the e-learner satisfaction with the e-learning system.
The methodology described in Figure 1 is applied to a case study. The goal of this case study is to evaluate and rank the quality performances of e-learning web sites, with the The e-learning web site evaluation process is performed by applying the following steps: Step 1.

Step 2. Determination of criteria weights for e-learning web site evaluation
DMs apply pairwise comparison to evaluate criteria as given in Table 2 and fuzzy AHP given in Section 3.2 is employed with the index of optimism value = 0.5 in order to determine e-learning web site criteria weights. The obtained results are shown in Table 3.
Step 3. Determination of alternatives  To assess the quality of e-learning web sites, 12 web sites given in Table 4 were designated, taking into account experts' opinions in the sector and research conducted through the search engines in January 2008. The first six of these web sites operate worldwide and the remaining six are active only in Turkey. Since the sites all offer mostly common services, the comparison is coherent.

Step 4. DMs' evaluation of web site alternatives and functional requirements
Linguistic terms employed in evaluating e-learning web sites needs to be translated into fuzzy numbers in order to operate on the judgments. In this study, 5-level fuzzy scale is used to assess the alternatives and another 5-level fuzzy scale is used to assess the   Table 7. Evaluation of alternatives by DM2. Table 8. Evaluation of alternatives by DM3.
functional requirements, as a bare minimum for functional requirements. Table 5, Figure 4 and Figure 5 describe the linguistic terms, their abbreviations and fuzzy membership functions.
As given in Table 5, in the evaluation process, 5-level scale was employed to translate linguistic terms into fuzzy numbers. The judgments of the experts on alternatives are illustrated in Tables 6,7 and 8.
The functional requirements of e-learning web sites are defined by three experts and illustrated in Table 9. Step

Fuzzification and aggregation of DMs' opinions
DMs' judgments on functional requirements and alternatives given in Step 4 are first translated into fuzzy numbers and then aggregated using the methodology described in Section 3.3. Table 10 displays aggregated evaluations on functional requirements and alternatives.

Computation of common area and information contents
Once fuzzified evaluations of DMs' judgments are aggregated, fuzzy AD methodology described in section 3.1 is applied to compute the common areas, which are shown in Table   11. The greater the common area, better is the response of the alternative to the functional requirements.    Table 11. Common areas (Ac)

Step 7. Calculation of the weighted information contents
Weighted information contents, given in Table 13, are calculated as described in section 3.1.
Step 8. Ranking the e-learning website alternatives The final ranking is also given in Table 13. Final results demonstrate that two web sites (W6 and W12) are eliminated, meaning that they do not meet the necessary functional requirements. The evaluation results point out that web site W11 web site has the best performance overall, followed by web site W2.
Step 9. Comparison with Fuzzy TOPSIS As described in Section 3.4, fuzzy TOPSIS is applied to the aggregated decision matrix in order to compare and justify the outcome of fuzzy AD. Tables 14 and 15 demonstrate distances from FPIRP and FNIRP and total distances. E-Learning web sites were ranked in increasing order of performance index. As seen in Table 16, the outcome of the fuzzy AD methodology is justified with fuzzy TOPSIS. W11 is ranked as the best alternative with both methodologies. Alternatives W6 and W12 are     and W12 are considered as well. This comparison of the outcome proves fuzzy AD to be a more suitable methodology to evaluate a large number of alternatives, since a bare minimum for the alternatives can be defined and unsuitable alternatives can easily be eliminated.

Concluding remarks
Current developments of information systems facilitate greatly the diffusion of knowledge. Knowledge and education form the source of more than 50% of the personal national incomes of especially developed western countries and a well educated manpower working at the jobs related with information (Kızılsu, 2006). On the other hand, the advances in information systems and internet change the nature of education (Poon et al., 2004). Therefore, education activities transfer to electronic platforms for higher speed and less effort. Consequently, the web sites hosting the e-learning system become an important interface for the end-user and web site quality directly affects e-learning provider's performance. Therefore, a performance analysis based on MDCM techniques is applied to measure the quality of e-learning web sites. More precisely, in this paper, a group decision based fuzzy AD methodology was applied to the problem of ranking e-learning web sites.
The proposed methodology is expected to provide additional contribution and decision support to the managers working in the learning and e-business industries, due to its advantages over already established techniques, such as fuzzy TOPSIS. Specifically, the proposed methodology incorporates functional requirements into the ranking and selection process, and can identify the alternatives that do not comply with the requirements.
For future research, the set of alternatives can be further extended and a two-stage MCDM analysis consisting of pre-assessment and detailed evaluation can be applied in order to thoroughly review e-learning web site alternatives. Given that service web sites are increasing in number, pre-assessment stage will eliminate rapidly the unsuitable candidates with general criteria and minimum assessment of alternatives, preferably with a single expert. A more meticulous evaluation with fewer alternatives will be realized with a hierarchic structure of criteria and a more detailed assessment of a group of expert. Also, criteria set may be altered to evaluate web sites other than e-learning, such as e-commerce, hospital, and bank web sites, based on the proposed methodology, since the general scheme of criteria can be applied to all web sites with only a few changes.