Performance analysis of support vector machines with polynomial kernel for sentiment polarity identification: A case study in lecturer’s performance questionnaire

The lecturers’ performance evaluation process can be carried out using an open questionnaire filled out by students at the end of the semester. In this questionnaire, students can provide an assessment in the form of comments, suggestions, and criticism of the ’lecturer’s performance, which can describe the level of student satisfaction with the lecture process. Conducting assessments or analyses on the open questionnaire entries manually will certainly impact the high costs time and energy. Sentiment polarity identification is a process in sentiment analysis that classifies text into a sentence or document and then determines whether the opinion expressed is positive, negative or neutral. In this research, a sentiment polarity detection system was developed in a lecturer evaluation questionnaire using the Support Vector Machine (SVM) method with a polynomial kernel. The test results showed that the SVM method’s performance with the Polynomial kernel was strongly influenced by the value of the learning rate parameter, the maximum iteration, and the degree, with the optimal parameter values, respectively, 0.001, 200, and 0.3. The use of optimal parameter values in the process of identifying sentiment polarity obtained an accuracy value of 84.88%.


Introduction
Lecturers are an important and essential element in an educational institution. The roles, duties, and responsibilities of lecturers are very important in realizing the goals of national education, namely to educate the 'nation's life, improve the quality of Indonesian people, which includes the quality of faith/piety, noble morals, and mastery of science, technology, and art, as well as realizing Indonesian society which is advanced, prosperous, and civilized. To carry out these very strategic functions, roles, and positions, professional lecturers whose performance can be measured are needed.
One of the lecturer assessment or evaluation processes can be done by using an assessment questionnaire for lecturers filled out by students. An assessment questionnaire is one of the commonly used evaluation tools, containing a list of a number of questions given to respondents to provide an assessment or feedback on an object or activity with a specific purpose [1]. Questionnaire is a data collection technique which is done by giving a set of questions or written questions to respondents to be answered [2]. A closed questionnaire is a questionnaire in the form of questions such as yes, no, multiple-choice, or a checklist, while open questionnaires are in the form of short answers or short descriptions in the form of fields.
One of the challenges in processing an honest assessment questionnaire is the analysis process. This is because, in the open questionnaire, the 'respondents' entries are in the form of suggestions, input, or in the form of opinions. It is not a choice of answers as in a closed questionnaire. Conducting an assessment or one by one analysis of the open questionnaire entries manually will certainly impact high costs, such as time and energy [1]. A large amount of data will also add to the challenge of assessing several lecturers, and the data is very diverse. Therefore, a tool is needed that can be used to assist the open questionnaire assessment process so that the analysis of the results can be carried out more quickly and efficiently.
Sentiment analysis is a technique for detecting opinions on a subject (such as individuals, organizations, or products) in a data set [3][4][5][6]. Sentiment analyst or Opinion mining is a system analysis process that converts a particular activity, topic, or product into a system whose process is carried out automatically, including affect analysis, emotion analysis, subjectivity analysis, and sentiment polarity or a positive and negative assessment from the comments on a document or review and the classification of a text [1,7]. Identification of sentiment polarity is one of the fundamental roles in sentiment analysis, namely classifying the text in a sentence or document and then determining the opinion expressed in the sentence or document whether it is positive, negative or neutral. There are two main approaches to extracting sentiment information on a text: the lexicon-based approach and the machine learning-based approach [8]. Several methods are often used in machine learning for sentiment analysis problems, including Naïve Bayes (NB), Support Vector Machine (SVM), Decision Tree, and Maximum Entropy Classification [8,9].
The SVM method is a classification method that uses a statistical approach to maximize the distance (margin) between the data and the hyper-plane separator. SVM has been used efficiently in many text classification cases because this method has many advantages, such as easily adjust functions and it works well on high-dimensional data [10]. SVM is a machine learning technique that solves over-fitting problems, local optimal solutions, low convergence ratios, and excellent generalizability in small sample scenarios. Research conducted by Chandani, et al, Arifin and Sasongko, and Supriyatna & Mustika also proved that SVM is a classification method that has better classification accuracy than other classification methods [11][12][13]. In this study, we developed and analysed the performance of the polarity identification system using the SVM method with the polynomial kernel in lecturer performance evaluation.

Method
As shown in Figure 1, to identify sentiment polarity in the lecturer evaluation questionnaire, several steps need to be done to ensure the best identification results are obtained.

Sentiment Data Collection and Labelling
The first stage is the data collection process. In this study, data were taken from the Academic Information System (SIAK) of the Ganesha University of Education. The data used was data from student suggestions and criticisms to lecturers recorded in 2017, 2018, and 2019. The specifications of the data to be used can be seen in the table. In the evaluation process or method performance measurement, 100 data samples from each year will be used, which are taken randomly. After the data have been successfully collected into a dataset, the next step is labelling. In the labelling process, the existing sample data were labelled with the appropriate polarity, namely positive comments or negative comments. In measuring method performance, the sample data were further divided into training data sets and testing data sets.

Text Pre-processing
Before the classification process with the SVM method was carried out, sentiment data should go through the text pre-processing stage. At this stage, the data was prepared to become data ready for [14][15][16][17]. The pre-processing stage of this text begins with tokenization. Tokenization (tokenization) is a process carried out to break or cut a sentence into a collection of parts or words. The result of this deduction is also often referred to as a token. In the next process, each token produced from the tokenization was carried out by a case-folding process. The case folding process madethe entire characters in the token became lower case. The next stage is the process of getting the root word from an affixed word by separating each word from the root word and its prefix and ending which is known as stemming. The stemming algorithm used was the Nazief & Adriani algorithm. The cleansing process will remove punctuation marks and unnecessary characters from a text. The next process was filtering, which eliminated words that appear in large numbers but were deemed to have no meaning (stop words). The final stage of text pre-processing is the process of calculating the word weight (Term Weighting). Each word was weighted using the Term Frenquency-Inverse Document Frequency (TF-IDF) method. TF-IDF method (w) is shown in equation 1.
Where, tf (t,d) is the occurrence of the word t in the document d, is N the number of documents in the document and df is the number of documents that contain the word t. In this study, a document was a comment written by students for a lecturer.

Polarity Identification Process using Support Vector Machine (SVM)
SVM is a method that can solve problems linearly or non-linearly. In solving non-linear problems in SVM, the kernel concept was used in high-dimensional workspaces by looking for hyperplanes to maximize margins between data classes. In classifying data using the SVM method, the K kernel function (xi, xd) was used. The kernels used in this study are as follows: Processing of training data used a sequential learning algorithm because it is a simple algorithm that 'doesn't take up much time, with the following calculation stages: 1. Initialize parameters required by SVM 2. Calculate the Hessian matrix obtained from the multiplication between the polynomial and y kernels, which are 1 and -1 vectors. The equation of the Hessian matrix is: 3. Perform the following calculations to iterate data i to j a. = ∑ b.
= + 4. Do the three steps above until you reach the maximum iteration limit. 5. The sequential learning process will get the value from the support vector (SV), where the SV = αi> SV threshold. After that, it is necessary to calculate the bias value b as in equation5. 6.
To find out the classification results of comments on a certain sentiment class, the f (x) function calculation process was carried out. If the result of this function was negative, then the sentiment polarity of the comment was identified as a negative sentiment. If the value of the function was positive, then the sentiment polarity of the comments was identified as a positive sentiment. The function f (x) is obtained from the following equation:

Performance Evaluation
System evaluation focuses on evaluating the performance of the SVM method in identifying polarity sentiment from the commentary data of the lecturer evaluation questionnaire. At this stage, the performance of the calculations that have been carried out canseen from the value of three parameters, namely accuracy, precision, and recall [14].

Polarity Sentiment Identification System Development Using SVM Method with Polynomial Kernels.
In this study, a web-based system was developed using the Laravel framework and the Python programming language. The Python programming language used when developing the SVM method utilized the Scikit-Learn library. Data communication between Laravel websites and Python was done by API using Flask. Flask is a micro web framework from Python. The results of implementing a webbased system for the identification process are shown in Figure 2. Identification of sentiment polarity using the SVM method with data sources in vectors containing weighted words using the TF-IDF method. These words have also gone through pre-processing stages such as stemming and stopword removal processes. The following is a snippet of source code in Python for the classification process using the SVM method.

Testing SVM Parameters with Polynomial Kernel.
In this study, testing was carried out to determine the effect of parameters on the SVM method's performance and determine the optimal parameters. The SVM parameters tested included the polynomial kernel degree value, the maximum iteration, and the learning rate constant. The observed values of this parameter testing were the value of F-Measure. The test results shown in Figure 3.a. shows the best learning rate constant is 0.001. This is indicated by an accuracy rate of 87.63%. The constant function of the learning rate is to control the speed of the training process. The Learning Rate constant depends on the number of iterations to achieve convergence. The test results showed that the optimal value of the learning rate constant was 0.01 and occurs when the maximum iteration reached 200 times. Changes in the value of the learning rate in the system affected the value of . This was because the learning rate was one of the candidate values that influenced the formation of alpha values and the formation of support vector sets. A decrease in the accuracy occurred when the learning rate was too large. Increasing the value of the learning rate had an impact towards the acceleration of the training process but reduced the level of accuracy.
The best accuracy value was obtained from the maximum iteration of 200 times from the maximum iteration test. This is shown in Figure 3.b., the best accuracy was obtained during the 200 iterations of 84.75%. However, increasing the number of iterations continuously did not mean an increase in the accuracy. A decrease in the level of accuracy occurred because the successive learning stages changed the value . The change in value was caused by the non-convergent value of , which can be proven by the change in the value of . High accuracy occurs because iterations give time to reach the convergent α value seen from the change in the α value. While the low accuracy occured because the α value obtained had not reached convergence, so the α value obtained as not optimal. The small iteration value caused less α value to be used as a support vector.
Testing the parameter value of the Degree Polynomial (lambda) was carried out to get the lambda value that produced the best accuracy. The value of λ (lambda) to be tested was 0.1; 0.3; 0.5; 0.7; 0.9; 1; 2; 3; and 4. Based on Figure 3.c.,it was found that the parameter λ (lambda) affected the accuracy results. This can be seen when λ (lambda) = 0.1 to 0.5 produced a high enough accuracy, then when the value of λ (lambda) = 0.7 began to decrease in accuracy until λ (lambda) = 1. Then accuracy has increased when λ (lambda) = 2. The value of λ (lambda) = 0.3 is the best result with an accuracy of 88.86%. A value of λ (lambda) that is too large will cause low accuracy. The value of λ (lambda) was too large and would affect when calculating the Hessian matrix. The Hessian matrix calculation would be slower because the large value of λ (lambda) will result in the speed of reaching convergence being slower and the instability of the learning process. Increasing the degree value in the polynomial kernel affected the results of the Hessian matrix calculation, which functions to find the optimum value in each data document. The hessian matrix results were used to calculate the error rate value for each document and affect the formation of support vector values.

Performance Test of Sentiment Polarity Identification Using the SVM Method.
In this test, testing the accuracy of the classification results and processing time by the system was carried out. This test used the 5-Fold cross-validation method with 1000 comments of data. By using the optimal parameter values, the measurement accuracy and processing time values can be seen in Table 1.

Conclusions
From the results of this study, it can be concluded that the SVM method with a polynomial kernel can be used to identify sentiment polarity in a lecturer performance evaluation questionnaire. The test results revealed the value of the learning rate parameter, maximum iteration, and degree in the polynomial kernel significantly affected the quality of the identification results of sentiment polarity. The optimal parameter values of the learning rate, maximum iteration, and degree found were 0.001, 200, and 0.3, respectively. Using these parameters produced an average accuracy of 84.88% with a processing time of 157.13 seconds.