Mining Association Rules from Clinical Narratives

Shallow text analysis (Text Mining) uses mainly Information Extraction techniques. The low resource languages do not allow application of such traditional techniques with sufficient accuracy and recall on big data. In contrast, Data Mining approaches provide an opportunity to make deep analysis and to discover new knowledge. Frequent pattern mining approaches are used mainly for structured information in databases and are a quite challenging task in text mining. Unfortunately, most frequent pattern mining approaches do not use contextual information for extracted patterns: general patterns are extracted regardless of the context. We propose a method that processes raw informal texts (from health discussion forums) and formal texts (outpatient records) in Bulgarian language. In addition we use some context information and small terminological lexicons to generalize extracted frequent patterns. This allows to map informal expression of medical terminology to the formal one and to generate automatically resources.

Introduction
Clinical narratives written by medical experts are a growing source of patient-related information that can be used in medical research and healthcare management. In addition, informal medical texts and conversations in social networks and popular TV broadcasts increase significantly. Public forums for medical consultations exist as well where doctors provide opinions and answer questions. In this case medical experts abandon the professional style of clinical writings (brief reports in cryptic form with medical terminology and specific words, abbreviations, acronyms, and phrases of their professional jargon). They switch to informal, conversational style (avoiding terminology, using casual words for popular explanations, with shorter and simpler sentences etc.). Differences between informal and formal communication are often addressed in translation where the objective is to reproduce language in all its variety. In (Lozano and Matamala, 2009) the authors analyse how the original English medical terminology, used to give a special atmosphere and to reproduce a real professional context in the television series E.R., was translated in colloquial language to reach a broad audience of Spanish native speakers in the Spanish dubbed version. The original English terms were manually categorized based on their degree of formality in order to differentiate between formal and informal medical language. Then rules for selecting translations in different settings were elaborated. This structuring enabled to approach systematically the evaluation of translation. Many mistakes have been found in the translation of medical terminology, hence lowering the realism of the dubbed version. The conclusion is that terminology misuse is rather often in informal communication.
Here we also consider formal and informal medical texts but as input data for mining Association Rules (AR) that explicate stable collocational patterns of terms and words. The overall objective is to find frequent patterns of lexical cooccurrences which, as we suppose, will help to structure patient descriptions. In general there are no standard templates how patients are to be examined, e.g. in diabetes there is no predefined structured questionnaire how to document the status of a diabetic patient in the form of indicators and their values. Earlier attempts to structure patient status descriptions took us months to study a large number of patient records and weeks of manual work to produce feature-value templates that were discussed with medical experts (Boytcheva et al., 2010). Now we believe that ARs will discover automatically attributes and their values as the latter are expressed by lexical units in the free text descriptions. Therefore it is important to consider patient-related texts in records produced by medical doctors as well as informal texts written by the patients themselves. In addition for low resource languages like Bulgarian, Data Mining provides better instruments to discover new knowledge than Text Mining via shallow analysis and information extraction techniques. We propose a method that processes raw formal clinical narratives (outpatient records, OR) and informal texts (postings in health forums) in Bulgarian language. Some context information is taken into consideration when mining the ARs, which is an original aspect of the proposal. Small terminological lexicons provide generalization of extracted frequent patterns. This approach allows to map informal expressions of medical terminology to the formal ones and to study in parallel both the professional and colloquial medical language.
The paper is structured as follows. Section 2 overviews related work. Section 3 presents the materials used and Section 4 -the methods. Section 5 details the experiments. Section 6 contains the conclusion and plans for future work.

Related Work
Almost no electronic resources with medical terminology exist for Bulgarian language (except for terms in standard medical nomenclatures) so we are interested in terminology extraction for low resource languages. For Polish, linguistic analysis and statistical methods identify automatically phrases that cover 84% of the occurring medical terms in over 1,200 discharge letters (Marciniak and Mykowiecka, 2014). At the top of the ranked list, only 4% out of 400 terms were incorrect. Another work deals with term extraction from sparse, ungrammatical and informallywritten texts with domain-specific contents (Ittoo and Bouma, 2013). This paper focuses on rare (low frequency) terms, detects multi-word terms of arbitrarily lengths which are often disregarded by existing term extraction systems, and involves external resources (Wikipedia) to support domainspecific term extraction and assessment of accuracy. A rather high F1-measure is achieved (88% against a baseline of 77%) and successful extraction of terms regardless of their length.
Discovering frequent word sequences also provides useful hints about analysis of units in for-mal and informal texts. A method for extraction of all maximal frequent word sequences, which allows for gaps, is presented in (Ahonen-Myka, 1999). The algorithms in (Ahonen-Myka, 2002) include pruning of all stop words, which might be problematic in case of terminological expressions. No stemming is applied since inflexion endings might be meaningful, moreover their removal will combine sequences and in this case some lowfrequency variations can exceed together the selected frequency threshold. In (Ahonen-Myka and Doucet, 2005) the discovery of maximal frequent word sequences is re-considered in the light of collocation discovery, where "collocation" is a recurrent, stable multiword expression without gaps. The authors assume that most existing methods for collocation discovery cannot be straightforwardly extended to find sequences with length more than 5 words, when applied to large corpora.
Text mining extracts essential information from texts while data mining (in particular ARs) discovers novel knowledge about the subject. ARs that are found in texts are used in various kind of document-processing applications, for instance: (i) Text classiffication based on ARs: sentences of documents are viewed as basic text units (Haralambous and Lenca, 2014). This approach is enriched by linguistic knowledge (delivered by the Stanford dependency parser). Words are replaced by their hyperonyms in WordNet, to optimize the itemsets. In the training phase ARs are mined from sentences. At the classification stage, for each sentence s in a document d the system finds the most confident AR that can be applied to it (i.e., such that the itemset of the rule is entirely contained in the itemset of the sentence). An aggregation procedure classifies the document by taking class by class the sum of rule confidence and selecting the class with the highest sum. The evaluation was done on 7,000 texts of Reuters corpus. The experiments show that using dependency property (nsubj) to select a word is a better choice than the one provided by the tf-idf -based method. However, no alternative classification techniques are applied in parallel to the same text collection so no real conclusion can be drawn about the effectiveness of the suggested approach.
(ii) Elucidating domain concepts based on ARs: the paper (Yolcular, 2011) presents 12 ARs that can point to significant medical concepts mined in 600 otorhinolaryngology discharge notes written in Turkish language. The n-gram method was used for discovering terms co-occurrences. A dataset of concept candidates has been generated for the validation step and then the Predictive Apriori algorithm for AR mining was applied to validate the candidate concepts.
(iii) Explication of relations among text units: (Sizov and Öztürk, 2012) present the SmoothApriori algorithm that finds association relations between sentences which may reveal a cause-effect type of relation or have a more implicit nature. SmoothApriori uses similarity between items; compared to a previously proposed SoftApriori algorithm, which also makes use of similarity, it is able to utilize similarity values directly rather than reducing them to binary values similar/notsimilar. The evaluation was done on "Findings as to Risk" section of 208 Air Investigation Reports, published by the Transportation Board of Canada. Many top confidence rules, automatically generated by SmoothApriori, are interesting and make sense. Some rules connect consecutive sentences in the same text but the implicit discourse relation of causality is explicated. This application illustrates the potential of using ARs in various areas.
The results presented here integrate text and data mining ideas, extending further our previous developments (Boytcheva et al., 2017).

Materials
We work with two data sources: ORs submitted to the Bulgarian National Health Insurance Fund (NHIF) and content from the online medical portal puls.bg 1 . The ORs concern 10,000 diabetic patients and were produced by Endocrinologists (ESs) (set S00) and General Practitioners (GPs) (set S05) in 2012-2013. In total these are 330,666 records, semi-structured files with predefined XML-format. We use only two free text fields of these ORs: "Anamnesis" and "Patient Status". The corpus of informal medical texts includes postigns at puls.bg forum (set SF). We process all questions and comments from three subforums 2 amounting to 1,425 records. Fig. 1 presents the distribution of words (items) in the sets after stemming and stop words removal.
We analyzed how the sources vocabularies are distributed in different categories using available lexicons. Table 1  The items of SF are mainly general lexica (36.6%) and names of diseases (12.2%). The most frequent units are stop words (43%) followed by numerical values (5%), some common words like have/has, year, value, month, examination, problem, result etc. signaling for the duration of the symptoms and medical terms signaling for the location of the finding -endocrine, hormon, thyroid gland.
The items of S00 are mainly common words and names of diseases. The most frequent units are stop words (19.84%), numerical values (7.54%), names of anatomic organs and systems as well as their conditions: breath, abdomen, liver, rhytmic, soft, RR (Riva Roci), vesicular, pulmo, limbs.
Similarly to S00, the items of S05 are mainly common words and names of diseases. Some of the most frequent units are stop words (17.12%), numerical values (12.60%), and indicators related to diabetes concepts like: sugar, treatment, activity, diabetes, rythmic, type, breath, BSP (blood sugar profile).
The common items for SF and S05 represent terminology related to diabetes. This is not surprising because the forum topic is about diabetes and S00 contains ORs written by Endocrinology specialists. Examples of such items are: enlarged, weight, hemoglobin, insulin, examination, pain, control, glycated, treatments, blood, sugar, consisting, profile, diabetes etc.
The common items for SF and S00 are mainly terms related to medical examinations. This is also easy to explain since SF consists of informal texts and S00 contains clinical notes of General Practitioners. Examples of items common for SF and S00 are: treatment, procedure, drugs, complains, pain, examination, condition, consultation, normal, increased, control, changed, etc.

Text Analysis Methods
Our approach has three main phases: preprocessing which converts the text documents into itemsets, processing based on frequent pattern mining (FPM) techniques and elicitation of ARs, and postprocessing that filters, maps and generalizes rules by using context information and small lexicons (Fig. 2). The system processes input texts in unicode format and is language independent in principle (stemming and stopword filtering can be replaced by modules for another language).

Preprocessing
We have three text collections: SF -questions and comments in forum postings, S05 and S00 -the free texts of "Anamnesis" and "Patient Status" sections of ORs written by ESs and GPs correspondingly. Each text in SF, S05, and S00 is turned to a sequence of word stems in their original order, using blank spaces and punctuation delimiters as tokenization separators. Stop words and numbers may be essential for some patterns so they are preserved and generalized -replaced by the constants STOP and NUM correspondingly. After this step the punctuation is eliminated because it is often erroneosly written or missing in both forums and in ORs.
Let S be one collection. The vocabulary used in all documents of S will be called items W = {w 1 , w 2 , ..., w n }. For the collection S we extract the set of all different documents P = {p 1 , p 2 , ..., p N }, where p i ⊆ W . This set corresponds to transactions; the associated unique transaction identifiers (tids) shall be called pids (patient identifiers). Each patient interaction with a doctor (question or comment in SF or an anamnesis or patient status section of an OR in S00 and S05) is viewed as a single document in P.

Processing
Our documents are written in different styles: the forum texts have quite informal syntax structure while the ORs are written in telegraphic style with phrases rather than full sentences. Usually the ORs list attribute-value (A-V) pairs -anatomic organs/systems and their status/condition: E.g.: Кор -ритмична нормофреквентна сърдечна дейност, Крайници-хипестезия от дистален тип, везикуларно дишане. (Cardiovascular system -rhythmic norm frequent heartrate, limbs -hypoesthesia distal type, vesicular breath.) where A 1 =Cardiovascular system, A 2 =limbs and A 3 =breath are attributes and their corresponding values are V1=rhythmic norm frequent heartrate, V2=hypoesthesia distal type following A 1 and A 2 and V3=vesicular preceeding A 3 .
Attribute names contain phrases and abbreviations in Cyrillic and Latin. Values can be long descriptions in case of status complications.
The order of A-V pairs can vary and parts of the value descriptions can surround the attributes: where A=breathing and the surrounding words describe the status (attribute value).
It is also possible that some attributes share the same value: E.g.: хепар ет лиен неувеличени, Глава и шияб.о., Кожа и видими лигавици бледорозови, ... (liver and spleen not enlarged, Head and neck -without peculiarities, Skin and visible mucous membranes pale pink, ...) At the same time the informal texts in the health forum contain descriptions of patient status in a rather sparse form with much richer language and wider context around the triggering terms. E.g.: Здравейте, моля ви кажете какво да правя, сърцето ми прескача от над 3 часа и то доста често... (Hello, please tell me what to do, my heart skips over three hours and quite often...) Thus, when searching for frequent patterns, we consider a window of more than 10-12 words around each attribute. The rich terminology and flexible syntax structure hinder the application of traditional methods for extraction of collocations with gaps. These approaches would rather find the OR clishe phrases as collocations with highest frequency, moreover many A-V pairs would be erroneously considered as n-grams. Therefore we treat documents as bag of words rather than sequences, they are transformed to itemsets with single word occurrences only.
Given a set of pids S, support of an itemset I is the number of pids in S that contain I. We denote it as supp(I). We define a threshold called minsup (minimum support). A frequent itemset (FI) I is one with at least minimum support count, i.e. supp(I) ≥ minsup. The task of FPM of S is to find all possible frequent itemsets in S.
Most FPM algorithms generate all possible frequent patterns (FPs). The search space grows exponentially with the size of W . Summarized information for data relations can be extracted as maximal frequent itemsets (MFI). The condensed information not only accelerates the process, reducing redundancy, but also decreases significantly the number of frequent patterns for post-analysis.
An implication in the form I ⇒ J is called association rule, where I ⊂ W, J ⊂ W, I ∩ J = ∅. I is called antecedent or ancestor and J is called consequent. Support of a rule is the number of pids in S that contain I ∪ J, i.e.
If C% of the documents in S that contain I, contain also J, then the association rule I ⇒ J holds with confidence C in S, i.e. this is the condition probability conf (I ⇒ J) = P (J|I) = sup(I∪J) sup(I) . The task of ARs mining in collection S is to generate all ARs with confidence above the user defined confidence (minconf ) and support above user defined support (minsup). Rules that satisfy both a minsup and minconf are called strong.
However, even for reasonable values of minconf and minsup, big datasets yield huge sets of strong ARs. Thus we can use an additional filter called lift that is defined as the ratio of the confidence of the rule and the confidence of its consequent.
lif t(I ⇒ J) = P (I∪J) P (I)P (J) . The lift represents the strenght of the relation between the consequent and its antecedent. A lift value < 1 indicates independence between them. If the lift value is > 1, this indicates that the antecedent and consequent appear together more often than expected, i.e. are correlated. Such rules are potentially usefull for predicting the consequent in new sets.
For ARs generation we use algorithms for mining all ARs with the lift measure in a transaction database (Agrawal and Srikant, 1994) with implementation at SPMF 3 . In the experiments we applied the FPmax algorithm (Grahne and Zhu, 2003) for MFI and All Association Rulse with FP-Growth with lift (Han et al., 2004).

Postprocessing
In order to find certain correlation among rare items, the minimal support needs to be set rather low. This causes generation of a huge amount of ARs and most of them are redundant. Adapting methods of Ashrafi et al. (2007), we apply some techniques for redundant ARs removal. In addition we select only those ARs that fulfill the requirements to have support, confidence and lift above predefined thresholds minsup and minconf. The lift value is very important because it gives additional information about the usefulness of the generated rules. Thus we filter only ARs with lift > 1. This is a necessary condition but not a sufficient one. For instance we obtain ARs like: where STOP is a marker for a stop word, "везикулар" means "vesicular" (concerning breath) and "кожа" means "skin". This rule has high support (about 8%), confidence (0.999) and although its lift (1.429) is quite high, obviously it is useless. Thus we add some additional constraints, like removing all ARs with consequent that contains only the ST OP constant.
At the next step we perform AR generalization, based on small lexicons. We use some terms as seeds and rule based prediction about the features of other words that appear in similar rules at the same position. Initially generalization is applied for symptoms/conditions and complains: • For association rules R 1 : I ∪ X ⇒ J and R 2 : I ∪ Y ⇒ J in case X and Y are symptoms/conditions for the same anatomic organ/systems C, we replace them by the marker "ST AT U S(C)" and define a more general rule R C : I ∪ ST AT U S(C) ⇒ J.
• For association rules R 1 : I ⇒ J ∪ X and R 2 : I ⇒ J ∪ Y in case X and Y are symptoms/conditions for the same anatomic organ/systems G, we replace them by the marker "ST AT U S(G)" and define a more general rule R G : I ⇒ J ∪ ST AT U S(G).
The variety of all status conditions is huge but some complications are rare and it is unlikely to have them all included in ARs. The main generalization advantage is that more general rules will help to predict/recognize some status conditions in the text that are not included in our original lexicon. The main disadvantage is that not all markers ST AT U S(i) are equivalent, i.e. the status descriptions for different anatomic systems and organs can differ. Thus we can not apply further generalization of already generalized rules.
Finally the ARs generated for different collections are mapped to each other in order to study the specifics of the extracted collocations in formal and informal medical language. As result we are able to enrich the possible contexts of the medical terminology occurring in both types of text ( Figure 3). Thus we can define word embeddings for some terms included in ARs for all three sets. For some item w i ∈ W and collection S we define its context for all ARs in S such that: In particular when I ⇒ J ∪ w i holds in S then it also holds that I ⇒ w i . Thus C(w i ) = C S (w i ) is the observed context of item w i for all collections. The observation of the terminology context will help for further study of its nature.

Experiments and Findings
The experiments were performed on the collections SF , S00 and S05. SF consists of informal   Table 4: Evaluation of the generated association rules for 9 test sets text and is rather small. S00 and S05 contain sections of ORs. S00 corresponds to ORs produced by GPs and is significantly larger than the other two collections. It presents more sparse information than S05 which contains more focused information about the specific domain of diabetes. For each of SF , S00 and S05 three experiments are provided by non-exhaustive cross-validation (3 iterations on sets in ratio 4:1 training to test). The sets are denoted by their original set name and the number of experiment, e.g. for the set S05 there are three training sets S05.1, S05.2, and S05.3. Table 3 presents numbers of constructions found in the FPM experiments: pids, frequent itemsets, MFI and ARs. In ARs generation for all collections we used minconf=1.0 but different minsup depending on the sets' size. We considered only ARs with lift > 1.1.
Filter the rules with lift <1.1 and consequent STOP automatically reduced approx. 50% of the ARs for S05 and S00, where language is more formal with limited vocabulary and strong support. The same constraints lead to about 40% reduction of the ARs in SF which has rich vocabulary and small support. The distribution of the lift measure values in these two types of sets -formal vs informal was also quite different. Lift variation in S05 and S00 is much smaller than in SF . Table 4 presents the evaluation of the generated ARs. Two types of tests are performed: with all generated ARs and for the top 2,000 ARs according to their antecedents' cardinality. High precision in consequent prediction is seen for all sets.
Some ARs bring new knowledge about patient status description: корем ==> мек неболезн ( abdomen => soft non-painful) This rule has not too high support (about 5%) but its lift is higher than 1.1. It is a reasonable rule which connects lexical units describing an anatomic organ and its typical status.
The following AR has also very high lift -12.21 and represents a set of attributes and values describing the condition of a diabetic patient. All of them are terms or typical phrasal expressions in the domain of diabetes.
тургор видим шия ссс запаз общо състояни ==> розов кожа ( turgour visible neck ccc preserved general condition => pink skin) In the following OR excerpt, items from the AR antecedent are highlighted in blue and the predicted consequent items are highlighted in pink: The support sets of the ARs present patients with different profiles. In general several ARs can represent the conditions of different anatomic organs and systems of one patient and thus the initial profiling groups can be partitioned into profiling subgroups. Thus patients can be clustered in groups with similar health condition. A more detailed analysis of patients that belong to several profile groups is a task for further investigation.
Although generalization did not decrease significantly the total number of ARs, the new general rules help to process unseen status descriptions in other collections of medical documents.
ARs mapping. By collapsing equal ancestors of the ARs generated from the respective data sets we observe regularities which describe the medical language in the ORs on the one hand and the informal language from the medical forum on the other hand. Figure 3 presents ARs whose ancestor contain "thyroid gland". The larger ellipses denote the ancestors of the rules and the smaller ones linked through arrows are the consequents. SF rule1 and SF rule2 (in gray) are result of processing the SF dataset; S05ruleset1 and S05ruleset2 come from the dataset S05. Each of the rule sets represents several rules with equal ancestor. Among the extracted ARs from SF only 21 contain the terms "thyroid" and "gland" either in the ancestor or in the consequent whereas in the ARs resulting from S05 there are 3,985 ARs containing both terms. This shows that the informal language of non-specialists, although in medical topics, is rather difficult to formalize. The patterns in SF are short with up to 5 tokens in the ancestor vs. up to 14 tokens in the ARs coming from S05. Only 3 rules in SF have equal ancestors, in contrary the ancestors often can be collapsed in S05 and the same ancestor can have up to 62 different consequents. The ARs coming from S05 are also much longer, they describe related symptoms and/or diseases. As shown on Figure 3 the various consequents may be numerous but their vocabulary is not so rich. In S05ruleset1 there are only 3 tokens in the union of consequent vocabularies. Finally we note that the circles of mappings, as illustrated in Fig. 3, also suggest different profiles of patients experiencing eventual changes of thy-roid.

Conclusion and Further Work
By fusing information from ORs written in a formal language with informal medical forum texts and applying frequent pattern matching techniques, we manage to extract and generalize ARs describing the context of medical terminology in the domain of diabetes. The obtained patterns have the means of stable term subsets which occur in the documents with a window longer than the usually utilized ones. In this sense our approach delivers higher benefits for generating resources that describe the terminology in the Diabetes domain in comparison to the word frequency based techniques such as tf, tf-idf. Moreover, the FP matching techniques have the advantage of being unsupervised thus they do not require any external knowledge or annotated data and can successfully deal with big data.
We generalize ARs for the common terms in three sets: terms related to diabetes generated by processing SF and S05 as well as terms related to medical examinations obtained by processing of SF and S00. We generalize also rules describing the context of terms describing the patient status, output from the processing of S05 and S00. These resources shall be further employed for automatic analysis of formal and informal medical records, symptoms and condition recognition.


Rapid-Rate: A Framework for Semi-supervised Real-time Sentiment Trend Detection in Unstructured Big Data

Commercial establishments like restaurants, service centres and retailers have several sources of customer feedback about products and services, most of which need not be as structured as rated reviews provided by services like Yelp, or Amazon, in terms of sentiment conveyed. For instance, Amazon provides a fine-grained score on a numeric scale for product reviews. Some sources, however, like social media (Twitter, Facebook), mailing lists (Google Groups) and forums (Quora) contain text data that is much more voluminous, but unstructured and unlabelled. It might be in the best interests of a business establishment to assess the general sentiment towards their brand on these platforms as well. This text could be pipelined into a system with a built-in prediction model, with the objective of generating real-time graphs on opinion and sentiment trends. Although such tasks like the one described about have been explored with respect to document classification problems in the past, the implementation described in this paper, by virtue of learning a continuous function rather than a discrete one, offers a lot more depth of insight as compared to document classification approaches. This study aims to explore the validity of such a continuous function predicting model to quantify sentiment about an entity, without the additional overhead of manual labelling, and computational preprocessing&feature extraction. This research project also aims to design and implement a re-usable document regression pipeline as a framework, Rapid-Rate, that can be used to predict document scores in real-time.

I. INTRODUCTION
Document labeling and significant feature discovery are already widely researched areas in the domain of natural language processing. The most common use-cases of document classification are product & service review rating predictions, automatic grading of essays, spam detection, plagiarism detection etc. However, most of the current approaches used for these tasks rely on lexicon based methods with a centralized dictionary indicating the weights of a word towards a set of emotions or sentiment. Some other approaches use document level similarity scores like TF-IDF or n-gram count vectorization approaches to assess the class of a previously unseen document.
However, models like these are difficult to train for the following reasons: • Having to hand-pick features • Having to use stepwise regression to eliminate nonrelevant features • Having to manually re-train the models on a subset of the features to determine the best-fit All of the processes described above are time-consuming and tedious. Also, n-gram approaches only partially model the language context probabilities i.e. the probably of a word occurring next in the corpus depends only on the n-1 words that precede it, and TF-IDF loses word context altogether. The objective of this research project is to implement and evaluate a software framework to effectively eliminate the above steps from the process of training a language model. Word2Vec, for instance, takes additional word context features into account while building the language model, a feature which will be elaborated upon in Section VII-A.
The proposition is to rely on unsupervised strategies like continuous bag-of-words and the skip-gram model to learn word embeddings. These word embeddings will serve as a syntactic and semantic proxy for the documents being processed. The features thus learned, can then be used to train regression models to predict document scores.

II. SIMILAR WORK
This section discusses the previous work done on mining and analytics pertaining to text data.
Most of the similar work in the area of learning wordembeddings is related to classifying documents into a given set of labels or topics. The purpose of this study is to extend the usage of the document vector representation to demonstrating  [10] its usage when the response variable (the output of a learning task) a continuous function (document-scoring) rather than discrete values (document classification). This work can be viewed as a generalization of the previous approaches, because this study aims to predict fine-grained document scores, rather than coarse document classifications.
Most of the current work in usage of regression techniques on text content are related to meta-data or features extraction [24] [25]. Similarly there have been several studies to identify document tags using classification techniques [12] [23]. However, there is a relative absence of studies to evaluate the accuracy of regression models that try to predict a document score using a continuous-function model.
As far the the author is aware, there has been no similar work to test the regression accuracy of paragraph vectors in an experimental research setup before, and is hence, novel in that respect. The evaluation of the effectiveness of this approach will be assessed using a generic evaluation metric, the coefficient of determination score, which compares the predicted score-set against the golden set of actual scores.

III. PROBLEM STATEMENT
The problem statement can be formulated as an evaluation of Paragraph Vectors [20] on a document score regression task to evaluate the accuracy of the prediction model that can be built using the shallow neural network language model learning algorithm previously described by Mikolov et al. [21], which was primarily used to train word embeddings in an unsupervised manner.
The prediction scores will be evaluated using the co-efficient of determination metric, also known as the R 2 metric [15]. The hypothesis is to prove that the document ratings have a positive correlation with the vectorized versions of the document text content, because the semantics of positive and negative reviews are expected to be captured by the shallow neural network trained on the corpus. The expected R 2 metric score being over 0, will confirm the hypothesis, and a negative value will contradict it.

IV. MOTIVATION
The motivation for the project is to prove that a substantial amount of the semantic signal about sentiment with respect to a product or service, as extracted from a snippet or document of text, should prove to be a good indicator of the labeled document scores. If proven to be true, the objective includes building a real-time streaming framework, which can analyze unseen reviews from the corpus of unlabeled data, and be able to emit a score-prediction with minimal processing delays. This framework can be set-up on commodity hardware, and a given business can simply pipe their unlabeled reviews or articles into the system, and expect a graphical representation of the sentiment about a particular topic that is updated in real-time. The framework also allows rules to be set for alerts, so, for instance, if the approval rating of a particular business falls below a user-defined threshold, an email alert could be triggered to the business decision makers to respond quicker to the negative sentiment, a process which would otherwise span several business days.

A. Semi-supervised Learning
The statistical learning formulation of the problem statement in this study, can be broken down into two distinct segments, namely: • Learning a projection of the document into vector space. This projection will act as a vector-space proxy for the sentiment that the document is trying to convey. This segment is completely unsupervised, and does not require any tuning or preprocessing to compute. • Learning a relation between the aforementioned projection of each labeled document, and the associated score or  [22] label. This segment of the learning pipeline is supervised relies on previously scored or rated documents.

B. Continuous function learning
Assume that a given output variable y depends on a variable x such that where x represents a vector of independent variables, y represents the dependent or response variable, β and c are model vector space functions that are applied to x to map it to y, and is the error term. Now, assume that the variable x increases by a small amount ∆x, and that x 0 was the initial value, such that the value of x is updated to A continuous function is defined as one in which an infinitely small change for the input variable x results in a corresponding infinitely small change in the response variable y[3], which can be expressed as Continuous functions are typically needed when the response variable calculated is a measurable quantity, rather than a class label. Since sentiment as fine-grained values offer more depth in terms of sentiment insight [16], the study in this paper is formulated as a regression problem.

V. NATURAL LANGUAGE UNDERSTANDING DOMAIN CHALLENGES
The section elaborates on the NLP domain-specific challenges faced during the implementation of this research project, and the proposed strategies used to tackle these challenges or minimize their impact on the final outcome of the experiment.

A. Language Model Learning
Language model learning is complicated because the most popular modeling approaches currently used have inherent limitations when it comes to being able to be used reliably for language understanding accuracy across the board. A few such instances are described in this section.
1) Lexicon based model learning: Lexicon based models are reliable for one-off tasks, but they require hand-annotation of the polarity of words, and an over-dependence on the fact that the lexicon of words needs to be updated regularly as and when it comes across new words. This requires time and effort and is rather tedious. This challenge is addressed in the research project, by using an unsupervised method to learn the document vector representations, and not relying on any lexicon for details about word or document semantics.
2) Statistical model based learning: TF-IDF and similar document scoring approaches, are based on the bag-of-words model. Hence, it does not capture position in text, semantics, co-occurrences in different documents, and other such features. Also, this shortcoming means that it cannot capture semantics of a word. It relies heavily on occurrence count across the entire corpus of documents.
3) Cross domain applicability: One of the major drawbacks of statistical model based learning in particular is that a language model trained on a dataset belonging to a particular domain is highly unlikely to yield good results when assessed or tested on another domain. For instance, a language model trained for detecting sentiment on a movie review data-set can't be used to accurately predict sentiment on a stockmarket discussion dataset. This is due to the inherent business domain specific differences in the vocabulary and connotations that will unlikely be modeled when trained on a completely different dataset than the one that the model needs to be tested upon. This problem is not directly tackled for this study, but the author proposes a potential direction in the Section XII which may help address this problem in future versions of Rapid-Rate.

B. Sarcasm/Irony Detection in Text
One of the biggest hurdles with accurately inferring sentiment of a document despite having a robust trained model, or even a lexicon, is the presence of sarcasm and irony, which are extremely likely for unmoderated comments and articles, especially on forums and social media. For now, there is no clear best-approach for sarcasm and irony detection, and often, the best features for detecting such sentiment is text meta-data like emoticons and tags rather than the text itself, as discussed by González-Ibánez et al. [17].

C. Deep Learning
The challenge with using deep learning for the model training is there deep learning models typically take hours to even days to completely train. If a system needs to be setup from scratch, deep learning does increase the time to market by being notoriously difficult to train. This challenge addressed by incurring only a one-time cost of training the labeled document corpus. Future iterations of the training does not block the current execution of the real-time prediction component.

VI. ENGINEERING DESIGN & IMPLEMENTATION CHALLENGES
The previous section described the natural language understanding challenges. This sections aims to describe the engineering design and implementation challenges that needed to be tackled during the implementation phase.

A. Unstructured Big Data
Unstructured data is defined as data, large amounts of it, in this case, which do not abide by a strict schema [14]. This happens when the process of data generation itself is unmoderated and does not impose too many constraints on the type of data allowed. It could also be that the data contains soft links or references to other entities within the data. As such, the onus is on the application to process large amount of unstructured data and attempt to extract some meaning and relations from it. This is tackled by adopting a Paragraph Vector approach to convert variable-length text into fixedlength vectors, described in detail in Section VII-B.

B. Distributed Document Vector Learning
The preference for the most computationally intensive tasks is to distribute them by scaling horizontally. This allows a driver to submit multiple chunks of the same job to a number of workers. These workers can then convey the output to the driver once they finish their respective jobs, and the driver can aggregate these results. However, the library used for this study, Gensim, does not, by default, include a distributed setup for document model learning. Instead the onus is on the implementation to integrate Gensim with a distributed processing framework like Spark or Hadoop. The current implementation of Rapid-Rate uses a low-memory footprint iterator based technique to reduce the system memory usage on a single node, for the off-line document vector learning task described in Section IX-A. This could be scaled to train even higher volumes of documents if integrated with a distributed processing framework.

C. Optimal Setup for Production Execution
The motivation behind building a framework, is to be able to quickly deploy it to a typical production environment in an industrial setting. Hence the decision to separate the business logic of the document vector training component This allows for easy maintenance if the off-line module needs to be temporarily paused for maintenance. Similarly, if the always-running daemon job for the on-line module needs to be temporarily paused, the streaming documents are not lost. The distributed queue keeps the documents queued, along with the offset of where the on-line module needs to resume reading from. This ensures that the on-line module can simply proceed from where it left off before being taken down for maintenance.

VII. BACKGROUND
The sections touches upon the background information that needs to be known prior to doing a deep-dive into the details of the research study and implementation done for this project.

A. Word2Vec
Word vectors are a step-away from the traditional bag-ofwords + n-gram model. The latter ends up losing context of the words because of the fact that the words that follow it are not taken into account, and only the preceding words are. The Word2Vec model [21] can take into consideration both preceding and following words, as a sliding window of text, to compute the probability of occurrence of the current word, as part of the Continuous Bag-of-words (CBOW) model. Alternatively, Word2Vec also provides a skip-gram model [22], which conversely, uses the context probabilitie of the current word, to predict the most likely words to precede and follow it. These word embeddings can be used to derive both syntactic as well as semantic relations in the vector space of the language model. A representation of this is given in Figure 1.

B. Paragraph Vectors
Paragraph vectors are a similar concept to Word2Vec, but rely on computing a fixed-length vector representation of a variable length group of words, which could be a sentence, a paragraph, or even an entire document.
Since vectors representations of documents seem to outperform bag-of-words models for the purposes of semantic summarization and document classification tasks, it follows that a vector representation based approach should be used to evaluate performance on a regression task as well. Paragraph vectors also address some of the key weaknesses of bag-ofwords models. First, they inherit an important property of the word vectors: the semantics of the words. The second advantage of the paragraph vectors is that they take into consideration the word order, at least in a small context, in the same way that an n-gram model with a large n would do. This is important, because the n-gram model preserves a lot of information of the paragraph, including the word order [20]. The paragraph representation framework is depicted in Fig. 2.
The core concept here, is that an aggregation of the word vectors are being associated with a paragraph ID that encapsulates all of the words in question. Thus, rather than just assigning each individual word a vector representation, each paragraph/group of words in the body of the text document have been assigned a vector representation.
The implementation of paragraph vectors used for the purpose of implementing the system architecture in this study is Doc2Vec, from the Gensim library [4].

C. Statistical Learning Models
Once the entities about which we wish to learn a representation are projected into vector space, there are an abundance of statistical learning methods that can be used to learn a continuous function including linear regression, polynomial regression, ridge regression, stepwise regression, lasso regression et. al.
This study utilizes linear regression and support vector regression to test the veracity of the model predictions. Both these methods are described in the below sub-sections.
1) Linear Regression: The problem of regression in mathematical statistics is characterized by the fact that there is insufficient information about the distributions of the variables under consideration[8].
In the case for this study, once the Doc2Vec implementation generates the vector space representation of each of the documents, each of the learned dimensions represent a different input variable for the regression problem. In reality, each of these dimensions represent a weighted probability score generated by the hidden layer of neurons in the paragraph vector model.
2) Support Vector Regression: Support Vector Regression (SVR) attempts to minimize the generalization error bound so as to achieve generalized performance. The idea of SVR is based on the computation of a linear regression function in a high dimensional feature space where the input data are mapped via a nonlinear function. SVR has been applied in various fields time series and financial (noisy and risky) prediction, approximation of complex engineering analyses, convex quadratic programming and choices of loss functions [11].
The input parameters to the support vector regression model remain the same as those described above for the linear regression model.

D. Distributed Messaging with Kafka
This project uses Apache Kafka for processing huge volumes of data from real-time streams. Like a messaging system, Kafka employs a pull-based consumption model that allows an application to consume data at its own rate and rewind the consumption whenever needed. By focusing on log processing applications, Kafka achieves much higher throughput than conventional messaging systems. It also provides integrated distributed support and can scale out. Kafka has been used successfully at LinkedIn for both off-line and on-line applications. [19]

E. Horizontal Scaling with Spark Streaming
This study used Apache Spark Streaming for the realtime processing of the unlabeled document data. Discretized streams (D-Streams), a stream programming model for large clusters that provides consistency, efficient fault recovery, and powerful integration with batch systems is the core offering of Spark Streaming. The key idea is to treat streaming as a series of short batch jobs, and bring down the latency of these jobs as much as possible. This brings many of the benefits of batch processing models to stream processing, including clear consistency semantics and a new parallel recovery technique that is a truly cost-efficient recovery technique for stream processing in large clusters [26].
VIII. SYSTEM ARCHITECTURE This section describes the proposed architecture design, including the constituent components and the flow of the control and data throughout the architecture. The system architecture is depicted in Fig. 3 The individual components and their interactions are described in the below sub-sections VIII-A and VIII-B.
A. Components 1) Document embedding tool: The labeled documents are the initial input parameters to the Rapid-Rate framework. These are variable length text snippets or paragraphs, each with an associated score. This tool is used to convert the document text into their corresponding vector representations. 2) ML Modeling tool: This is a library used by the framework to infer a model from the vectorized documents (independent variable) that predict the rating/score of that particular document (dependent variable). It uses algorithms such as stochastic gradient descent in an attempt to converge to a lowest possible value of learning error [13].
3) Text Aggregation: This component is responsible for merging multiple different streams of data into a unified stream that can be used to predict sentiment ratings of unlabeled documents. This can be implemented in frameworks like Apache Flume or Apache Storm, which allow definitions of multiple sources and sinks, with a topology for data processing as part of the aggregation business logic.

4) Message Broker
Queue: This component acts as the arbiter for the incoming unlabeled documents. Each document is read from the queue and is processed in turn. The queue is capable of holding a large buffer, as well as an offset for each of it's consumers, to serve as a memory or bookmark of where the consumer stopped during their last subscription to the queue's updates.

5) Stream Processor:
The stream processor is responsible for reading and processing the unlabeled documents, predicting the score/rating of each incoming document, and forwarding the scores for further usage. This component's setup is distributed, and is also able to subscribe to the updates being pushed by the message broker queue (Section VIII-A4).
6) Time-Series Database: A time series database is optimized to handled collections of data objects that are indexed by time. It supports fast querying of data over temporal scan ranges and is ideal for time-series based analytics 7) Analytics Dashboard: The analytics dashboard provides a visual representation of overall sentiment over time, and supports setting alerts to warn the end-user when overall sentiment dips below a certain user-defined threshold.

B. Component Interactions
A simple processor reads the labeled documents and feeds it into the Document embedding tool, which, in turn, emits a representation of each of the labeled documents as a vector. The vectors representations of the documents are now associated with their corresponding scores and the vector representations and the scores fed together into the ML Modeling tool to generate Linear Regression and Support Vector Regression models. These models are then 'pickled' or flushed to disk, to be used by the real-time prediction module. Along with these models, the trained Document embedding model is also flushed to disk.
Once these models are flushed to disk, the system can kick-start it's real-time prediction framework. The unlabeled documents are published to the message broker queue, to which the daemon component of the architecture is subscribed to. It processes lists of documents in micro-batches and emits a list of scores. These scores are posted in real-time to a timeseries database, which in turn, acts as the data back-end for the analytical dashboard.
The analytical dashboard provides a line graph of the sentiment trend over time, and can also provided for additional features such as Bollinger Band plotting, and threshold breaching alerts.

IX. IMPLEMENTATION
This section briefly discusses the implementation specifics chosen for the code-base, and the architectural decisions that govern this particular framework. This is helpful to know in case a new application to be chosen to act as a pipeline into, or a sink of the data processed by the Rapid-Rate framework.
The primary language of implementation of this research project is Python, and the other frameworks and components have been described in the Tech Stack section. (Section IX-C)

A. Off-line learning module
The off-line learning module is comprised of the shallow neural-network implementation using the library Gensim (Section IX-C2). The algorithm of this module is shown in Algorithm 1.

B. On-line real-time prediction engine
The on-line prediction module is the real-time component submitted to Spark (Section IX-C5) workers. Each of the spark workers listen to a distributed queue in Kafka (Section IX-C4). As and when the unlabeled documents are streamed, the engine reads the streamed data and predicts a score before posting it into OpenTSDB (Section IX-C6). The general algorithm for handling the streaming data is described in Algorithm 2. A sub-section of this module, the algorithm used to infer sentiment, is described in Algorithm 3.  2) Gensim: Python library. The Doc2Vec class in Gensim is used to learn document vector representations [4].
3) Scikit-Learn: Python library. Enable the computation of linear regression and support vector regression models [9]. 4) Apache Kafka: Distributed messaging queue. Used to receive new unlabeled documents, and is also a potential sink for dumping the results of the result ratings of unlabeled documents [19].

5) Apache Spark Streaming:
The real-time prediction component written in Python is submitted as a Spark Streaming job [26]. 6) OpenTSDB: Time series database. This database houses each individually predicted document rating and their corresponding time-stamps [5].

7)
Bosun: Bosun comprises of 2 things -a graphical interface with which to perform easy temporal scans, and a configurable alerting system based on the rules on the incoming values [2].
The chosen tech-stack is state-of-the-art for framework design and development, with a strong developer community and guaranteed long-term support for each, making the framework future-proof for the foreseeable time.

A. Testbed
The testbed is integrated into the actual framework, and the processors for model training evaluate the model accuracy.
The tests were carried out on an Amazon document corpus of 600,000 unique product reviews. The complete dataset of Amazon reviews is openly available to download [1].

B. Metrics
The experimental results are being evaluated for goodness of model fit using the co-efficient of determination metric [18]. The co-efficient of determination conveys the quantized amount of signal of the input vector that is being captured by the trained model by validating it against a test set. The value of the co-efficient of determination is derived from the below equations: The R 2 score derived in Equation 7 is the the co-efficient of determination use to predict the goodness of fit for each of the machine learning models learned.

C. Experimental Results
The accuracy scores of the experimental results are presented in Table I. The score presented in this table is the coefficient of determination or R 2 score. The positive scores of both the experiment analyses indicate that the model does capture a good amount of the sentiment in each of the labeled documents, and ensures a reasonable amount of accuracy for the prediction of the real-time unlabeled documents.
As indicated in Figure 5, the current trend seems to suggest at least a polynomial-function improvement in the accuracy with respect to increasing the number of documents to be trained on. The initial experimentation was done on a corpus of 100,000 labeled documents. The subsequent experiments used corpora with 300,000 documents and eventually 600,000 documents.
Also, from the code profiling information presenting in Figure 4, it is evident that the unsupervised learning segment of obtaining the document vectors of the labeled documents is the longest running part of the off-line learning module.
In comparison, the document processing and statistical model training is negligible.

XI. CONCLUSIONS
To summarize, this research project encapsulated the below objectives: • Evaluating the usage of document embeddings for a regression task • Building a framework to predict real-time sentiment trends Given that the attempt is to infer sentiment from unstructured data, the result of having a positive R 2 value is promising (Table I). It indicates that the model captures a significant amount of the signal present in the independent variable. It can hence be relied to a degree of confidence to predict a close enough value to the actual intended document sentiment.
Additionally, this work has resulted in a generic framework that can be adopted to build a sentiment analytics dashboard on a user friendly interface. This framework can be extended to process any type of document content both for vector training and for the real-time sentiment prediction module.
The overall framework has been built using state-of-the-art product offerings for neural-net document embedding libraries, distributed stream processing in Spark, distributed queue architecture in Kafka, and the stable machine learning framework which is a part of scikit-learn. The author believes this framework can be used both in industrial settings as well as in evaluation testbeds for any document embedding and machine learning libraries used as an ensemble architecture. The hypothesis made prior to undertaking this research project, as described in the Problem Statement (Section III) has been validated and proved to be true.
The framework itself, is open-sourced, and is free to be used to replicate the results or to be extended independently [7].

XII. FUTURE WORK
With respect to the statistical models trained, the current approach supports only the simplest regression models i.e. Linear Regression and Support Vector Regression (SVR). This could be expanded to offer additional training mechanisms that can integrate more sophisticated regression algorithms for a higher level of accuracy.
Additional channels could be chosen as the culmination point of the newly rated unlabeled documents. For now, the data is stored in a time-series database. But this can easily be converted into something more generic, like publishing it to another Kafka queue. Multiple consumer applications can, thus, benefit from the stream of insights published to the stream.
With reference to the natural language understanding challenge described in Section V-A3, a proposed solution could be to merely obfuscate the entities being spoken about the the document for which the sentiment analysis model is being trained for. Similarly, the obfuscation also needs to be done for the test dataset. Since this removes the dependence of the sentiment in the document to the sentiment expressed in it, it would logically allow for a more generic prediction model. Although this won't completely mitigate the presence of domain-specific jargon, it does allow for a more accurate prediction model, and remains to be evaluated.
Another potential aim for future work is to reduce the time taken for the initial document vector training of the off-line model. As shown in the code profiling statistics present in Figure 4, it is evident that a significant reduction in the cost of training the initial model could be achieved if the document vectors training time were to be reduced. This can be achieved by the horizontal scaling described in Section VI-B.

XIII. ACKNOWLEDGMENTS
The author would like to thank Dr. Paulo Alencar, for his guidance, feedback and suggestions during the conceptualization of this research project.
The author would also like to thank Dr. Olga Vechtomova for her feedback and ideas on document evaluation and sentiment polarity tasks.


extfieldsofstudy:
Computer Science
Mathematics

text:
On the Joint Impact of Hardware Impairments and Imperfect CSI on Successive Decoding

In this paper, a spatial multiplexing multiple-input multiple-output (MIMO) system when hardware along with RF imperfections occur during the communication setup is analytically investigated. More specifically, the scenario of hardware impairments at the transceiver and imperfect channel state information (CSI) at the receiver is considered, when successive interference cancellation (SIC) is implemented. Two popular linear detection schemes are analyzed, namely, zero forcing SIC (ZF-SIC) and minimum mean-square error SIC (MMSE-SIC). New analytical expressions for the outage probability of each SIC stage are provided, when independent and identically distributed Rayleigh fading channels are considered. In addition, the well-known error propagation effect between consecutive SIC stages is analyzed, while closed-form expressions are derived for some special cases of interest. Finally, useful engineering insights are manifested, such as the achievable diversity order, the performance difference between ZF- and MMSE-SIC, and the impact of imperfect CSI and/or the presence of hardware impairments to the overall system performance.

Performance assessment of either ZF-or MMSE-SIC has been extensively reported in the technical literature to date (e.g., see [2]- [13] and references therein). Nevertheless, all the previous studies assumed perfect channel state information (CSI) at the receiver and/or a non-impaired hardware at the transceiver; an ideal and a rather overoptimistic scenario for practical applications. More specifically, the hardware gear of wireless transceivers may be subject to impairments, such as I/Q imbalance, phase noise, and high power amplifier non-linearities [14], [15]. These impairments are typically mitigated with the aid of certain compensation algorithms at the transceiver. Nevertheless, inadequate compensation mainly due to the imperfect parameter estimation and/or time variation of the hardware characteristics may result to residual impairments, which are added to the transmitted/received signal [14]. Moreover, an erroneous CSI may occur due to imperfect feedback signaling and/or rapid channel variations. It can cause crosstalk interference (see [16] and [17] for explicit details on this effect) within the SIC process, while it can affect the detection ordering [11]. It is noteworthy that an analytical performance assessment of ZF-and/or MMSE-SIC under the aforementioned non-ideal communication setup (i.e., impaired hardware at the transceiver and imperfect CSI) has not been reported in the open technical literature so far.
Capitalizing on the above observations, current work presents a unified analytical performance study of ZF-and MMSE-SIC for non-ideal transmission systems. The ideal (traditional) scenario is also considered as a special case. Particularly, the ordered ZF-SIC scheme is considered, where the suboptimal yet computational efficient Foschini ordering is adopted (i.e., strongest stream is detected first, while weakest stream is detected last, upon the ZF equalization). It should be mentioned that the norm-based Foschini ordering requires only m(m + 1)/2 − 1 comparisons, with m denoting the number of transmit antennas. This represents a remarkable computational gain over the optimal ordering, which operates over an exhaustive search of m! combinations. Interestingly, it was recently demonstrated that Foschini ordering coincides with the optimal one, in the case when the transmission rate is uniformly allocated among the transmitters [4]. Additionally, the scenario of MMSE-SIC with a fixed-ordering is analytically presented and studied, which can serve as a benchmark for the more sophisticated ordered MMSE-SIC scheme.
The contributions of this work are summarized as follows: • New closed-form expressions for the outage probability for both the ordered ZF-SIC and unordered MMSE-SIC schemes are derived. These expressions are reduced to the corresponding conventional outage probabilities, when ideal systems are assumed with perfect CSI and without hardware impairments at the transceiver. • The well-known error propagation effect between consecutive SIC steps is analyzed for the general scenario. Relevant closed-form expressions are provided for some special cases of interest. • A new MMSE linear filter is presented when the variances of noise, hardware impairments and imperfect CSI are known. Based on this filter, a substantial performance gain of MMSE-SIC over ZF-SIC is observed. • Simplified expressions in the asymptotically high SNR regime are obtained, revealing useful engineering insights, achievable diversity order and impact of nonideal communication conditions to the overall system performance.
The rest of this paper is organized as follows: In Section II, the system model is presented in detail. New analytical performance results with respect to the outage probability of ZF-and MMSE-SIC are derived in Sections III and IV, respectively, while relevant asymptotic approximations are provided in Section V. The error propagation effect is analyzed in Section VI. Moreover, numerical results are presented in Section VII, while Section VIII concludes the paper.
Notation: Vectors and matrices are represented by lowercase bold typeface and uppercase bold typeface letters, respectively. Also, [X] ij , denotes the element in the ith row and jth column of X, (X) −1 is the inverse of X and x i denotes the ith coefficient of x. The superscript (.) H denotes Hermitian transposition and |.| represents absolute (scalar) value. In addition, I v stands for the v × v identity matrix, E[.] is the expectation operator, d = represents equality in probability distributions, Pr[.] returns probability, while o(.) is the Landau symbol (i.e., f (x) = o(g(x)), when f (x)/g(x) → 0 as x → ∞). Also, f X (.) and F X (.) represent probability density function (PDF) and cumulative distribution function (CDF) of the random variable (RV) X, respectively. Complex-valued Gaussian RVs with mean µ and variance σ 2 , while chi-squared RVs with v degrees-of-freedom are denoted, respectively, as CN (µ, σ 2 ) and X 2 2v . Furthermore, Γ(a) (a − 1)! (with a ∈ N + ) denotes the Gamma function [18,Eq. (8.310

II. SYSTEM MODEL
Consider a point-to-point MIMO system where the transmitter and receiver sides are equipped with m and n ≥ m antennas, respectively. The input-output relation of the received signal stems as [19] y = H (s + n T ) + n R + w, where y ∈ C n×1 , s ∈ C m×1 and w ∈ C n×1 denote the received, the transmit and the circularly symmetric Gaussian noise signal vectors, respectively. In addition, n T ∈ C m×1 and n R ∈ C n×1 correspond to the distortion noise due to residual hardware impairments at the transmitter and receiver, respec-tively. 1 Moreover, H ∈ C n×m is the channel matrix, while assuming that the coefficients of H d = CN (0, 1), i.e., a Rayleigh flat fading scenario. Also, E[ww H ] = N 0 I n , where N 0 is the noise power, while E[ss H ] = pI m is assumed, where p denotes the transmitted power per antenna. Typically, n T and n R are Gaussian distributed (see, e.g., [19] and references therein), i.e., n T d = CN (0, pκ 2 T I m ) and n R d = CN (0, pκ 2 R mI n ), where κ T and κ R denote the level of residual impairments 2 at the transmitter and receiver, respectively. It is noteworthy that the variance of residual impairments is proportional to the transmission power per antenna [19,Eqs. (7) and (8)]. Also, the last two terms of (1), i.e., n R + w, denote the total postnoise added onto the received signal, which can be modeled as CN (0, (pκ 2 R m + N 0 )I n ). In the ideal scenario where {κ T , κ R } = 0 (i.e., no hardware impairments), (1) is reduced to the conventional MIMO signal relation, given by Further, in the rather realistic scenario when imperfect CSI occurs, the estimated channel at the receiver is given bŷ whereĤ is the estimated channel matrix, ∆H ∈ C n×m stands for the channel estimation error matrix, while the coefficients of ∆H d = CN (0, ω) with ω representing the channel estimation error variance [11]. Also, H and ∆H are statistically independent [21].
In the following, we turn our focus on two quite popular linear detection schemes, namely, ZF and MMSE. These schemes, combined with SIC, are extensively used in spatial multiplexing transmissions [3].

A. ZF-SIC
In principle, ZF-SIC enables spatial multiplexing transmission, i.e., it can distinguish the received streams from different users and/or antennas with the aid of spatial structures (individual spatial signatures) of the signals to be detected [24]. It is performed in three main steps, namely, the symbol ordering that aims to enhance the overall reception performance, the interference nulling via ZF from the yet-to-be detected symbols, and the interference cancellation from the already detected symbols. These steps are performed in a number of consecutive stages, until all given symbols are successfully decoded.
The interference nulling can be efficiently implemented by applying the QR decomposition on a given channel matrix, which is widely adopted in ZF equalizers, since it provides computational complexity savings [25]. LetQ be a n × n unitary matrix (with its columns representing the orthonormal ZF nulling vectors) andR an n × m upper triangular matrix, givenĤ. Accordingly, Q and R correspond to the true channel matrix H. It follows from (2) that Hence,Q H y is performed at the receiver, yieldinĝ Interestingly, it has been demonstrated in [11,Eq. (30)] and [16,Eq. (16)] thatR ≈ R, whereas the resultant approximation error can be considered negligible in terms of distributions [11]. Also, note that the latter approximations become exact equalities in the case when perfect CSI is available. Thereby, (4) can be reformed aŝ Thus, the sequential signal decoding, which involves the decision feedback, is given by whereŝ i is the estimated symbol of the ith detected stream, r ij (or r ij ) is the coefficient at the ith row and jth column of R (or R) and Q[.] stands for the slicing operator mapping to the nearest point in the symbol constellation.
Therefore, based on the unitary invariant property of Gaussian vectors (i.e., isotropic distribution [26, Theorem 1.5.5]), the signal-to-interference-plus-noise-and-distortion ratio (SINDR) of the ith decoding layer 3 for ZF-SIC is expressed as Notice that in the ideal scenario of perfect CSI and no hardware impairments, (6) becomes the classical SNR expression of the ith layer, since SNR i = pr 2 ii /N 0 [6].

B. MMSE-SIC
Unlike ZF-SIC, the more sophisticated MMSE-SIC detector achieves an optimal balance between interference suppression and noise enhancement. To this end, it requires the knowledge (or estimation) of the noise variance and, thus, it represents the optimal linear detection scheme [27, App. A]. Since the main difference between ZF-and MMSE-SIC is in the equalization process, we retain our focus on the discussion of the typical MMSE, while the description of the more advanced MMSE-SIC is provided subsequently.
The conventional MMSE (non-SIC) detector strives to minimize the mean-square error (MSE) of the jth transmitted stream, i.e., s (j) , as follows where g (j) is the optimal weight vector andŷ denotes the post-detection received signal, subject to channel estimation imperfections and hardware impairments of the transceiver.
To facilitate the analysis, we can formulateŷ as the classical MIMO model where w ′ (H + ∆H) n T +∆Hs+n R +w with a (colored) noise covariance matrix given by [19,Eq. (9)] Based on (7), it can be seen that (see Appendix A for details) whereas, after some straightforward manipulations (see Appendix A), the total SINDR of the jth stream is obtained as [29, Eq. (5)] Based on Woodbury's identity [30, Eq. (2.1.4)], (12) reads also as where The form of (13) is preferable than (12) for further analysis, because h j and K j are statistically independent. Also, in ideal conditions of perfect CSI and no hardware impairments, (13) is reduced to the classical signal-to-interference-plus-noise ratio (SINR) expression of MMSE detectors [7,Eqs. (11) and (13)] On the other hand, when MMSE-SIC is applied to the receiver, the corresponding SINDR of the ith SIC step (1 ≤ i < m) can be expressed as since no inter-stream interference is experienced at the last SIC stage. 4

III. PERFORMANCE ANALYSIS OF THE ORDERED ZF-SIC
In this section, closed-form formulae with regards to the outage performance of the ordered ZF-SIC for each transmitted stream are provided. We start from the general scenario, when both CSI errors and hardware impairments are present, followed by some simplified special cases of interest.

A. General Case
We commence by deriving the CDF of the SINDR for each transmitted stream, which represents the corresponding outage probability, as follows.
In fact, (16) represents the optimal combining scheme in interference-free environments. In other words, it coincides with the maximal ratio combining (MRC) scheme, when imperfect CSI and hardware-impaired transceivers are present. Notice that when {ω, κ T , κ R } = 0, (16) is reduced to the classical SNR expression of MRC. where Notice that the condition κ 2 T < 1/γ th should be satisfied, which is typically the case in most practical applications. Thus, it holds that where P (i) out (.) denotes the outage probability for the ith stream. To proceed, we have to determine the distributions of the mutually independent RVs, namely, Y i and pr 2 ii .
Proof: From [11], while conditioning on R, in a similar manner as in [31,Eq. (11) On the other hand, f pr 2 ii (.) depends on the precise ordering that is adopted. In current study, the classical Foschini (normbased) ordering is investigated, where the strongest stream is decoded first while the weakest stream is decoded last. It was recently demonstrated that the Foschini ordering coincides with the optimal ordering in the case when the transmission rate is uniformly allocated among the transmitters [4]. Lemma 2: In the case when Foschini ordering is applied, is given by where Proof: The detailed proof is relegated in [5]. In the simplified scenario of fixed symbol ordering (i.e., no ordering), f pr 2 ii (.) d = X 2 2(n−i+1) [8]. Thereby, in this case, We are now in a position to formulate the outage probability for the ordered ZF-SIC as follows: Theorem 1: Outage probability for the ith decoding layer is obtained in closed-form as where Proof: The proof is provided in Appendix B. It is noteworthy that the derived result includes finite sum series of simple elementary functions and factorials and, thus, can be efficiently and rapidly calculated. 5

B. Imperfect CSI without hardware impairments
In this case, the system suffers from imperfect CSI, which in turn reflects to channel estimation errors, but it is equipped with ideal hardware. The corresponding outage probability of each stream is directly obtained from (23), by setting κ T = κ R = 0.

C. Perfect CSI with hardware impairments
This scenario corresponds to the case when channel is correctly estimated (e.g., via pilot or feedback signaling), but the transmitted and/or received signal is impaired due to lowcost hardware equipment at the transceivers.
Proposition 1: The exact closed-form outage probability of the ith stream under perfect CSI conditions with hardware impairments is expressed as Proof: The proof is given in Appendix C. 5 At this point, it should be mentioned that the auxiliary parameters Ξ i and Ψ i include the required multiple nested sum series, while they are introduced for notational simplicity and presentation compactness.

IV. PERFORMANCE ANALYSIS OF MMSE-SIC WITH FIXED ORDERING
A closed-form expression for the PDF/CDF of SINDR with regards to the ordered MMSE-SIC is not available so far. To this end, we retain our focus on the unordered (fixed) MMSE-SIC scenario in this section, which can be used as a benchmark and/or as a lower performance bound for the more sophisticated ordered MMSE-SIC scheme.
Theorem 2: Outage probability of the ith SIC stage, when 1 ≤ i < m, is derived in a closed-form as given by (25) and for the mth SIC stage as Proof: The proof is provided in Appendix D. In general, γ th < 1 (κ 2 T (ω+1)+ω) should hold for the evaluation of every SIC stage (see Appendix D for details). When the latter condition is not satisfied, an outage occurs with probability one. As previously mentioned, typically κ T ≤ 0.175 [22]. Moreover, practical values of ω could not exceed 30% (i.e., ω ≤ 0.3), because higher values of channel estimation error reflect to a rather catastrophic reception [11], [21]. Thereby, based on the latter extreme values, γ th < 4.27dB is required. Equivalently, since γ th 2 R − 1 (where R denotes a target transmission rate), R < 1.88bps/Hz is required for a feasible communication. Nonetheless, higher γ th values can be admitted for more relaxed CSI imperfections and/or hardware impairments, while there is no constraint in the ideal scenario.
Moreover, notice that the special cases of non-impaired hardware or perfect CSI are directly obtained by setting κ T = κ R = 0 or ω = 0 in (25) and (26), respectively.
Corollary 1: The ideal scenario of non-impaired hardware at the transceiver and perfect CSI conditions corresponds to the typical MMSE-SIC outage probability for the ith SIC stage (when 1 ≤ i < m), given by while for the mth SIC stage is expressed as which coincides with the outage probability of the conventional MRC, as it should be.
V. ASYMPTOTIC ANALYSIS Although the previous formulae are presented in closed formulations, it is rather difficult to reveal useful insights, straightforwardly. Therefore, in this section, outage probability is analyzed in the asymptotically high SINDR regime. Thus, more amenable expressions are manifested, while important outcomes regarding the influence of imperfect CSI and hardware impairments are obtained.
A. Ordered ZF-SIC 1) General Case: The following proposition presents a sharp outage floor for the general scenario of erroneous CSI under hardware impairments. Proposition 2: When p N0 → ∞, outage performance reaches to a floor, given by Proof: The proof is provided in Appendix E.
2) Imperfect CSI without hardware impairments: The following corollary describes this simplified scenario. Corollary 2: Asymptotic outage floor in the case of imperfect CSI but with ideal transceiver equipment is expressed as .

(31)
Proof: In the absence of hardware impairments, it holds that P ii (pγ th y)f Yi (y)dy. Evaluating the latter integral with the aid of (E.1) yields (31).
3) Perfect CSI with hardware impairments: In this case, the following corollary describes the corresponding asymptotic outage performance.
Corollary 3: Asymptotic outage performance is derived as .

B. MMSE-SIC with Fixed Ordering 1) General Case:
The following proposition presents an outage floor for the general scenario of erroneous CSI under hardware impairments.
Proposition 3: When p N0 → ∞, outage probability of the ith SIC stage reaches to a floor, which is given by (25) and (26), when 1 ≤ i < m and i = m, respectively, by neglecting the N 0 /p term.
The special cases of channel estimation error without hardware impairments or vice versa are obtained by setting κ T = κ R = 0 or ω = 0, respectively.
Most importantly, the system scenario with ideal (nonimpaired) hardware at the receiver provides full diversity order (i.e., n − m + i), regardless of the presence of imperfect CSI or the amount of hardware impairments at the transmitter. The following proposition explicitly describes this effect.
Proposition 4: Asymptotic outage probability of the ith SIC stage in the presence of imperfect CSI and when hardware impairments occur only at the transmitter reads as and for the mth SIC stage as Proof: The proof is provided in Appendix F. Notice that when {ω, κ T } = 0, (33) and (34) reflect the corresponding asymptotic outage expressions for the ideal MMSE-SIC receivers.
Collecting all the aforementioned asymptotic results, a number of conclusions can be drawn and, hence, the following remarks are outlined.
Remark 1: When hardware impairments and/or imperfect CSI are present, outage performance reaches to an upper bound (i.e., outage floor), regardless of the adopted equalization technique (ZF or MMSE). This is explicitly indicated in (30), (31) and (32) for ZF-SIC and in (25) and (26) for MMSE-SIC. Therefore, there is no feasible diversity order in this case.
Remark 2: Diversity order manifests itself, only in the case of MMSE-SIC and when there is a non-impaired receiver, regardless of the presence of hardware impairments at the transmitter and/or imperfect CSI at the receiver. This is indicated in (33) and (34), where both expressions tend to zero as p/N 0 → ∞ (by noticing the existence of the N 0 /p term within these expressions). Particularly, the diversity order in this case is n − i + 1 with respect to the ith decoding layer or n − m + i with respect to the ith SIC stage.
It can be easily seen that the latter remark indicates no difference in the diversity order of the considered MMSE-SIC and the classical MMSE-SIC of an ideal communication setup (see, e.g., [32]). Apparently, performance difference between these two scenarios appears to the underlying coding (array) gains. Observe that ZF-SIC does not achieve diversity order, even when hardware impairments occur only at the transmitter. This effect occurs due to the fact that ZF, in principle, operates by fully eliminating interference but enhancing the noise at the same time. When noise power is proportional to the transmission power, then it unavoidably reflects to the aforementioned outage floor. Such observations could be quite useful for system designers of various MIMO practical applications. As an indicative example, it is preferable to enable higher quality hardware gear for the antennas of the receiver rather than the transmitter. When such a condition occurs, the performance difference of MMSE-SIC over ZF-SIC is emphatically increased for larger SINDR regions. Yet, in order to achieve this performance gain, the variances of channel estimation error and hardware impairments at the transceiver are required, i.e., see the linear filter in (11).

VI. ERROR PROPAGATION EFFECT
One of the most important degradation factors of SICbased reception is the well-known error propagation effect. To date, it has been studied mainly numerically (e.g., see [3] and references therein) and semi-analytically [11] in terms of integral or bound expressions. The limited scenario of m = 2 was analytically studied in [12], but the derived expressions were in terms of infinite series representations. In this section, error propagation is analyzed with regards to the average symbol error probability (ASEP). A formula including numerical verifications is presented for the general case, while closed-form expressions are obtained for some special cases of interest.
ASEP of the ith decoding layer, namely ASEP i , explicitly reads as where ǫ i denotes an error event at the ith decoding layer, ǫ c i is the complement of ǫ i , while M represents the number of modulation states. Also, the second equality of (35) arises by assuming that an earlier error (with probability one) results in a uniform distribution over the constellation for a subsequent symbol decision (equal-power constellation). Hence, ASEP describing the overall behavior of the system, namely ASEP, is given by where P si Pr ǫ i | m l=i+1 (1 − ǫ l ) is the conditional ASEP at the ith decoding layer given that there are no errors in prior layers. Thereby, finding P si represents a key issue to prescribe the total ASEP. It holds that [33] where Z = 1/κ 2 T for ZF-SIC, while Z = 1/(κ 2 R (ω + 1) + ω) for MMSE-SIC. Note that Z → +∞, in ideal conditions of both schemes. Also, A and B are specific constants that define the modulation type [34].
Unfortunately, there is no straightforward closed-form solution for P si for the general case of ZF-SIC and MMSE-SIC, which is based on (23), (25) and (26), to our knowledge. Thus, P si and ASEP can be resolved only via numerical methods. Still, the involvement of a single numerical integration is much more efficient than classical simulation methods (e.g., Monte-Carlo). In the following, some certain scenarios of special interest admit a closed formulation of P si , which in turn provide a corresponding solution to ASEP.

A. Ordered ZF-SIC
Proposition 5: The closed-form expression for P si in the presence of channel estimation errors, an impaired receiver and an ideal transmitter is derived as .
Notice that although (38) is involved with a special function (i.e., Tricomi confluent hypergeometric function), it is in a form of finite sum series, whereas is included as standard builtin function in several popular mathematical software packages. Hence, this expression can be easily and efficiently calculated. 6

B. MMSE-SIC with Fixed Ordering
Proposition 6: P si , for the ith SIC stage (1 ≤ i < m), in the presence of perfect CSI, a non-impaired transmitter, and an impaired receiver is expressed as The asymptotic ASEP expressions could be easily extracted, by following the same methodology as in the previous section. Yet, they have omitted herein since they present very similar insights as the previously derived asymptotic outage probabilities.
Proof: By invoking (25) in (37) For i = m, in the last SIC stage, the expression of (26) does not admit a closed formulation of ASEP. However, it can be numerically calculated quite easily by using (26) in (37) over the valid integration range {0,

VII. NUMERICAL RESULTS
In this section, analytical results are presented and crosscompared with Monte-Carlo simulations. There is a good match between all the analytical and the respective simulation results and, hence, the accuracy of the proposed approach is verified. Note that in Figs. 1 and 2, for ease of tractability and without loss of generality, we assume symmetric levels of impairments at the transceiver, i.e., an equal hardware quality at the transmitter and receiver. To this end, let κ T = κ R κ.
In Fig. 1, the outage performance for the 1st stage of the ordered ZF-and unordered MMSE-SIC is presented for various system settings/conditions. There is an emphatic performance difference between the two schemes in all the considered cases, despite the fact that no optimal ordering is used in MMSE-SIC. This observation verifies the superiority of MMSE against ZF detectors in non-ideal communication setups. In addition, it is obvious that CSI imperfection impacts the performance of ZF-SIC in greater scale than hardware impairments. When this imperfection is more relaxed, the performance gap between the two extreme hardware impairment scenarios starts to grow. This occurs because ZF, fundamentally, relies on channel estimation accuracy to achieve performance gains, counteracting the unavoidable noise enhancement. Thereby, CSI imperfection dramatically affects its performance in comparison to the (noise-oriented) hardware imperfection. Interestingly, this does not comply with MMSE-SIC, whereas quite the opposite condition holds. This is consistent with Remark 2. Also, the traditional MMSE-SIC scheme (taking into consideration only the channel gains and N 0 ) is included for performance comparison reasons. The performance gain of the presented MMSE-SIC over its traditional counterpart is straightforward. Figure 2 depicts the ordered and unordered outage performance of ZF-SIC in ideal and non-ideal communication setups. Obviously, diversity order is manifested only in the former case, while an outage floor is presented in the latter case. This is consistent with Remark 1. It is also noteworthy that the diversity order remains unaffected from the ordering strategy, in accordance to [36]. Moreover, the superiority of the ordered 1st SIC stage against the corresponding unordered stage can be clearly seen. This is the price of performing optimal detection ordering. Furthermore, an important observation from the non-ideal scenario is the fact that the 2nd stage has worse performance as compared to the 1st stage of the ordered ZF-SIC in the entire SNR region. This should not be confusing since the 2nd stage of the ordered SIC has always the worst SNR, whereas this is not the case for the unordered SIC (on average). It seems that less interference (at the 2nd stage) is not enough to counteract the presence of channel imperfection severity and impaired hardware and, hence, to outperform 1st stage. This is in contrast to the traditional (ideal) SIC receivers, where the 1st SIC stage influences more drastically the overall system performance, representing a lower outage performance bound [12], [9]. Figure 3 highlights the important outcome of Remark 2 in non-ideal communication systems. Specifically, it can be seen that when hardware impairments occur only at the transmitter side, MMSE-SIC maintains its diversity order, while ZF-SIC introduces an outage floor, confirming the previous analysis. Also, in dense multi-stream transmissions (i.e., when m = 6), outage performance of ZF-SIC is rather inefficient in comparison to MMSE-SIC.
ASEP of the 1st MMSE-SIC stage is presented in Fig.  4 for various settings, using (37). Again, it is verified that providing a higher-cost/higher-quality hardware gear at the receiver side is a much more fruitful option. Finally, Fig. 5 presents the overall ASEP using (36), for the two considered SIC schemes. All the results for the ZF-SIC are obtained using (38). In addition, the corresponding results of MMSE-SIC for the scenarios with imperfect and perfect CSI are obtained via numerical integration (as in Fig. 4) and using (39), respec-  tively. Considering all the above, both the outage and error rate numerical results confirm the theoretical framework, while the following important outcomes are summarized: a) In the case of ZF-SIC, hardware impairments at the transmitter are as crucial (proportionally) as the impairments at the receiver; b) in ZF-SIC schemes, CSI imperfection influences more the performance than hardware impairments; c) MMSE-SIC appropriately counterbalance the impact of CSI imperfection and the amount of impaired hardware; d) when κ R = 0, MMSE-SIC maintains diversity order and, thus, there is an emphatic performance gain over ZF-SIC, especially in medium-to-high SNR regions.

VIII. CONCLUSIONS
Successive decoding of multiple individual streams was thoroughly investigated under practical communication scenarios. Particularly, ZF-SIC detection/decoding with symbol ordering and MMSE-SIC with fixed ordering were studied for hardware-impaired transceivers and when CSI is imperfectly provided at the receiver side. The analysis included i.i.d. Rayleigh multipath fading channels. New analytical and quite simple (in terms of computational complexity) expressions regarding the outage probability for each SIC stage were obtained. In addition, a general formula indicating the error rate performance with regards to the error propagation effect is provided. Moreover, it was indicated that MMSE-SIC outperforms ZF-SIC in non-ideal communication systems in spite of utilizing no optimal ordering. In addition, an unavoidable performance floor is introduced in the general scenario for both schemes, while diversity order is maintained in MMSE-SIC only when an ideal hardware equipment is enabled at the receiver.

APPENDIX
A. Derivation of (11), (12) and (16) From (7), we have that Since only the first term of (A.1) depends on g (j) , the optimal solution that minimizes MSE is g (j) = pC −1 h j and, hence, (11) is obtained.
At the receiver, (g (j) ) H y is performed, yielding where β j (g (j) ) H h j and η j l =j (g (j) ) H h l s l + (g (j) ) H w ′ . Then, the variance of η j is computed as Thus, the SINDR of the jth stream is given by , 0 < β j < 1, and thus we arrive at (12).
At the MMSE-SIC receiver, the corresponding SINDR expression is presented in (15) for the ith SIC stage, when i < m (i.e., except the final SIC stage). At the last SIC stage, when i = m, there is no residual interference caused by other streams. Following the same methodology as in (A.2), we have that while the corresponding SINDR stems in (16).

C. Derivation of (24)
Referring back to (5) and neglecting the ∆H term, (6) becomes where SNDR stands for the signal-to-noise-plus-distortion ratio. Hence, following similar lines of reasoning as for the derivation of (18), the (unconditional) CDF of SNDR yields as . (C.2) Using (B.1) in (C.2), we have that .
Fortunately, based on the pioneer work in [37,Eq. (11)], and some further elaborations on this result (e.g., see [38,Eq. (6)] and [39, Eq. (61)]), CDF of Φ i yields as F Φi (x) = 1 − exp − (κ 2 R m + N 0 /p)x (κ 2 T (ω + 1) + ω + 1) Nonetheless, (D.2) is quite cumbersome and it is not amenable for further analysis. Due to this, we slightly modify it in order to derive a more convenient formation. Noticing that n ≥ m, using (D.1), and the fact that where Y n l=1 |h l | 2 . This is due to the fact that h m h H m (which produces a rank-one column matrix) and h H m h m share the same single nonzero eigenvalue, defined as λ. Note that 2n [40]. Thereby, CDF of SINDR m is expressed as Hence, we can reach (26), after some straightforward manipulations. Note that 1 − (κ 2 T (ω + 1) + ω)γ th > 0 should hold in (D.4) to be a valid CDF.
At the last SIC stage, (26) can alternatively be expressed as By retaining only the first summation term (i.e., k = n), the final expressions can be extracted.