Choosing the Right Citation Management Tool: Endnote, Mendeley, Refworks, or Zotero

There are now many bibliographic management packages available and many factors to consider when choosing the product that best meets the needs of the individual user or institution. Popular tools include RefWorks, EndNote, Zotero, and Mendeley.

INTRODUCTION
Citation management has not always been as easy as it is today. Years ago, references were manually organized on index cards, an awkward and time-consuming process. Citation management software was introduced in the 1980s and used primarily to organize references, search databases for articles on a particular topic, and generate bibliographies [1]. Over the years, users' needs have changed, technology has advanced, and many new features have been added, including options for social networking and portable document format file (PDF) management.
There are now many bibliographic management packages available and many factors to consider when choosing the product that best meets the needs of the individual user or institution. Popular tools include RefWorks, End-Note, Zotero, Mendeley, and F1000 Workspace. This review will cover the first four; F1000 Workspace was reviewed in the Journal of the Medical Library Association (JMLA) in 2017 [2].
First released in 1988 [3], End-Note is a commercial product that is primarily marketed via sales of its desktop application (currently version X8). A basic online version is free, but it has limited features and functionality. RefWorks, first released in 2001 [4], is an entirely web-based application marketed to libraries as an institution-wide tool, though a vendor representative indicated that individual accounts used to be available and will be offered again [5]. The product is currently transitioning to a new interface, referred to by the vendor as "new RefWorks." Zotero's free, open source citation manager was initially introduced in 2006 as an extension for the Firefox web browser. It is now available as a standalone application [6]. First released in 2008 [7], Mendeley is a free cloud-based citation manager with desktop and online versions. It also serves as an academic research network, offering a variety of social networking features.
All four products share a core set of features that allow users to import, organize, and manage citations and associated full text. Users can import references from a variety of databases, create in-text citations and bibliographies, and import bibliographic information from web pages. All offer an extensive list of citation styles and the ability to edit existing styles and create new ones.
The remainder of this review focuses on how these products differ with respect to the most commonly used features of citation managers and the advantages and disadvantages of each product. Table 1 summarizes key differences between the products. This review is based primarily on current desktop versions (if applicable) of the products as of February 2018, though online versions are discussed as needed to provide a complete picture of a tool's functionality. For Mendeley, this review covers the free version only. For RefWorks, this review covers the new RefWorks only; it does not address the older version, known as Legacy RefWorks.

SEARCHES FOR AND IMPORTING OF REFERENCES
All four tools allow users to import files of references from databases or other citation management tools.
Users can search within databases, mark references to save or export, and select from a variety of options to add references to their preferred citation manager tools. Choosing a direct export option opens any of these tools that are installed on users' computers, and references can be added with one mouse-click. Each of the products has direct export options for at least one of the following databases: PubMed, Web of Science, Science Direct, EBSCO (CINAHL), and ProQuest (PsycINFO). All four systems allow direct export of records from EB-SCO (CINAHL), while EndNote is the only tool that has a direct export option for PubMed.

jmla.mlanet.org
Users can also use the browser add-ons to automatically import references into their reference collections. The add-ons for Mendeley, RefWorks, and Zotero allow users to import references to their reference collections from multiple databases. Depending on the database, users can select individual references or batches, and the references and associated PDFs are imported. Mendeley and Zotero users can use the browser add-ons to import references from PubMed, Web of Science, and Science Direct. Using Zotero, the reviewers were able to import references from ProQuest (PsycINFO), but the Mendeley browser add-on was not able to recognize the bibliographic metadata in ProQuest (PsycINFO) references. Errors were experienced with both Mendeley and Zotero when we imported references from EBSCO (CINAHL).
EndNote's Capture Reference bookmarklet has more limited functionality than the browser addons for the other three products. When displaying a list of PubMed search results, Capture Reference only imported all references on the page; it did not allow us to select specific references to import. Capture Reference did not work at all for us with a list of results from Google Scholar. The only way to import these results was to open each one and then capture it. It also did not directly capture bibliographic information about web pages as easily as the other add-ons did. When we attempted to import information about a web page using Capture Reference, it created an RIS file that we then had to import into EndNote, whereas the other three add-ons added information about web pages directly.
The tools also offer several other ways to add references. Mendeley users can add references by entering a PubMed ID (PMID), digital object (DOI), or ArXivID. Similarly, Zotero users can add references using the international standard book number (ISBN), DOI, or PMID. In the online version of Mendeley, users can search and import references from Mendeley's web catalog, a collection of all the references that have been added to the personal libraries of Mendeley users [8]. EndNote and RefWorks also allow users to search databases and library catalogs from within the application and import selected search results. EndNote offers an extensive list of free and commercial databases for searching. As of this writing, the new RefWorks only offers PubMed and the Library of Congress as search options, and, when tested, neither search option was functional. According to the RefWorks lead product manager, institutional account administrators can allow users to search any database that is accessible via the Z39.50 search standard. He also indicated that ProQuest is building application programming interfaces (APIs) to integrate RefWorks with other ProQuest tools such as Summon and Primo, which should increase in-app search options [5].

CREATION OF BIBLIOGRAPHIES
All four applications allow users to create standalone bibliographies in virtually any word processor, including Google Docs. With End-Note, users can create a standalone bibliography by selecting citations and an output style, and copying and pasting into a word processor document. EndNote also allows users to create a subject bibliography that is based on one or more keywords in users' citations. Both Mendeley and Zotero allow users to drag references from the desktop client into a word processor, where they will be formatted according to the style that users have selected, the quickest and most user-friendly method of bibliography creation. RefWorks includes a feature that allows users to generate a bibliography from a batch of references in a folder, but that feature did not work when we tested it, leaving no way to generate standalone bibliographies from citations.
More commonly, users create bibliographies from in-text citations in a manuscript. All four tools offer Microsoft Word plug-ins to support this functionality. Table 1 provides details about which tools work with other word processors. In EndNote, the bibliography is automatically generated as the citations are inserted into the document. In Mendeley, RefWorks, and Zotero, inserting a citation and creating a bibliography are separate steps, and at least one citation must be added to the document in order to create a bibliography. All four products made occasional small errors in citations, especially when we cited web pages, but Mendeley performed especially poorly, omitting key information from web page citations, such as date accessed.

MANAGEMENT AND ANNOTATION OF PORTABLE DOCUMENT FORMAT FILES
Each tool offers different options for adding PDF documents. All four systems allow users to add PDF documents by dragging and dropping them into their reference collections and by attaching them to existing citations. EndNote and Mendeley users can drag and drop PDFs both individually and in folders. RefWorks users can only add PDFs one at a time, while Zotero users can add multiple PDFs at once. Mendeley users can also add PDFs by putting them in a designated folder called a Watch Folder. Mendeley monitors the contents of these folders and automatically adds any PDFs to reference collections.
All four products can generate metadata from PDFs to create a citation record, but they use somewhat different methods to do so. When we tested articles from three different journals, all four products extracted metadata inconsistently and occasionally inaccurately. For example, one product extracted metadata completely for a given article, while another failed to extract key information (e.g. author name, page numbers) from the same PDF, and a third failed to import any metadata from the PDF. All products exhibited these failures, though RefWorks appeared to be the least accurate, with at least one significant error with each of the three PDFs that we tested.
All of the products, other than Zotero, support PDF annotation in the application. Zotero users can open PDFs in the application of their choice, annotate them, and save them back to the Zotero database. An add-on called Zotfile [9] allows users to extract annotations and perform other PDF management tasks.

INTEGRATION WITH LIBRARY COLLECTIONS
EndNote and Zotero can use an openURL link resolver to help users retrieve full text from a library's electronic collections. Users can specify the baseURL of their libraries' link resolver in the product settings, and the products will use metadata from a citation in their libraries to attempt to locate full text for that item. In Zotero, this feature is called Library Lookup. Users click on a reference in their collections, and if full text is found, the PDF file can be easily dragged and dropped into their reference collections. EndNote users can access full-text through their institutions by using the Find Full Text feature. Mendeley used to allow integration with a library's link resolver but no longer offers this feature [8]. For RefWorks, institutional administrators can configure a link resolver for all users at that institution.

COLLABORATION AND SOCIAL NETWORKING
According to RefWorks documentation, RefWorks users can only share collections with users at their own institutions [10]. The Ref-Works senior product manager indicated, however, that as of fall 2017, RefWorks users can share folders with other RefWorks users across institutions [5]. EndNote X7 and X8 users can share with each other in groups of up to 100 members [11]. Mendeley and Zotero users can create both public and private groups [12,13], though Mendeley users with a free account can create and own only one private group, and private groups created by free accounts are limited to three members [14]. Mendeley offers additional social networking features in the online version that the other products do not provide. Mendeley users can search for and follow other researchers with similar interests and receive updates on actions and events of researchers they are following via the Mendeley Newsfeed [12].

OFFLINE AVAILABILITY
EndNote, Mendeley, and Zotero collections and documents are stored locally and, therefore, available offline. RefWorks is a purely cloud-based system, so access to the application itself is not available offline. Users can, however, link a DropBox account to RefWorks to provide offline access to full-text documents in RefWorks [10].

UNIQUE FEATURES
Of the four products, EndNote is the only one that offers a journal matching feature, known as Manuscript Matcher, to help users find the right journal for their manuscripts. Users of the online version can provide their article titles, abstracts, and references, and End-Note will provide a list of journal recommendations based on its analysis of Web of Science citation data [15]. RefWorks is the only product to offer a plug-in for Google Docs, an especially useful feature at universities where Google tools are used heavily by students. It is also the only fully cloud-based product. While both Mendeley and Zotero are free, Zotero is the only open-source product among the four. Its source code is hosted on GitHub and freely available under an AGPLv3 license [16].

CONCLUSION
All four of the tools reviewed here are usable for standard reference manager functions, and each has strengths and weaknesses. For example, in our testing, Zotero's browser add-on was the easiest to use and captured data more accujmla.mlanet.org rately than the other add-ons did. EndNote offered the most choices for searching databases within the tool, and Zotero generated the most accurate bibliographies. Each also offers unique features that may be especially valuable to certain populations (e.g., RefWorks' integration with Google Docs, Mendeley's social networking functions). Often, though, the best choice for a given purpose may be determined by factors other than the functionality of the applications themselves. These factors include cost, support provided by institutions, research needs, familiarity with a product from previous experience, and accessibility for the research team members. For example, if users are working on a systematic review with authors at several institutions, they will need to choose a tool that is accessible to everyone on the team. Since users are not limited to the citation managers supported by their institutions, information professionals need to be familiar with all popular choices in order to guide and support their users effectively.


extfieldsofstudy:
Computer Science
Mathematics

text:
Coding Schemes for Discrete Memoryless Multicast Networks with Rate-limited Feedback

Coding schemes for discrete memoryless multicast networks with feedback from receivers and relays to the transmitter are proposed. These coding schemes are based on block Markov coding, {joint backward decoding} and hybrid relaying strategies. In each block, the receivers and relays compress their channel outputs and send the compression indices through the feedback link to the transmitter. In the next block, after obtaining compression indices, the transmitter sends them together with the source message. Each receiver uses backward decoding to jointly decode the source message and all compression indices. Our coding schemes generalize Gabbai\&Bross's results for the single relay channel with partial feedback, where they used the restricted decoding and deterministic partitioning. It is shown that for the single relay channel with relay-transmitter feedback, our coding schemes improve the noisy network coding scheme by Lim, Y-H Kim and El Gamal, the distributed decode-forward coding scheme by Lim, Kim and Y-H Kim and all known lower bounds on the achievable rate in the absence of feedback. Furthermore, we modified our schemes and propose a new coding scheme for discrete memoryless multicast networks without feedback, which can potentially improve the noisy network coding and distributed decode-forward coding schemes.

I. INTRODUCTION
The relay channel, first introduced by van de Meulen [1], describes a 3-node communication channel where the transmitter sends message to the receiver with the assistance of relay. In their seminal paper, Cover and El Gamal proposed two basic coding strategies: compress-forward and decode-forward, for the discrete memoryless relay channel. Both strategies are based on block Markov coding. In the compress-forward strategy, relay compresses its outputs and sends compression index to the receiver. In the decode-forward strategy, the relay first decodes the full or part of source message and then sends the decoded message to the receiver. Both strategies have been generalized to the discrete memoryless multiple-relay channel [2]. The compress-forward strategy was later extended to a more general network-discrete memoryless network [3], [4]. Recently, a distributed decode-forward coding scheme was proposed by Lim, Kim, Y-H Kim that generalizes the decode-forward strategy to the general multicast network [5] and broadcast relay network [6].
When introducing (perfect) feedback from the receiver to the relay, the relay channel was shown to be a physically degraded [7], and therefore decode-forward achieves the channel capacity. For the case with feedback from the receiver or relay to the transmitter, the capacity is general unknown. In [8] Gabbai and Bross studied this problem and proposed inner bounds by using the restricted decoding and deterministic partitioning, introduced by Willems and Van der Meulen in [9].
In this paper, we consider the general discrete memoryless multicast network with rate-limited feedback. This network consists of N ≥ 3 nodes where the transmitter sends a message to different receivers with the assistance of multiple relays and in presence of feedback from the receivers and relays to the transmitter. For this setup, we propose two coding schemes based on block Markov coding, joint backward decoding and hybrid relaying strategy. In our first scheme, relays and receivers use compress-forward strategy to create their compression indices and then send them both into the forward communication and feedback channels. The transmitter, after obtaining the compression indices through feedback, sends them together with the source message. Receivers jointly decode the source message and all compression indices. Our second scheme is similar to the first scheme, except that the relay not only uses compress-forward strategy as in the first scheme, but also use partial decodes-forward strategy [7] to decode the source message. It is shown that our coding schemes generalize Gabbai&Bross's results [8] for the relay channel with relay-transmitter feedback. For some channels, such as the general Gaussian relay channel and Z relay channel, our coding schemes improve over the noisy network coding scheme by Lim, Y-H Kim and El Gamal, the distributed decode-forward coding scheme by Lim, Kim and Y-H Kim and all known lower bounds on the achievable rate in the absence of feedback.
Motivated by our feedback scheme, we propose a new coding scheme for discrete memoryless multicast networks without feedback, which can still potentially improve the noisy network coding and distributed decode-forward coding.

II. SYSTEM MODEL
Consider N -node discrete memoryless (DM) multicast networks with feedback from the receivers and relays to the transmitter, see This setup is characterized by 2N finite alphabets X 1 , . . . , X N , Y 1 , . . . , Y N , a channel law P Y1···YN |X1,...,XN and nonnegative feedback rates R Fb,k , for k ∈ [2 : N ]. Specifically, at discrete-time i, node j ∈ [1 : N ] sends input x j,i ∈ X j and then observes output y k,i ∈ Y j . After observing Y k,i , for k ∈ [2 : N ], node k sends a feedback signal F k,i ∈ F k,i to the transmitter, where F k,i denotes the finite alphabet of F k,i and is a design parameter of a scheme. The feedback link between the transmitter and node k is assumed to be instantaneous and noiseless but rate-limited to R Fb,k bits on average. Thus, if the transmission takes place over a total blocklength n, then Fig. 1. N -node discrete memoryless multicast network with partial feedback In the transmission, the transmitter communicates a message M ∈ [1 : 2 nR ] to the set of receivers D with the assistance of the set of relays R. A (2 nR , n) code for this channel consists of • a set of relay and receiver encoders that maps • a set of feedback-encoder that is to produce feedback symbols F k,i (Y k,1 , . . . , Y k,i ), for each k ∈ [2 : N ] and i ∈ [1 : n], • a set of decoders that estimatesM (d) based on Y n d , for d ∈ D. Assume M is uniformly distributed over the message set. The average probability of error is defined as P

III. MAIN RESULTS
In this section, we present our main results as the following theorems. The detailed proof are given in Section V and Section VI.
Theorem 1: For DM multicast networks with feedback from receivers and relays to the transmitter, the rate R is achievable if for all T ⊆ [2 : N ], d ∈ D, and for some pmf Proof: See Section V-A. Theorem 2: For DM multicast networks with feedback from receivers and relays to the transmitter, the rate R is achievable if for all T ⊆ [2 : N ], d ∈ D, and for some pmf such that Proof: See Section V-B Remark 1: By setting U k = const., the achievable rate in Theorem 2 specializes to the rate in Theorem 1. (Setting U k = const. means that all relay nodes only perform compress-forward strategy without partially decode-forwarding the source message.) Based on coding schemes for Theorem 1 and 2, we propose another coding scheme for DM multicast networks without feedback. The new achievable rate is shown as below.
Theorem 3: For DM multicast networks without feedback, the rate is achievable if R satisfies (8) for all T ⊆ [2 : N ], d ∈ D, and for some pmf such that IV. EXAMPLES

A. The classic relay channel with relay-transmitter feedback
Consider the classical relay channel with perfect feedback from the relay to the transmitter, see Figure 2.

Fig. 2. Discrete memoryless relay channel with relay-transmitter feedback
For this channel, the achievable rate in Theorem 1 specializes to for some pmf P X1X2 PŶ 2 |X2Y2 . The achievable rate in Theorem 2 specializes to for some pmf P X1X2U2 PŶ 2|X2 U2Y2 . In [8] Gabbai and Bross considered this channel and proposed coding schemes based on restricted decoding and deterministic partitioning. It's easy to check that the rates (11) and (12) recover Gabbai&Bross's rates of Theorem 2 and Theorem 3 in [8], respectively.
By using noisy network coding [3], we obtain the achievable rate for some pmf P X1 P X2 PŶ 2|X2 Y2 , which coincide with the compress-forward lower bound (Theorem 6 in [7]). By using distributed decode-forward coding scheme [5], [6], we obtain the achievable rate for some P X1X2U2 , which coincide with the partial decode-forward lower bound (Theorem 7 in [7]).
In [8] Gabbai and Bross showed that for the general Gaussian relay channel and Z relay channel, the rate (12) is larger than the known lower bounds on the achievable rate in the absence of feedback, including the compress-forward lower bound as in (13), and the partial decode-forward lower bound as in (14). In view of this fact, we have the following Corollary: Corollary 1: For the DM single-relay channel with relay-transmitter feedback, our coding schemes recover Gabbai&Bross's results, strictly improve over the noisy network coding [3], the distributed decode-forward coding in [5] and all known lower bounds on the achievable rate in the absence of feedback.

B. Enhanced relay channel
Consider an enhanced relay channel where the transmitter accesses its channel output Y 1 , see Figure 3.

Fig. 3. Discrete memoryless enhanced relay channel
For this channel, the achievable rate in Theorem 3 specializes to . When using noisy network coding [3] for this channel, we obtain the following achievable rate for some pmf P X1 P X2 PŶ 2|X2 Y2 PŶ 1|X1 Y1 . It's easy to find that the optimal choice of Y 1 isŶ 1 = const., resulting the same achievable rate as (13). When using distributed decode-forward coding [5] for this channel, the achievable rate is same as (14). Note that by setting V 2 = X 2 ,Ŷ 2 = const., (15) reduces to the rate (14), which means our coding scheme could potentially improve distributed decode-forward coding.
One interesting finding is that for this enhanced relay channel, both noisy network coding and distributed decode-forward coding fail to make use of Y 1 . While in our scheme for Theorem 3, instead of compressing or ignoring Y 1 , the transmitter decodes the compression messages sent by receivers and relays based on Y 1 , which could potentially increase the rate achieved by noisy network coding and distributed decode-forward coding.
For each l b−1 , randomly and independently generate 2 nR sequences Encoding and decoding are explained with the help of Table I. . To ensure that source node perfectly knows l b−1 , we havê 3) Relays and receivers encoding: Relays and receivers both perform compress-forward strategy. In each block b ∈ [1 : B], node k ∈ [2 : N ] compresses y n k,b by finding a unique index l k,b s.t.
Then, it sends l k,b through the feedback link at rateR k ≤ R Fb,k and in block b + 1 sends x n k,b+1 (l k,b ). By the covering lemma, this is successful with high probability if . By the independence of the codebooks, the Markov lemma, packing lemma and the induction on backward decoding, this step is successful with high probability if and Combine (17-20), and by using Fourier elimination to eliminateR 2 , . . . ,R N , we obtain Theorem 1.

B. Scheme 1B
Note that in Scheme 1A above, relays and receivers only perform compress-forward strategy. In this subsection we present a scheme where relays perform mixed compress-forward and partial decode-forward strategy. 1 Since each Receiver d ∈ D make its own estimation of m b and l b−1 , the exact notation should be (m b−1 ). For simplicity of notation, we omit the superscript (d).
, l r,B ) Yrŷ r,1 (l r,1 |1, 1)ŷ r,2 (l r,2 |m 1 , l r,1 ) . . .ŷ r,B (l r,B |m ′ B−1 , l r,B−1 )ŷ r,B+1 (1|m ′ B , l r,B ) For each l d,b−1 , randomly and independently generate 2 nR ′ sequences . Encoding and decoding are explained with the help of Table II. 2) Source encoding: In each block b ∈ [1 : B + 1], assume that the transmitter already knows l b−1 through the feedback links. It sends . To ensure that source node perfectly knows l b−1 , we havê 3) Relays encoding: Relay nodes perform hybrid compress-forward and decode-forward strategy. For each block b ∈ [1 : then it compresses y n r,b by finding a unique index l r,b s.t.
Then, it sends l r,b through the feedback link at rateR r ≤ R Fb,r and in block b + 1 sends x n r,b+1 (m ′ b , l r,b ). By the covering and packing lemma, this is successful with high probability if for r ∈ R.

4) Receivers encoding:
Receiver d ∈ D compresses y n d,b by finding a unique index l d,b s.t.
Then, it sends l d,b through the feedback link at rateR d ≤ R Fb,d and in block b + 1 sends x n d,b+1 (l d,b ). By the covering lemma, this is successful with high probability if 2 Since each Relay r ∈ R makes its own estimation of m ′ b , the exact notation should bem b . For simplicity of notation, we omit the superscript (r).

5) Decoding:
Receiver d ∈ D performs backward decoding. For each block b ∈ [B +1, . . . , 1], it looks for . By the independence of the codebooks, the Markov lemma, packing lemma and the induction on backward decoding, the decoding is successful with high probability if and a). if T ∩ R = ∅ R +R(T ) For convenience, we ignore the constraints (25) Combine (21-24) and (27), and by using Fourier elimination to eliminate R ′ , R ′′ ,R 2 , . . . ,R N , we obtain Theorem 2.
VI. DISCRETE MEMORYLESS MULTICAST NETWORK In Section V we proposed two block Markov coding schemes for DM multicast networks in the presence of instantaneous, rate-limited and noisy-free feedback. Recall the noisy network coding scheme [3], [4] for DM multicast networks without feedback, where each node (including the transmitter) compresses its observation and sends the new compression index in next block. Comparing our coding scheme with noisy network coding, we observe that both schemes involve block Markov coding, compressing channel outputs and sending compression messages. However, our schemes allow hybrid relaying strategies at relays nodes and in each block, instead of creating new compression index, the transmitter forwards all compression indices sent by receivers and relays from previous block. In our scheme, different nodes operate differently according to the features of the network, which leads to a larger achievable rate than noisy network coding, as shown in examples in Section IV.
Motivated by Scheme 1A and 1B, we propose another scheme for N -node DM multicast networks without feedback. The main idea is as following: in each block b, node k ∈ [2 : N ] creates compression index l k,b−1 and sends (l k,b−1 , l k,b−2 ). The transmitter, after observing Y n 1,b , first decodes compression indices l b−1 , which is in essence a coding problem on a multiple access channel P Y1|X2,...,XN with side information X 1 . Then in block b + 1, the transmitter sends compression messages l b−1 with source message m b+1 . In this section we extend Scheme 1B to DM multicast networks. Similar extension can be applied to Scheme 1A.

Relay r
Rx d Tx , randomly and independently generate 2 nR ′′ sequences x n  (28) since the transmitter knows the source messages it sent.


text:
A Dynamic Jamming Game for Real-Time Status Updates

We study timely status updates of a real-time system in an adversarial setting. The system samples a physical process, and sends the samples from the source (e.g., a sensor) to the destination (e.g, a control center) through a channel. For real-time monitoring/control tasks, it is crucial for the system to update the status of the physical process"timely". We measure the timeliness of status updates by the time elapsed since the latest update at the destination was generated at the source, and define the time elapsed as age of information, or age in short. To sabotage the system, an attacker aims to maximize the age by jamming the channel and hence causing delay in status updates. The system aims to minimize the age by judiciously choosing when to sample and send the updates. We model the ongoing repeated interaction between the attacker and the system as a dynamic game. In each stage game, the attacker chooses the jamming time according to the jamming time distribution, and the system responds by delaying the sampling according to the sampling policy. We prove that there exists a unique stationary equilibrium in the game, and provide a complete analytical characterization of the equilibrium. Our results shed lights on how the attacker sabotages the system and how the system should defend against the attacker.

I. INTRODUCTION
The success of many existing and emerging engineering systems hinges upon the timely status updates of the underlying physical processes. Such systems range from real-time monitoring systems, such as sensor networks monitoring air pollution [1] and Internet-of-Things (IoT) [2], to networked control systems, such as autonomous vehicles [3] and power grids [4]. Timely status updates ensure that the system always has fresh information about the physical process, and that it can monitor and control the process effectively.
The age of information, or age in short, is becoming the prevailing metric to measure the timeliness of status updates [5]- [20]. At any time instant, the age of information is defined as the amount of time elapsed since the most recently received update was generated. A large age indicates that the information about the physical process is outdated, since the most recently received update was generated long time ago. Therefore, the goal is to minimize the age of information. However, minimizing age of information requires careful decisions on when to sample the process and send the samples. Suppose that we have just updated the status. On the one hand, we do not want to update the status immediately, because the Yin Sun was supported in part by ONR grant N00014- 17-1-2417. new update would be almost the same as the old update, and hence carry little fresh information. On the other hand, we do not want to wait for a long time before updating the status, because the previous update would become outdated before the next update. In summary, we need to make sure that both the information sent and that received are fresh, in order to minimize the age of information [8], [16].
While there have been a growing body of works that address the problem of minimizing the age of information in various scenarios [5]- [10], [13], [15], [16], few works have studied this problem in an adversarial setting [17]. Since timely status updates are of particular importance to real-time monitoring and control systems, the attackers of these systems may specifically target their capability of timely updating the status [21], [22]. Therefore, it is crucial to understand how would an attacker sabotage a real-time monitoring/control system, how should the system defend against the attack, and to what extent the attacker can degrade the timeliness of status update. In this paper, we try to answer these questions in a stylized model.
We formulate a dynamic game to model the repeated interaction between a real-time monitoring/control system and its attacker. The system takes samples from a physical process to monitor and sends the samples to a destination through a wireless channel. An attacker jams the channel so that the system cannot send new updates in timely fashion. The attacker aims to find a jamming time distribution to maximize the age of information. After the jamming is over, the system chooses when to sample and send the next update. The system aims to find a sampling policy to minimize the age of information. We summarize the main contributions of this paper as follows.
• We prove that there exists a unique stationary equilibrium in the game, and that the system cannot further reduce the age at the stationary equilibrium by using more complicated equilibrium strategies. • We provide a complete analytical characterization of the equilibrium, which sheds lights on how the system should defend against the attacker. Specifically, the equilibrium sampling policy tries to make the time differences between subsequent update deliveries as equal as possible. • We also provide insights on how the attacker should abotage the system. Specifically, the attacker will choose a jamming time distribution with as high variance as possible.

II. RELATED WORKS
Most works on age of information do not consider an attacker attacking the system [5]- [10], [13], [15], [16]. The most related work is [17], where the authors study age of information in an adversarial setting. There are two major differences between our work and [17]. First, our work studies the interaction of the user and the attacker at the MAC layer, while their work [17] studies the interaction in the physical layer. This difference results in different models of actions and more importantly, different models of interaction (e.g., different payoff functions). Second, our work models the interaction as a dynamic game, while their work [17] uses a static game model. Therefore, we study the ongoing interaction between the user and the attacker, where the user needs to develop a defense policy that specifies a sequence of sampling actions over time. In contrast, their work studies the one-shot interaction, where the user needs to determine a scalar action.

A. System Model
Consider a real-time monitoring system in the presence of an attacker (see Fig. 1 for an illustration). The real-time monitoring system could be a stand-alone system by itself (e.g., sensor networks [1] and Internet-of-Things [2]), or a component of a real-time control system (e.g., autonomous vehicles [3] and power grids [4]). The monitoring system consists of a sensor that collects information about some physical process, and a transmitter that sends information to a receiver. The (monitoring) system aims to timely update the information collected, which is crucial for any real-time monitoring or control system. The system achieves timely update of information by judiciously choosing when to sample the physical process and when to transmit the samples. The attacker aims to sabotage the timely status updates of the system, through jamming the wireless channel between the transmitter and the receiver.
Next, we describe the system model in details. 1 Since the system updates a small amount of information (e.g., a sensor reading, a control signal) at each time, the duration of each transmission is negligible and assumed to be zero [7], [13], [15]. Therefore, although the attacker can monitor the channel and detect a transmission, it is difficult for the attacker to target the transmission (e.g., through colliding with the transmitted packet). Instead, after detecting the transmission of a sample, the attacker can occupy the channel for some amount of time in order to delay the delivery of the next sample. Specifically, to delay the transmission of the n-th sample, the attacker jams the channel for a random duration of A n . This formulation physical process sampler Tx sensor Rx monitor control receiver attacker jamming Fig. 1. Illustration of the system model.
includes the case when the attacker jams the channel for a deterministic amount of time (by making the distribution a Dirac delta function). We make the following assumption. 2 Assumption 1: The attacker's jamming time for each transmission is identically and independently distributed (i.i.d.) with probability density function (pdf) f A .
We impose the following constraints on the maximum and average jamming time: For all n = 1, 2, . . ., it holds almost surely that The maximum jamming time A max is set up to reduce the chance that the system can locate the attacker. The attacker would like to limit the jamming time to be no longer than A max to remain elusive. The constraint on the average jamming time comes from the energy constraint of the attacker. Since the attacker's action is the jamming time distribution f A , we translates the constraints in (1) into constraints on f A as follows The system employs carrier sensing to monitor the channel. Once the channel is idle (i.e., after the attacker stops occupying the channel), the system decides when to sample the physical process and when to transmit the sample. One may wonder why the system does not sample and transmit immediately after the channel is idle? While the system should transmit the sample immediately after sampling (for timely update of information), it may not be optimal to sample immediately after the channel becomes idle. We refer interested readers to [16, Sec. I] for an example illustrating why the system may want to delay the sampling. We write D n as the delay of the n-th sample after the channel is idle. Denoting the sampling time of the n-th sample as S n , we have where we define S 0 = 0.
We use the age of information to evaluate the timely update of information [5], [6]. At any time t, the age of information is the time since the last status update of the physical process t ∆(t) jamming by attacker: t sampling by system: t Fig. 2. Illustration of the system dynamics and evolution of the age of information. Sn is the sampling time of the n-th sample. The transmission of the sample is assumed to be instant. An is the attacker's jamming time for the n-th sample, and Dn is the system's delay of the n-th sample after the channel is idle.
(i.e., since the last sample). We denote the age of information at time t by ∆(t), which can be calculated as The evolution of the age is shown in Fig. 2.

B. Game Formulation
Now we formulate the dynamic game between the system and the attacker. The dynamic game consists of an infinite number of stage games. The n-th stage game takes place between the transmission of the (n − 1)−th and the n-th samples (i.e., between S n−1 and S n ). In each stage game n, the attacker takes action by choosing the jamming time A n , and the system reacts by choosing delay D n for the n-th sample. In other words, each stage game is a Stackelberg game with the attacker taking action first. Since we have restricted to i.i.d. jamming time distribution, the attacker's action is simply choosing the jamming time distribution f A . We allow the attacker to predict how the system reacts to its action, based on which the attacker will seek its optimal jamming time distribution.
The system's sampling policy dictates how to choose the delay D n for each sample. We focuses on causal policies that determine each delay D n based on information available, which is the historical and current jamming time A 1 , . . . , A n and the historical delay D 1 , . . . , D n−1 . We write the set of histories at the beginning of the n-th stage game as H n [0, A max ] n × [0, ∞) n−1 , and the set of all histories as H = ∪ ∞ n=1 H n . Then a causal sampling policy is defined as where Θ ([0, ∞)) is the set of all possible probability distributions with support within [0, ∞). In other words, π(A 1 , . . . , A n , D 1 , . . . , D n−1 ) is the probability distribution of the delay D n . We write the set of all causal policies as Π.
The payoff of interest to both the system and the attacker is the average age of information, defined as which depends on the attacker's jamming time distribution f A and the system's sampling policy π.
The system aims to minimize the average age ∆(f A , π). Its optimization problem to solve is The attacker aims to maximize the average age ∆(f A , π), in anticipation of the system's reaction to its jamming time distribution. Given any jamming time distribution f A , the system reacts by employing a sampling policy that solves (7). We write BR(f A ) as the system's reaction to f A (i.e., a solution to (7)). The function BR(f A ) is also called best response in game theoretic term. Note that under the same jamming time distribution f A , the best response BR(f A ) of the system may not be unique. However, since all best responses result in the same average age, we only need to consider an arbitrary best response.
The attacker's optimization problem can then be written as Note that we use the best response BR(f A ), instead of an arbitrary policy π, in the objective function of the attacker. This reflects the fact that the attacker can anticipate the system's best response when taking actions. In other words, each stage game is a Stackelberg game with the attacker being the leader [23]. Therefore, the attacker has strategic advantage, because it can predict how the system reacts to its action, and choose its optimal action based on this prediction. By giving the attacker strategic advantage, we can evaluate how well the system can defend in the worst case. Now we can define the equilibrium of the dynamic game. Definition 1 (Equilibrium): Aa equilibrium is a duple of two functions (f * A , π * ) such that • the system's sampling policy π * minimizes the average age under the jamming time distribution f * A , namely π * is a solution to (7). • the attacker's jamming time distribution f * A maximizes the average age in anticipation of the system's best response, namely f * A is a solution to (8). The definition of equilibrium ensures that neither the system nor the attacker could deviate from the equilibrium and get more favorable outcomes.

IV. MAIN RESULT
We give a complete characterization of equilibria in this section. Note that both optimization problems of the system and the attacker in (7) and (8) are complicated functional optimization problems. We first argue that we can simplify the problem by restricting our attention to stationary deterministic sampling policies. We then define the corresponding stationary equilibrium, prove that the stationary equilibrium is unique, and provide analytical expressions of the equilibrium jamming time distribution and the equilibrium sampling policy.

A. Restriction to Stationary Deterministic Sampling Policies
We consider stationary deterministic sampling policies, a special class of causal policies defined as follows.
A stationary deterministic policy chooses the delay D n as a fixed, deterministic function of the jamming time A n , independent of the historical jamming time A 1 , . . . , A n−1 and historical delay D 1 , . . . , D n−1 .
Although the system could choose any causal policy π ∈ Π, the following lemma suggests that given any jamming distribution f A , there exists a unique stationary deterministic sampling policy that is optimal among all causal policies.
Lemma 1: Given any jamming time distribution f A , there exists a unique stationary deterministic sampling policy that is optimal among all causal policies.
Proof: For a given jamming distribution f A , the age minimization problem of the system is equivalent to the one in [16]. Therefore, we can use [16,Theorem 4]. The results in Lemma 1 directly come from [16,Theorem 4].
Lemma 1 allows us to restrict to stationary deterministic sampling policies. According to (9), a stationary deterministic policy is completely specified by the deterministic function u. Hence, we will refer to a stationary deterministic policy as u.
Since we restrict to stationary deterministic sampling policies, we focus on stationary equilibrium defined as follows.
Definition 3 (Stationary Equilibrium): A stationary equilibrium is an equilibrium (f * A , π * ) where the sampling policy π * is stationary deterministic.
We also write a stationary equilibrium as (f * A , u * ).

B. Characterization of Equilibria
Now we give a complete characterization of stationary equilibrium.
Theorem 1: There is a unique stationary equilibrium, where the attacker's equilibrium jamming time distribution is where δ(·) is the Dirac delta function, and the system's equilibrium sampling policy is D n = u * (A n ), where the function u * is given by where (·) + max {·, 0}, and β * is a constant given by Proof: See Section V. Our result provides analytical characterization of the equilibrium, and hence shed lights on the equilibrium behavior of the system and the attacker. First, the system's policy can be viewed as a "water-filling" policy with β as the water level. The purpose of water-filling is to make the intervals between receipts of consecutive samples (i.e., S n − S n1 ) as equal as possible. When the attacker's jamming time A n = a is short, the system chooses a larger delay D n = u(a), so that the new transmission carries "fresh" information. More specifically, if the jamming time is zero, and if the system chooses zero delay, the new transmission contains exactly the same information as the previous transmission, and hence the new transmission is "wasted". On the contrary, the system chooses a smaller delay if the attacker's jamming time is long, and does not delay the transmission at all if the jamming time is above the threshold β. This ensures that the received information is not outdated by a large delay in the transmission. With such a water-filling policy, the system strikes the balance between sending fresh enough information (by not letting the delay too small) and receiving fresh enough information (by not letting the delay too large).
Since the system wishes to keep the time between receipts of consecutive samples equal, the attacker tries to make it difficult for the system to achieve this. The worst jamming distribution for the attacker is a fixed deterministic jamming time, against which the system can easily make sure that the updates are received periodically. Therefore, the best jamming distribution for the attacker should have high variance, where the realized jamming time is either the largest (A max ) or zero. This is exactly the equilibrium jamming time distribution specified in our theorem.

V. PROOF OF MAIN RESULT
In this section, we prove our main results in Theorem 1. We first derive the best response sampling policy BR(f A ), and then solve for the equilibrium jamming time distribution f * A . Based on these results, we can obtain the equilibrium sampling policy u * = BR(f * A ).

A. Best Response Sampling Policy
First, we derive the best response sampling policy BR(f A ) under any jamming time distribution f A .
Lemma 2: For any jamming time distribution f A that satisfies (2), the best response sampling policy is BR( with β satisfying Proof: We have proved in Lemma 1 that the best response sampling policy is characterized by [16,Theorem 4]. The results directly follow from [16,Theorem 4].

B. Equilibrium Jamming Time Distribution
Next, we derive the equilibrium jamming time distribution f * A . We first simplify the attacker's optimization problem (8) by the following lemma.
Lemma 3: The attacker's optimization problem (8) is equivalent to Proof: See Subsection A of Appendix. We verify that the equilibrium jamming time distribution f * A in Theorem 1 is a feasible solution to (15). Lemma 4: The equilibrium jamming time distribution f * A in Theorem 1 is a feasible solution to (15). Moreover, under f * A , the unique β that satisfies the first constraint of (15) is the β * defined in Theorem 1.
Proof: See Subsection B of Appendix. We need two properties of the objective function of (15) to prove that the feasible solution f * A is indeed the unique optimal solution. We write the objective of (15) as a function of β and f A as follows: Note that the argument β does not need to satisfy the first constraint in (15). The first useful property of the function g(β, f A ) is that when the jamming time distribution is equal to the equilibrium distribution (i.e., when f A = f * A ), the largest β that satisfies β = 1 2 g(β, f * A ) is β * . Lemma 5: The largest β that satisfies β = 1 2 g(β, f * A ) is β * . In addition, we have β > 1 2 g(β, f * A ) for any β > β * . Proof: See Subsection C of Appendix. The second useful property of the function g(β, f A ) is that for any β, the equilibrium distribution f * A maximizes g(β, f A ) among all distributions that satisfy (2).
Lemma 6: Given any β, the equilibrium distribution f * A is a solution to the following problem: s.t. f A satisfies (2).
Proof: See Subsection D of Appendix. Base on Lemma 5 and Lemma 6, we can prove that f * A is the unique optimal solution to (15) by contradiction. Suppose that there is a better solutionf A such that g(β,f A ) > g(β * , f * A ). Sincef A is feasible, we haveβ = 1 2 g(β,f A ), which implies β > 1 2 g(β * , f * A ) = β. Base on Lemma 6, we have Base on Lemma 5 and the fact thatβ > β, we have Therefore, we have g(β,f A ) < 2β, which violates the first constraint of (15). This contradiction implies that f * A is the unique optimal solution to (15).

C. Equilibrium Sampling Policy
It was proved in Lemma 4 that given equilibrium jamming time distribution f * A , the β * specified in Theorem 1 is the unique β in the best response to f * A . This concludes the proof.

VI. SIMULATION RESULTS
We have argued that the equilibrium jamming time distribution has a large variance and maximizes the age, while a deterministic jamming time is worst in maximizing the age. Now we consider a jamming time distribution that is the mixture of the two distribution: 1] is the weight of the equilibrium jamming time distribution and f D A is a deterministic jamming time with f D A (a) = δ (a − A avg ) . We choose the deterministic jamming time to be A avg so that the mixture distribution has a constant mean value.
Under the mixture distribution, we compute the age under the equilibrium sampling policy defined in (2), and that under the zero-wait sampling policy (i.e., samples immediately after the jamming). Fig. 3 shows the ages versus the weight α.
First, we can see that the equilibrium jamming distribution f * A is indeed better for the attacker, since the ages in both cases increase as the mixture distribution becomes closer to the equilibrium jamming distribution. Second, we can see that the equilibrium sampling policy achieves smaller ages than the zero-wait policy.

VII. CONCLUSION
We studied a dynamic game between a real-time system and an attacker. The system samples a physical process and sends the samples through a wireless channel, in order to perform some monitoring and control tasks. The attacker jams the channel to delay the status updates. We used the age of information as the performance metric to measure the timeliness of the status updates. We proved that there exists a unique stationary equilibrium in this game, and characterized the equilibrium analytically. Our results indicate that the attacker chooses a jamming time distribution with high variance, while the system chooses a sampling policy that results in low variance in the time between the receipts of two consecutive updates.

A. Proof of Lemma 3
We first calculate the average age ∆(f A , u) defined in (6). Since the jamming time distribution is i.i.d., and since the sampling policy is stationary deterministic, the average age ∆(f A , u) over the infinite time horizon is equal to the average age of each stage game. Therefore, we have We can further simplify (20) when u = BR(f A ) (i.e., when u is best response), by observing that a + (β − a) + = max{β, a}.
Hence, we have At this point, the attacker's optimization problem (8) can be rewritten as Note that we need the first constraint to ensure that the system is choosing the best response sampling policy.

B. Proof of Lemnma 4
Since we have and we know that f * A satisfies the second constraint of (15). Now it remains to show that there exists a unique β = β * that satisfies Suppose that β = 0. Then (25) reduces to which is impossible. Suppose that β ≥ A max . Then (25) reduces to which is impossible. Therefore, we must have β ∈ (0, A max ). Then (25) reduces to β = 1 2 It is not difficult to verify that the only positive solution to the above equation is Hence, under f * A , the unique β that satisfies the first constraint of (15) is the β * defined in Theorem 1.

D. Proof of Lemma 6
The objective function is a ratio of the weighted sum of [max{β, a}] 2 to the weighted sum of max{β, a}, where the weight is determined by f A . Since [max{β,a}] 2 max{β,a} is nondecreasing in a, we should assign the largest possible weight when a = A max . However, the distribution needs to satisfy the average jamming time condition. The equilibrium distribution f * A has the largest f * A (A max ) among all the distributions that satisfy (2).


Second Language Acquisition Modeling: An Ensemble Approach

Accurate prediction of students’ knowledge is a fundamental building block of personalized learning systems. Here, we propose an ensemble model to predict student knowledge gaps. Applying our approach to student trace data from the online educational platform Duolingo we achieved highest score on all three datasets in the 2018 Shared Task on Second Language Acquisition Modeling. We describe our model and discuss relevance of the task compared to how it would be setup in a production environment for personalized education.

Introduction
Understanding how students learn over time holds the key to unlock the full potential of adaptive learning. Indeed, personalizing the learning experience, so that educational content is recommended based on individual need in real time, promises to continuously stimulate motivation and the learning process (Bauman and Tuzhilin, 2014a). Accurate detection of students' knowledge gaps is a fundamental building block of personalized learning systems Tuzhilin, 2014b) (Lindsey et al., 2014). A number of approaches exists for modeling student knowledge and predicting student performance on future exercises including IRT (Lord, 1952), BKT (David et al., 2016) and DKT (Piech et al., 2015). Here we propose an ensemble approach to predict student knowledge gaps which achieved highest score on both evaluation metrics for all three datasets in the 2018 Shared Task on Second Language Acquisition Modeling (SLAM) (Settles et al., 2018).
We analyze in what cases our models' predictions could be improved and discuss the relevance of the task setup for real-time delivery of personalized content within an educational setting.

Data and Evaluation Setup
The 2018 Shared Task on SLAM provides student trace data from users on the online educational platform Duolingo (Settles et al., 2018). Three different datasets are given representing users responses to exercises completed over the first 30 days of learning English, French and Spanish as a second language. Common for all exercises is that the user responds with a sentence in the language learnt. Importantly, the raw input sentence from the user is not available but instead the best matching sentence among a set of correct answer sentences. The prediction task is to predict the word-level mistakes made by the user, given the best matching sentence and a number of additional features provided. The matching between user response and correct sentence was derived by the finite-state transducer method (Mohri, 1997).
All datasets were pre-partitioned into training, development and test subsets, where approximately the last 10 % of the events for each user is used for testing and the last 10 % of the remaining events used for development . Target labels for token level mistakes are provided for the training and development set but not for the test set. Aggregated metrics for the test set were obtained by submitting predictions to an evaluation server provided by Duolingo. The performance for this binary classification task is measured by area under the ROC curve (AUC) and F1-score.
Although the dataset provided represents real user interactions on the Duolingo platform, the model evaluation setup does not represent a realistic scenario where the predictive modelling would be used for personalizing the content presented to a user. The reason for this is threefold: Firstly, predictions are made given the best matching correct sentence which would not be known prior to the user answering the question for questions that have multiple correct answers. Secondly, there are a number of variables available at each point in time which represent information from the future creating a form of data leakage. Finally, the fact that interactions from each student span all data partitions means that we can always train on the same users that the model is evaluated for and hence there are never first time users, where we would need to infer student mistakes solely from sequential behaviour. To estimate prediction performance in an educational production setting where next-step recommendations must be inferred from past observations, the evaluation procedure would have to be adjusted accordingly.

Method
To predict word-level mistakes we build an ensemble model which combines the predictions from a Gradient Boosted Decision Tree (GBDT) and a recurrent neural network model (RNN). Our reasoning behind this approach lies in the observation that RNNs have been shown to achieve good results for sequential prediction tasks (Piech et al., 2015) whereas GBDTs have consistently achieved state of the art results on various benchmarks for tabular data (Li, 2012). Even though the data in this case is fundamentally sequential, the number of features and the fact that interactions for each user are available during training make us expect that both models will generate accurate predictions. Details of our model implementations are given below.

The Recurrent Neural Network
The recurrent neural network model that we use is a generalisation of the model introduced by Piech (2015), based on the popular LSTM architecture, with the following key modifications: • All available categorical and numerical features are fed as input to the network and at multiple input points in the graph of the network (see A.1) • The network operates on a word level, where words from different sentences are concatenated to form a single sequence • Information is propagated backward (as well as forward) in time, making it possible to predict the correctness of a word given all the surrounding words within the sentence • Multiple ordinary-as well as recurrent layers are stacked, with the information from each level cascaded through skip-connections (Bishop, 1995) to form the final prediction In model training, subsequences of up to 256 interactions are sampled from each user history in the train dataset, and only the second half of each subsequence is included in the loss function. The binary target variable representing word-level mistakes is expanded to a categorical variable and set to unknown for the second half of each subsequence in order to match the evaluation setup.
Log loss of predictions for each subsequence is minimised using adaptive moment estimation (Kingma and Ba, 2014) with a batch size of 32. Regularisation with dropout (Srivastava et al., 2014) and L2 regularisation (Schmidhuber, 2014) is used for embeddings, recurrent and feed forward layers. Data points are used once over each of 80 epochs, and performance continuously evaluated on 70 % of the dev data after each epoch. The model with highest performance over all epochs is then selected after training has finished. Finally, Gaussian Process Bandit Optimization (Desautels et al., 2014) is used to tune the hyperparameters learning rate, number of units in each layer, dropout probability and L2 coefficients.

The Gradient Boosted Decision Tree
The decision tree model is built using the Light-GBM framework (Ke et al., 2017) which implements a way of optimally partitioning categorical features, leaf-wise tree growth, as well as histogram binning for continuous variables (Titov, 2018). In addition to the variables provided in the student trace data we engineer a number of features which we anticipate should have relevance for predicting the word level mistakes • How many times the current token has been practiced • Time since token was last seen • Position index of token within the best matching sentence • The total number of tokens in the best matching sentence • Position index of exercise within session • Preceding token • A unique identifier of the best matching sentence as a proxy for exercise id Optimal model parameters are learned through a grid search by training the model on the training set and evaluating on the development set to optimize AUC. The optimal GBDT parameter settings for each dataset can be found in the Supplementary Material A.2.

Ensemble Approach
The predictions generated by the recurrent neural network model and the GBDT model are combined through a weighted average. We train each model using its optimal hyperparameter setting on the train dataset and generate predictions on the dev set. The optimal ensemble weights are then found by varying the proportion of each model prediction and choosing the weight combination which yields optimal AUC score ( Figure 1).
Finally, the RNN and GBDT were trained using their respective optimal hyperparameter settings on the training and development datasets to generate predictions on the test sets. The individual model test set predictions were then combined using the optimal ensemble weights to generate the final test set predictions for task submission.

Discussion
Our ensemble approach yielded superior prediction performance on the test set compared to the individual performances of the ensemble components (Table 1). The F1 scores of our ensemble are reported in Table 2. We note that although the within-ensemble prediction correlations are high (

Feature Importance
Given the predictive power of our model we can use the model components to gain insight into what features are most valuable when inferring student mistake patterns. When ranking GBDT features by information gain, we note that 4 out of 5 features overlap between the three datasets ( Figure 4). The unique user identifier is ranked as second on all datasets, suggesting that very often a separate subtree can be built for each user. This implies that generalisation to new users for the GBDT model would result in performance degradation.

Relevance for Real Time Prediction Delivery
In the setup at hand we have a unique identifier and most of the data available for each user during model training. This means that for example the GBDT can naturally build a subtree representing each individual user. For the model evaluation  setup where there is no need to generalize to new users this is not an issue. In a production setting however, the model has to serve new users, which would then have to be handled separately. Frequent retraining of the model would also be necessary to prevent performance degradation. This means that the unique user identifier is typically replaced by engineered features that represent the user history. An alternative would be to apply state based models such as Recurrent Neural Networks which by default encode user history without computational overhead or extra engineering effort.

Error Analysis
Although the predictive power of our model is high, there are mistake patterns that our model is not able to capture. The following sections cover two ways of characterizing subsets of the data where the model performs worse than on average. These observations could potentially be used to improve the overall model performance.

Performance Decay over Time
Due to the sequential partitioning of the training, development and test subsets, the model does not have information about each user's mistakes for the most recent events. In Figure 2 we note that this lack of information results in a degradation in performance as the predictions get further away from the horizon of labeled data points. Effects which drive this phenomenon include: 1. The data is non-stationary, i.e. the distribution it comes from varies over time 2. The model has seen less relevant information about each user when the prediction is far away from the label horizon 3. The model is overconfident far away from the label horizon since it has never experienced missing information on a user level during training We note that 3 would not be an issue if the model setup did not include a unique user identifier, which would be desirable in a production setting. For models that do include a unique user identifier as a feature, one way to potentially overcome this performance degradation would be to systematically sample subsequences of the training dataset on a user level, train models separately for each sample and then combine the models. In this way each submodel should be less reliant on the most recent exercise answers at any point in time and thus generalise better to the evaluation setup. This is in effect bagging with a sampling strategy taking consecutive time steps into account (Breiman, 1996). We did not attempt to apply this error correction here but leave it for future work.

The Influence of Rare Words
We note that the 4% of instances with the least common words contribute to 10% of the prediction error measured in Log Loss, Figure 3. This insight gives opportunity to increase prediction performance. Although not attempted here, future work includes building another ensemble component specialized in predicting mistake patterns of words not previously encountered.
In conclusion, we have developed an ensemble approach to modeling knowledge gaps applied here within a second language acquisition setting. Albeit not evaluated in a realistic production environment, our ensemble model achieves high pre- dictive performance and allows insights about student mistake patterns. Thus our approach provides a foundation for further research on knowledge acquisition modeling applicable to any educational domain.

A.1 The recurrent neural network model design
Our neural network model desisgn is described below: 1. For each word the network takes as input all available categorical features, excluding morphological features for each word. The exclusion was motivated by the fact that predictive ability added by morphological features was low when evaluated by a decision tree model.
2. Preprocessed numerical features for days and time are concatenated to an input vector.
(Preprocessing in this case means to normalize to mean zero, variance 1, remove outliers that are larger than 100, and concatenate the value itself with the value exponentiated to 0.5 as well as 2.0) 3. The categories token, part of speech, format, correct and exercise id (as described in 3.2), are each mapped to an embedding vector of length 15.
4. The above categorical features are further combined with the feature correct by using the cartesian product, and then mapping each category to an embedding vector.
5. All categorical embeddings and numerical features are concatenated together forming an input vector.
6. The input vector is fed through a two layer bidirectional recurrent neural network, where the input to both of the layers are summed with the output, forming a user state vector.
7. Another input vector is formed by concatenating categorical embeddings for the features token, part of speech, format, dependency label, dependency token, user id as well as preprocessed numerical features.
8. The user state vector is then projected to two scalars. This is done by dot multiplying it with a vector of trainable variables, as well as dot multiplying it with the second input vector from step 7. The second part accounts for the original operation done by (Piech et al., 2015). 9. We furthermore compute one scalar for each categorical feature, that is specific for the category of the feature, similar to a logistic regression model.
10. Finally, the second input vector together with all computed scalars are concatenated and fed to a 3 layer feed forward neural network.
11. The sum of all scalar values and the output of the feed forward network forms our logit, which is fed through a sigmoid function outputting the probability of a token level mistake.


A Survey of Video Based Action Recognition in Sports

Received Jan 25, 2018 Revised Apr 21, 2018 Accepted Jan 14, 2018 Sport performance analysis which is crucial in sport practice is used to improve the performance of athletes during the games. Many studies and investigation have been done in detecting different movements of player for notational analysis using either sensor based or video based modality. Recently, vision based modality has become the research interest due to the vast development of video transmission online. There are tremendous experimental studies have been done using vision based modality in sport but only a few review study has been done previously. Hence, we provide a review study on the video based technique to recognize sport action toward establishing the automated notational analysis system. The paper will be organized into four parts. Firstly, we provide an overview of the current existing technologies of the video based sports intelligence systems. Secondly, we review the framework of action recognition in all fields before we further discuss the implementation of deep learning in vision based modality for sport actions. Finally, the paper summarizes the further trend and research direction in action recognition for sports using video approach. We believed that this review study would be very beneficial in providing a complete overview on video based action recognition in sports.

INTRODUCTION
Sports are synonym with the active movements of athletes on space either court or field. These movements are usually used by coach or trainer in evaluating the performance of their athletes. In sport practice, performance analysis can be divided into two: technique analysis and tactical analysis. According to Lees [1], technique analysis studies how the actions or movements were performed by the players. Tactical analysis or so called notational analysis studies what actions were carried out and the evaluat ion of the these actions take place [2]. Therefore, activity recognition is an important layer in tactical analysis before further analysis can be done by other researches. However, this paper will be focusing on the method in recognizing the action from sport video for establishing the automated notational analysis system.
Evaluating the players' performance has becomes a challenging task due to limitation on activity recognition phase. Hence, a technology intervention for games such as wearable sensor an d video camera act as a tool to overcomes the challenges. Wearable sensor refers to the wearable device used by the athletes to collect the data of the activities in form of one-dimensional signals and the common wearable sensor used is inertial sensor [3]. Although this approach is effective to recognize the physical activities, unfortunately, the wearable sensor is less practical as athletes are not allowed to wear the sensors during the match. Not only that, the placement of sensors on to any part of human body also provide limitation to the movements of players during training or match.
Video camera is a visual sensing facility that provides visual data or video of the monitored activities and environmental changes. It has been used widely in high profile sports such as football and tennis for tactical analysis, tracking players, detecting the court lines, events recognition and video summarization [4][5][6][7][8][9]. The practicality of video based modality is more higher compared to the sensor based as no additional hardware will be attached to the athletes' body. Video camera captured the events and produced the broadcast video that receive high viewership. Therefore, these widely available video will be used by the researches to segment the useful part of video for the performance analysis [10].
Many research have implemented the video in recognizing the action, object or even vehicle [11][12][13]. But, the review study on this modality is still lacking. Therefore, this paper will be reviewed about the current and previous works on recognizing the action in sport using the video based modality.

VIDEO BASED SPORTS INTELLIGENCE TOOL
Since video based modality has attracted many researchers' attention in sport performance analysis, there are several tools developed by international company either for mobile or desktop usage. A summary of the existing intelligence system based on video for various sports is shown in Table 1. Description [14] A software developed by Dartfish company to highlight interested information from a match or training video for any sports. It is a useful for desktop and mobile tool for coach to analyses the performance of the athletes. [15] Vizrt introduced a few intelligence systems for modelling and animation, sport analysis, automated recording, graphic, video management and etc. for various high profile sports such as football, basketball, tennis and hockey. [16] Nacsport has been a marketer for video sport analysis software since 2008. It offers sport professionals to evaluate behaviors of all kinds of athletes. T he data provided by software are both qualitative and quantitative and arranged according to need. Hence, the analysis can be done faster. [17] Sportradar provide services in collecting and analyzing sports data for over 1000 companies including sport federation, media companies and bookmakers. [18] Coach's eye is a video analysis app from TechSmith Corporation that analyses the video recorded from any device. It provides the real time analysis for individual athlete or team which enable coach to share the analyzed results immediately with the athletes for fast performance improvement.
However, in the aforementioned systems, there is no tool that can automatically recognize and classify the activities in sport for notational analysis. The highlighted actions for analysis are based on human perception. For example, in Dartfish, the analyst must manu ally watch, select and interpret the action of the athlete before further analysis by the software can be taken.

ACTION RECOGNITION FRAMEWORK
The video consists of temporal sequences of 2D images or can be defined as a set pixels in 3D space [19]. Figure 1 shows the overview of video based action recognition framework. According to [19][20][21], the action recognition can be divided into two approaches: simple approach and complex approach. Simple approach involves low level mechanism in which the recognized action is obtained from the detection and tracking of the human in each frame [22]. For example, to recognize "smash" activity in badminton match, firstly the players are detected from the surrounding background and then tracked to create a motion description of their movement. However, complex approach uses a lot of high level mechanism. This includes complex feature extraction and classification to recognize the human action.

Detection and Tracking
In action recognition, tracking was used to generate the movement trajectory from the detected region of interest to infer the performed action. In [23], dense trajectories approach were proposed to describe videos. From each frame, dense points were sampled and tracked based on displacement information from a dense optical flow field. To evaluate video description in the context of action classification, a bag of features approach has been used. Proposed method shows an improvement on state-of-the-art on datasets with different level of difficulty such as KTH and Hollywood2. Since optical flow method is not suitable to be used in low resolution video, Zhao et.al [24] proposed a Region -based Mixture Model (RMM) for action classification of low resolution video. In this method, a set of long term motion trajectories and long term common shape is extracted from each video sequence using Layered Elastic Motion Tracking (LEMT) method. In addition, Particle Filter (PF) approach is a technique of Bayesian sequential importance sampling. Both work in [25,26] utilized this approach in their studies. While work in [26] defined the human gestures and real time human tracking from depth data using PF method, work in [25] utilized a human -robot interface system which incorporates PF and Adaptive Multi-space Transformation (AMT) to track the pose of the human hand for controlling the robot manipulator. PF is used to estimate the translation of the human hand while AMT is used to improve the accuracy and reliability in determining the pose of the robot.

Feature Extraction and Classification
In machine learning, feature extraction is described as a pre-processing part to remove redundant part and reduce the dimensionality [27]. In [28], feature extraction was defined into low-level and high-level features. The key point for low level feature are corners, edges, blobs or contours while high level feature is more holistic like the structured information related to the action being taken. For sport video, important features including field, court, athlete and score board are extracted. However, classification is described as method to recognize the types of actions after feature extraction phase is completed.
In [29], different kind of features were extracted using fast featu re descriptor which is 3D Histograms of Texture (3DHoTs) method. This method is derived from projecting depth frames onto frontal, side and top planes. And then, to classify the features a new classifier which is Multi-class Boosting Classifier (MCB) has been proposed. By providing a better margin distribution, the method was claimed to be efficient by maximizing the mean of margin whereas the variance of margin is still in minimum level. Not only that, work by Li et.al [30] presented an automatic players d etection and analysing system in sports video sequences. There are three levels in recognizing the action of player in the moving background video. At granularity level, global motion estimation for filtering was proposed. Then, at the middle level, there is a segmentation of the highlighted object and finally at fine level, action recognition using Continuous Hidden Markov Model (CHMM) was proposed. As for work in [31], Hidden Markov Model (HMM) was trained to highlight the summary of RGB-D sport video that has been extracted by HAR method. Zhou et.al used linear Support Vector Machine (SVM) classifier to predict the class of action based on action representation after the extraction of local motion and appearance features has been done [32].

DEEP LEARNING IN VIDEO BASED APPROACH
Deep learning is a subtype of machine learning but more promising approach as compare to other conventional machine learning approaches. Deep learning is similar with one of the machine learning model called artificial neural network layer. Both consist of input layer, hidden layer and output layer. But, the number of hidden layer in deep learning could reach to hundreds layer and this is where the term "deep" is came from. Since deep learning shows an impressive result on several applications such as image classification, it has been implemented in the action recognition application. Table 2 shows the differences between deep learning and machine learning in term of preprocessing phase, size of data set, training time and hardware requirement. In deep learning, preprocessing phase is not required. As illustrated in Figure 2, deep learning eliminates the manual feature extraction phase because the network extract features directly from images during training. To train the deep learning model, large data sets are required to compensate large size of hidden layers. However, due to the vast development of broadcast sport video online which is accessible, sport video analysis using deep learning has become the emerging research interest [33]. Due to complexity of the deep learning, more time is needed to train the model. Hence, high performance GPU is important to reduce the training time. GPU is chosen compared to CPU because it has parallel architecture that accelerates the computing process. One of the most popular deep learning model is Convolutional Neural Network (CNN). Table 3 shows a few various types of CNN which were formulated to classify difference type of action. Deep ConvNet [36] 3D ConvNet [37,38] T wo-stream ConvNet The work in [35] modelled the Convolutional Neural Network (CNN) to classify 1 million Youtube videos contain 487 classes (called Sports -1M dataset). The model which can be seen in Figure 3 was divided to process the input into low resolution context stream and high resolution fovea stream before the networks was trained using UCF-101 dataset alone in order to increase the performance of runtime. Both streams consist of convolutional, normalization and pooling layers that alternate each other and the two s treams finally converged into two fully connected layers. The performance of top layers on UCF-101 dataset show significant improvement compared to the UCF-101 baseline model. One of the major issues in action recognition is high inter and intra class variations and large class imbalance. Hence, to overcome the aforementioned problem, work in [36] implemented a 3D Convolutional Neural Network (CNN) for multi-label class-imbalanced in hockey videos on two deep approaches: 1) ensemble of k-binary network; and 2) single multi-label k-output network. The 3D convolutional and pooling process were embedded in the proposed approaches to tackle the multi label recognition.
Wang et.al [37] proposed Trajectory-Pooled Deep-Convolutional Descriptors (TDD) conducted on two challenging datasets: HMDB51 and UCF101. In this method, firstly, the two -stream ConvNets were trained on both datasets as a deep ConvNet to extract multiclass feature maps from video sequences [38]. Then, the TDD descriptor was obtained using pooling process of these ConvNets before classifying the action using SVM classifier to perform action recognition.
Besides CNN, another model of supervised deep learning which is good in handling sequential data is recurrent neural network (RNN) and one of the common RNN is called Long Short -Term Memory  [39,40]. The work in [36] which explained earlier also attempt to embedded the proposed 3D CNN with LSTM. The LSTM was added in between flattened CNN feature layers and dense layers (see Figure 4). However, the result shows that the combination of LSTM reduces the performance because the short sequences of frame were used. Figure 4. General structure of the network [36] Not only that, there are several attempts to implement LSTM in football video analysis. For example in [41], LSTM was proposed to classify football video sequence on MICC-Soccer-Actions-4 dataset which contains four action classes using visual and motion cont ent. In this study, the features that represent the visual content were established using the Bag of Words (BoW) technique while SIFT -based model was proposed to extract the motion features. These features were used in LSTM to classify the action. The resu lts show that the performance in action classification for the combination proposed models is 92%. Tsunoda et.al [42] also worked on action recognition for football video by implementing hierarchical LSTM. In the proposed model, several CNN models were integrated with two layers of LSTMs (see Figure 5). CNN was used to extract multiple person-centered features. Then, the first layer of LSTM was computed to integrate all k-numbers of person-centered features before the last LSTM layer integrated the temporal sequence of integrated multiple person features. The work in [43] focused in implementation of LSTM in ice hockey video which has been extracted into sequence of images to classify five puck possession events: dump in, dump out, pass, shot and loose puck delivery. In this work, firstly, the whole frame and each player were extracted using pre-trained CNN to obtain the content information, individual action and interaction between players. The pre-trained AlexNet model was chosen for the extraction phase as the number of available data is small and the model showed the great achievement in various computer vision tasks. Later in events prediction phase, one layer of LSTM model is used to classify the five puck possession events.
However, in [44] , deep fusion framework was introduced by combining spatial features from CNN with temporal features from LSTM on three datasets: UCF11, UCFSports and jHMDB. Four CNNs and LSTM fusion methods for the recognition of human actions were proposed (two direct mapping mode ls and two merged models). First two models are single stream models called as conv -L and fc-L (direct mapping models). These models extract CNN activation outputs from the last convolutional layer and the first fully connected layer for each frame of each video. The final output of these models is determined by considering the output from the soft-max layer of LSTM network which fed with features obtained from the CNN. The second remaining models called fu-1 and fu-2 (two merged models) are two stream approaches where two networks are merged. From the experiment, it shows that the direct mapping methods are less accurate compared to the two merged models. But, among two merged models, fu -2 shows the good accuracy value. It proved that this fusion method produces best results with the aid of deep layer wise structure.

CONCLUSION AND FURTHER RESEARCH DIRECTION
Due to the rapid development of broadcast video online on sport match, it has become a tool for action recognition for sport analysis. Two main approaches used by researches are machine learning and deep learning. There are various works have been done on both approaches. Recently, deep learning approach such as CNN and RNN have been tremendously used in many works as it provides a better accuracy and capable of eliminating the complex preprocessing phase. However, it has becoming the issues since each proposed method can only classify the actions for certain sport because different sport has different context and features. Hence, in future, a flexible method for action recognition can be proposed in which one method can classify different type of sports.