<jupyter_start><jupyter_text>Exercise recommander with flashcards August 2019 - <NAME>*** Flashcards are well-known revision tools for students used well before the arrival of smartphones. However recent apps made learning easier by allowing anybody to manage large amount of flashcards on their mobile, and so have them anywhere and anytime. Lot of useful options come with those apps, such as frequency management of individual cards, or reminders. We propose in this work to develop a new option making learning more efficient. Modern smartphones' artificial intelligence are getting more and more efficient, allowing us to develop learning aptitudes to flashcards app. The one we will focus on will be the aptitude to find correlations between cards.__Let us begin with some examples :__Imagine a set of 5 cards with correlated exercises of increasing difficulty A>B>C>D>E . Multiple users use this set of cards.The app does not have prior information about those correlations, however it has to figure them out by itself by noticing that users have in average more difficulty to answer correctly to C than to A and B. The app thus can adapt each card apparition frequency according to the user level, estimated with its past history.This will be even more useful with large set, so if different cards cover the same topic and have the same difficulty, the app could focus on this subset of cards when it think that the user should work on this one to improve its level.__Problematic :__It would be sufficient to use only one single user data. A large amount of data is required to find patterns among cards. So the different account should be able to share information. This leads us to privacy questions: how to insure and preserve any user privacy?__Project:__To tackle this task, we will set up a federated learning protocol allowing the data to be processed on the users' machine. Only information about new estimated correlation about cards will be shared publicly. First, define a dictionnary for our prototype of card set:  0. Set up a virtual environment We define a cards-user couple which makes our environment for a reinforcement learning algorithm<jupyter_code># Dictionary of cards with different levels of difficulty (A = easy < B < C < D < E = hard)
# Dependencies are then randomly generated

import numpy as np

list_of_cards = []
dict_of_cards = {}

for i in range(10):
    list_of_cards.append('A' + str(i))
    list_of_cards.append('B' + str(i))
    list_of_cards.append('C' + str(i))
    list_of_cards.append('D' + str(i))
    list_of_cards.append('E' + str(i))

list_of_cards = np.sort(list_of_cards)

for i in list_of_cards:
    if i[0] == 'A':
        diff = 1
    if i[0] == 'B':
        diff = 2
    if i[0] == 'C':
        diff = 3
    if i[0] == 'D':
        diff = 4
    if i[0] == 'E':
        diff = 5
    diff *= 3 * (np.random.rand() + 1) 
    dict_of_cards[i] = {'diff':diff } 
# Dependencies are then introduced :
# An integers measures the increase in chances to solve the card if
# the one specified has been seen before

for j in range(50):

    for n, i in enumerate( list_of_cards ):
        N = len(list_of_cards)
        k = np.random.randint(0 ,N )
        if list_of_cards[k] != i:
            bonus = ((np.power(n,2)/10) * np.random.rand()/(k+1))
            if bonus > 1 :
                if list_of_cards[k] in dict_of_cards[i]:
                    dict_of_cards[i][list_of_cards[k]] += int(bonus)
                else:
                    dict_of_cards[i][list_of_cards[k]] = int(bonus)
dict_of_cards
# We now have a dictionnary of 50 items of different difficulties
# 10 cards are randomly picked and proposed to the user

# The goal for the algorithm is to present the best order, that has the highest expected score
# Considering that the user aquires experience with the cards s.he sees
# We define the function which simulates the user solving an ordered list of cards
# It takes into account the cards seen before the current one

import numpy as np

def solve_cards(ordered_cards, dict_of_cards = dict_of_cards):
    solved = []
    done = []
    for i in ordered_cards:
        
        N_try = 1
        for k, l in dict_of_cards[i].items():
            if k in done:
                N_try += l
        t = 0
        for k in range(N_try):
            t += np.random.rand()
        if t > dict_of_cards[i]['diff']:
            solved.append(1)
        else:
            solved.append(0)
        done.append(i)
    return solved
essai = np.random.choice( list(dict_of_cards.keys()), 20 )
mes = []
for i in range(40):
    perm = np.random.permutation(essai)
    moy = []
    for j in range(100):
        moy.append(np.sum(solve_cards(perm)))
    mes.append((np.average(moy), np.std(moy)))
import matplotlib.pyplot as plt

t = list(zip(*mes))[0]
u = list(zip(*mes))[1]
plt.errorbar(range(len(t)), t, yerr=u)
# This plot shows that with our fictitious cards-user model, for a given subset of flashcards,
# the order in which they are presented matters for the expected score.

# Before designing a federated learning model, we will do a centralised one. It will be train by reinforcement.

# The states will be given by the ordered subset of cards selected (randomly or not) to the user
# The actions will be the ordering in which the card will be presented for resolution
# The reward will be the score obtain by the user

# We will use an actor-critic policy:
# A critic deep learning model outputs the expected reward of a couple (state, action), meaning in this case
# (subset, permutation)<jupyter_output><empty_output><jupyter_text>1. Define a critic to estimate rewards<jupyter_code>import numpy as np
import torch as th
from torch import nn

import matplotlib.pyplot as plt


import torch.nn.functional as F

class Critic(nn.Module):
    '''
    in_state_size = dimension of the states (number of picked cards to order)
    in_action_size = dimension of the actions
    out_size = dimension of the output 
    All input sizes should be equal
    
    output : a float tensor giving the expected rewards
    '''
    
    def __init__(self, in_state_size, in_action_size, out_size = 1):
        super().__init__()
        # hidden layers
        self.hidden_state = nn.Linear(in_state_size, 32)
        self.hidden_action = nn.Linear(in_action_size, 32)
        self.hidden_state1 = nn.Linear(32, 64)
        self.hidden_state2 = nn.Linear(64, 64)
        self.hidden_action1 = nn.Linear(32, 64)
        self.hidden_action2 = nn.Linear(64, 64)
        self.hidden_sum1 = nn.Linear(64, 64)
        self.hidden_sum2 = nn.Linear(64, 64)
        self.hidden_sum3 = nn.Linear(64, 64)
        # Output layer
        self.output = nn.Linear(64, out_size)
        
    def forward(self, state, action):
        # Hidden layers with relu activation for states
        x = F.relu(self.hidden0_state(state))
        x = F.relu(self.hidden_state1(x))
        x = F.relu(self.hidden_state2(x))
        
        # Hidden layers with relu activation for action
        y = F.relu(self.hidden0_action(action))
        y = F.relu(self.hidden_action1(y))
        y = F.relu(self.hidden_action2(y))
        
        # sum channels
        
        z = x + y
        z = F.relu(self.hidden_sum1(z))
        z = F.relu(self.hidden_sum2(z))
        z = F.relu(self.hidden_sum3(z))
        # Output layer with softmax activation
        out = self.output(z)
        
        return out
my_critic = Critic(20, 20)
my_critic<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07940744695048713, 'fraction_numerical': 0.013479247297477646, 'mean_word_length': 3.053001622498648, 'pattern_counts': {'":': 0, '<': 14, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '636776', 'license': 'Apache-2.0', 'n_tokens_mistral': 2249, 'n_tokens_neox': 2164, 'n_words': 900, 'path': 'Dupuy_Nicolas/Exercise_recommander_with_flashcards-v2.ipynb', 'repo_name': 'bharatnishant/UdacityOpenSource', 'size': '7500'}