Handling statistical heterogeneities such as heterogeneous distributions in labels, data and models across devices and FL iterations.
Our Approach
In this section, we discuss the problem formulation of new classes in FL, and our proposed framework (Algorithm REF ) to handle the same. The overall architecture is given in Figure REF .
Problem Formulation
We assume the following scenario in federated learning. Suppose there are \(M\)  nodes (devices) in the FL network, holding private local data \(\mathcal {D}_i = \lbrace x_{i,j},y_{i,j}\rbrace \)  where \(i\)  is the FL iteration and \(j\)  is the user index. Each node consists of public data \(\mathcal {D}_0=\lbrace x_0,y_0\rbrace \) . The public data is assumed to be present across the global and all local users as discussed in [1]} to handle the various statistical (model) heterogeneities which is a common phenomena in FL. The overall label-set of public dataset is \(Y = \lbrace y_0\rbrace \) , which represent its unique labels. We re-purpose this public dataset as test set and do not expose it to local models during FL training iterations, but expose only during testing for consistency. Our work's main contribution is to propose a framework to identify new labels across different users without transferring private data in FL setting. We also assume each user can stream data with new labels at any iteration which does not belong to public label-set \(Y\) , i.e. \(\mathbf {y}_{i,j} \notin Y\) . In other words, the global user has no idea of these new labels.
[ht]
Our Proposed Framework


Input: Public Dataset \(\mathcal {D}_0\lbrace x_0,y_0\rbrace \) , Private Datasets \(\mathcal {D}_m^i\) , Total users \(M\) , Total iterations \(I\) , LabelSet \(l_m\)  for each user, Overall Public LabelSet \(Y\) ,
Output: Trained Model scores \(f_G^I\)

Initialize \(f_G^0 = \mathbf {0}\)  (Global Model Scores)

\(i=1\)  to \(I\) 
\(m=1\)  to \(M\)

Build: Model \(\mathcal {D}_m^i\)  and predict \(f_{\mathcal {D}_m^i}(x_0)\)

Local Update:
Choice 1: New classes are not reported
\(f_{\mathcal {D}_m^i}(x_0) = f_G^I(x_0^{l_m})+\alpha f_{\mathcal {D}_m^i}(x_0)\) , where \(f_G^I(x_0^{l_m})\)  are global scores of \(l_m\)  with \(m^{th}\)  user, \(\alpha = \frac{len(\mathcal {D}_m^i)}{len(\mathcal {D}_0)}\)

Choice 2: New classes are reported
Train a new model with \(\mathcal {D}_0\)  and \(\mathcal {D}_m^i\)  (new data) together, and send weights of the last layer (\(\mathbf {W}_m^i\) ) to global user.

Global Update:
Choice 1: No user reports new classes
Update label wise
\(f_G^{i+1} = \displaystyle \sum _{m=1}^{M}\beta _m f_{\mathcal {D}_m^i}(x_0)\) , where
\(\beta ={\left\lbrace \begin{array}{ll}1 & \text{If labels are unique} \\\text{acc}(f_{\mathcal {D}_m^{i+1}}(x_0)) & \text{if labels are not unique}\end{array}\right.}\)

where \(\text{acc}(f_{\mathcal {D}_m^{i+1}}(x_0))\)  is the accuracy metric, defined by the ratio of correctly classified samples to total samples for a given local model.

Choice 2: Any user reports new classes
Create Data Impressions (DI) for each user \(m\)  with weights \(\mathbf {W}_m^i\)  (Section REF ). Average DI of all users with new classes, \(\mathbf {X}^i=\sum _{m \in M_{S_k}}\mathbf {X}_m^i\) , where \(M_{S_k}\)  is set of users with new label \(k\) .
Perform k-medoids clustering on \(\mathbf {X}^i\)  across \(M_{S_k}\) . Number of clusters = Number of new labels (\(l_{new}\) ).

Update public dataset with new DI (\(\mathbf {X}^i\) ), \(\mathcal {D}_{new} = \mathcal {D}_0 \bigcup \mathbf {X}^i\) , add \(l_{new}\)  to \(l_{m}\)  and \(Y\) .

Anonymized Data Impressions
The main challenge/objective is to detect similar labels across different users in FL heterogeneous settings without the knowledge of local user data. This necessitates us to construct anonymized data without transferring raw sensitive data, and identify new class similarities on the anonymized data. We motivate our framework from the creation of Data Impressions (DI) using zero-shot learning as proposed in [2]} to compute Anonymized Data Impressions. Let us assume a model \(\mathcal {M}\)  with input \(\mathbf {X}\)  and output \(\mathbf {y}\) , where \(\mathbf {X} \in \mathcal {R}^{M\times N}\)  is the set of features and \(\mathbf {y}\in \mathcal {R}^M\) . Now, the anonymized feature set \(\bar{\mathbf {X}}\) , which has same properties of \(\mathbf {X}\) , can be synthesized in two steps:
<TABLE>(a) Sample Softmax Values:
The first step is to sample the softmax values from the Dirichlet distribution [3]}. The Class Similarity Matrix (CSM) is created which contains important information on how similar the classes are to each other. If the classes are similar, we expect the softmax values are concentrated over these labels. CSM is obtained by considering the weights of the model's last layer. Typically, any classification model has the final layer as fully-connected layer with a softmax non-linearity. If the classes are similar, we find similar weights between connections of the penultimate layer to the nodes of the classes [2]}. The Class Similarity Matrix is constructed as,

C(i,j) = wiTwj||wi||||wj||

where \(\mathbf {w}_i\)  is the vector of weights connecting the previous layer nodes to the class node \(i\) . \(\mathbf {C} \in \mathcal {R^{K\times K}}\)  is the Class Similarity Matrix for \(K\)  classes. We then sample the softmax values as,

Softmax = Dir(K,C)

where \(C\)  is concentration parameter which controls the spread of softmax values over class labels.
(b) Creating Anonymized Data Impressions:
Let \(\mathbf {Y}^k = [\mathbf {y}_1^k,\mathbf {y}_2^k,\cdots ,\mathbf {y}_N^k] \in \mathcal {R}^{K\times N}\)  be the N softmax vectors corresponding to class \(k\) , sampled from Dirichlet distribution from previous step. Once we obtain the softmax values, we compute the synthesized data features (Data Impressions) by solving the following optimization problem using model \(\mathcal {M}\)  and sampled softmax values \(\mathbf {Y}^k\) 

x = x LCE (yik,M(x))

To solve this optimization problem, we initialize the input \(\mathbf {x}\)  to be random input and iterate until cross-entropy loss (\(L_{CE}\) ) minimization. This process is repeated for all \(K\)  categories. In this way, anonymized data impressions are created for each class without the visibility of original input data. We use the TensorFlow framework [5]} for all our experiments.
<FIGURE>
Proposed Framework
There are three steps in our proposed framework (Algorithm REF ).
(a) Build: Each local user creates their own model with their local private data for a specific iteration.
(b) Local Update: In this step, if new classes are not reported, we perform simple weighted \(\alpha \) -update [6]}, where \(\alpha \)  governs the contributions of new and old models across FL iterations shown in Algorithm REF  Choice 1. If new classes are reported, we train the new class data along with public dataset, and send the new model weights to global user (Choice 2).
(c) Global update: In this step, if no user reports new classes, we perform label-based averaging using the parameter \(\beta \) , which governs contributions of overlapping labels using corresponding test accuracies (Choice 1). If user reports new classes, we create Anonymized Data Impressions (DI) for new classes followed by unsupervised clustering using k-medoids with motivations from [7]} (Choice 2).
Typically, statistical heterogeneities are widely observed in practical FL settings, hence Choice 1 handles heterogeneities in local and global update steps [8]}, while Choice 2 handles new classes in our proposed framework.

Experiments and Results
We simulate our experiments using Raspberry Pi 2 as our user device with Google Speech Commands (GKWS) [9]} and UrbanSound8K (US8K) [10]} datasets across 10 FL iterations/communication rounds using our proposed framework. In GKWS, we choose the keywords: Yes, No, Up, Down, Left, Right, On, Off, Stop and Go, and perform regular Mel-frequency Cepstral Coefficients (MFCC) extraction as performed in [11]}, with sampling frequency of 14400 HZ. The MFCC data is divided into 20 windows and each window is of size 50 ms. US8K, an environmental sound dataset, consists of 10 classes of sound events: air conditioner, car horn, children playing, dog bark, drilling, engine idling, gun shot, jackhammer, siren and street music. We perform similar preprocessing as performed in GKWS for US8K as well.
Public Dataset:
We create a Public Dataset (\(D_0\) ) with 2400 audio frames for GKWS (8 keywords with 300 frames each), and 400 audio frames for US8K (8 sounds with 50 frames each) as shown in Table REF . \(D_0\)  is visible to both global and local users in each FL iteration, and is updated with data synthesized for unseen/new classes only – Anonymized Data Impressions.
We initially consider eight labels with an initial Public Dataset in both datasets before streaming new classes (Table REF ). We simulate two scenarios for testing just our zero-shot framework – 1) new classes only (homogeneous) with limited users and FL iterations (3 users and 10 FL iterations) for effective analysis of results, 2) new classes with statistical heterogeneities in both labels and models as performed in [8]}, (10 users and 30 FL iterations). This exhibits near-real-time model heterogeneities as shown in Table REF , and effective convergence.
New Classes: We introduce two new/unseen labels {Stop, Go} for GKWS and {Siren, Street music} for US8K across four FL iterations and two users. In the homogeneous case, for GKWS, we induce 400 samples each with Stop class in iteration 4 for both User 1 and User 2, and 500 samples each with Stop in User 1 iteration 8 and Go in User 2 iteration 8. Similarly, we induce 50 samples each with Siren class in iteration 4 for both User 1 and User 2, and 50 samples each with Siren in User 1 iteration 8 and Street music in User 2 iteration 8. This is the FL scenario with new classes without any heterogeneities. We also discuss similar FL scenarios with statistical heterogeneities.
<TABLE>(a) Label Heterogeneities:
In every FL iteration, we consider a random number of audio frames generated between 200-300 samples per label for GKWS, while 40-50 samples per label for US8K. We split these labels across three users such that labels can either be unique or overlapping across users. We also simulate non-IIDness across FL iterations with disparities in both labels and distributions in data (statistical heterogeneities).
(b) Model Heterogeneities:
We consider the three model architectures as shown in Table REF  motivated from [11]}, [14]}, and also change model architectures, filters and activation functions across FL iterations in addition to label heterogeneities with new classes (Table REF ). The user iterations are chosen at random.
<FIGURE>Discussion on Results
<TABLE>From Table REF , we can observe that there is an accuracy increase in the FL scenario with just new classes (without heterogeneities) in corresponding global updates for all three users than their respective local update accuracies for both datasets in spite of new classes streaming in. The average local-global accuracy increase across all 10 FL iterations and 3 users is \(\sim \) 3.361% and \(\sim \) 3.529% respectively for GKWS and US8K. Similarly, we can also observe that with our proposed framework, the final global accuracies (with convergence after all FL iterations) even with new classes and heterogeneities are 96.541% and 82.498% (Figure REF ) which are much higher than their respective local update accuracies. The corresponding local-global update accuracies across 30 FL iterations and 10 users are shown in Figure REF . The class similarity matrix of different classes for GKWS is showcased in Figure REF , which elucidates the misclassifications. We can also infer that the clusters effectively formed with k-medoids are equal to the number of new classes, which are visualized using Principal Component Analysis (PCA) in two-dimensions. The new classes can either be different or same across user devices (Figure REF ), and these classes are correctly mapped back to the respective end-user devices. The new labels are then finally added to the overall label set while the corresponding averaged data impressions are added to the public dataset.
<FIGURE>
On-Device Performance
Raspberry Pi 2 (900MHz quad-core ARM Cortex-A7 CPU with 1GB RAM) is used for evaluating our proposed FL framework as it has similar hardware and software (HW/SW) specifications to predominant contemporary IoT/mobile devices. The computation times are identical for both datasets due to similar preprocessing. The size of the models used are also 520 kB, 350 kB, 270 kB respectively for user architectures mentioned in Table REF .
<TABLE>
Conclusions
This paper presents a novel framework for handling new labels in a federated learning setting. We propose a zero-shot learning framework by synthesizing Anonymized Data Impressions from Class Similarity matrices to learn new classes across different user devices. We also account for heterogeneities in labels and models across different FL communication rounds, and systematically analyze the results for two widely used audio classification applications – keyword spotting and urban sound classification. We further demonstrate the effectiveness and scalability of our proposed FL framework by simulating our experiments on-device using a Raspberry Pi 2.
