extfieldsofstudy:
Computer Science
Biology

text:
Neuronal Circuit Policies

We propose an effective way to create interpretable control agents, by re-purposing the function of a biological neural circuit model, to govern simulated and real world reinforcement learning (RL) test-beds. We model the tap-withdrawal (TW) neural circuit of the nematode, C. elegans, a circuit responsible for the worm's reflexive response to external mechanical touch stimulations, and learn its synaptic and neuronal parameters as a policy for controlling basic RL tasks. We also autonomously park a real rover robot on a pre-defined trajectory, by deploying such neuronal circuit policies learned in a simulated environment. For reconfiguration of the purpose of the TW neural circuit, we adopt a search-based RL algorithm. We show that our neuronal policies perform as good as deep neural network policies with the advantage of realizing interpretable dynamics at the cell level.

The functions of many neural circuits within its brain have been identified (Wicks & Rankin, 1995;Chalfie et al., 1985a;Li et al., 2012;Nichols et al., 2017). In particular, a neural circuit which is responsible for inducing a forward/backward locomotion reflex when the worm is mechanically exposed to touch stimulus on its body, has been well-characterized (Chalfie et al., 1985a). The circuit is called tap-withdrawal (TW) and it comprises 9 neuron classes which are wired together by means of chemical and electrical synapses. Synaptic polarities (either being excitatory or inhibitory) of the circuit have then been predicted, suggesting that the circuit realizes a competitive behavior between forward and backward reflexes, in presence of touch stimulations (Wicks & Rankin, 1995;Wicks et al., 1996).
Behavior of the tap-withdrawal (TW) reflexive response is substantially similar to the control agent's reaction in some standard control settings such as the impulse response of a controller operating on an Inverted Pendulum (Widrow, 1964;Doya, 2000;Russell & Norvig, 2010), a controller acting on driving an under-powered car, to go up on a steep hill, known as the Mountain Car (Moore, 1990;Singh & Sutton, 1996), and a controller acting on the navigation of a rover robot that plans to go from point A to B, on a planned trajectory, with two control commands of angular and linear velocity.
We intend to take advantage of the similarity and reconfigure the synaptic and neuronal parameters of a deterministic dynamic model of the TW neural circuit, in each of the mentioned control settings. We use publicly available reinforcement learning toolkits, to evaluate the performance of our neuronal circuit policies. The environments include the inverted pendulum (Schulman et al., 2017), the continuous mountain car of OpenAI Gym 1 and rllab 2 , and the Cart-pole of rllab . In a real robotic setting, We also determine a control task for a rover robot to park autonomously in a specific parking spot, by a learned TW neuronal policy. For all three control challenges, we preserve the near-optimal wiring structure of the TW circuit and adopt a search-based reinforcement learning (RL) algorithm for synaptic parametrization of the network. The approach is named as neuronal circuit policies.
Our principle contribution in this work is to demonstrate the performance of a compact neuronal circuit model from the brain of the C. elegans worm, as an interpretable continuous time recurrent neural network, in standard control and RL settings. In our experimental evaluations, we demonstrate that our control agent can achieve the performance of the conventional and the state-of-the-art artificial intelligence (AI) control agents, by solving five RL tasks. We show how a learned neuronal circuit policy in a simulated environment, can be transferred to a real robotic environment. We also demonstrate that the function of the neurons in a learned neuronal network is interpretable.

Preliminaries
In this section, we first briefly describe the structure and dynamics of the tap-withdrawal neural circuit. We then introduce the mathematical neuron and synapse models utilized to build up the model of the circuit.

Tap-Withdrawal Neural Circuit Revisit
A mechanically exposed stimulus (i.e. tap) to the petri dish in which the worm inhabits, results in the animal's reflexive response in the form of a forward or backward movement. This response has been named as the tap-withdrawal reflex, and the circuit identified to underly such behavior is known as the tap-withdrawal (TW) neural circuit (Rankin et al., 1990). The circuit is shown in Figure 1. It is composed of four sensory neurons, PVD and PLM (posterior touch sensors), AVM and ALM (anterior touch sensors), four interneuron classes (AVD, PVC, AVA and AVB), and two subgroup of motor neurons which are abstracted as a forward locomotory neurons, FWD, and backward locomotory neurons, REV. Neurons recurrently synapse into each other with excitatory and inhibitory synaptic links. It has been shown that sensory neurons of the TW circuit get activated as a result of an input tap, and transfer the stimulus through the modulatory interneurons PVC and AVD to the command neurons AVA and AVB (Wicks et al., 1996). The TW reflex is then modulated by a competition between these two command neurons, either resulting in a forward scape response (AVB's activation dominates AVA's) or a reversal scape response. Throughout the paper, we illustrate how such recurrent neuronal network can be deployed in standard RL settings. We first sketch how we modeled neurons and synapses to build up the TW circuit.

Neuron Model
Most of the neurons in C. elegans are observed to exhibit electrotonic dynamics (Kato et al., 2015), meaning that electric charges spread passively inside a neuron creating graded potentials. This implies that the neurons are non-spiking. Dynamics of the neurons' membrane potential therefore, were modeled by the well-known, deterministic ordinary differential equation (ODE), the single-compartment mem-brane equation (Koch & Segev, 1998): where C m , G Leak and V Leak are parameters of the neuron and I (i) in , stands for the external currents to the cell. We adopted Eq. (1) to govern interneurons' dynamics.
For interacting with the environment, We introduced sensory and motor neuron models, separately. A sensory component consists of two neurons S p , S n and a measurable dynamic system variable, x. S p gets activated when x has a positive value, whereas S n fires when x is negative. Mathematically, the potential of the neurons S p , and S n , as a function of x, can be expressed as This maps the region [x min , x max ] of system variable x, to a membrane potential range of [−70mV, −20mV ]. Note that the potential range is selected to be close to the biophysics of the nerve cells, where the resting potential is usually set around -70 mV and a neuron can be considered to be active when it has a potential around -20 mV (Hasani et al., 2017). Similar to sensory neurons, a motor component is composed of two neurons M n , M p and a controllable motor variable y. Values of y is computed by y := y p + y n and This maps the neuron potentials M n and M p , to the range [y min , y max ]. FWD and REV motor classes in Figure 1, are modeled in this fashion.

Synapse Model
Chemical synapses are points at which two neurons trade information by the release of neurotransmitters. The chemical synaptic current depends on a non-linear component standing for their conductance strength, which is a function of the presynaptic neurons' potential, V pre , and have a maximum weight of w,(standing for the maximum conductance of the synapse) as, (Koch & Segev, 1998): Moreover, the synaptic current linearly depends on the postsynaptic neuron's membrane potential, V post , and therefore can be formulated as, (Koch & Segev, 1998): where by varying E, the reversal potential of the synapse, it realizes inhibitory or excitatory connection to their postsynaptic neurons.
An electrical synapse (gap-junction), which is a physical junction between two neurons, was modeled by a constant conductance,ω, where based on the Ohm's law their bidirectional current between neurons j and i, can be computed asÎ For simulating neural networks composed of such dynamic models, we adopted a implicit numerical solver . Formally, we realized the ODE models in a hybrid fashion which combine both implicit and explicit Euler's method. See supplementary materials, Section 2, for a

Algorithm 1 Random Search + Objective Indicator
Input: A stochastic objective indicator f , a starting parameter θ Input: concrete discussion on the model implementation, and the choice of parameters.
Note that one objective of the solver is to be employed in a real-time control system. For reducing the complexity therefore, our method realizes a fixed-step solver. The solver's complexity for each time step ∆ t is O(|# neurons|+ |# synapses|). The solver is implemented in C++ in which we construct the TW circuit for performing specific control tasks. We now need to formalize a learning platform to tune the parameters of the circuit for the desired control problem.

Search-based Reinforcement Learning
In this section we formulate an RL setting for training the parameters of the neural circuit to perform the balancing of the inverted pendulum, control the mountain car, and to park the rover robot.
The behavior of a neural circuit can be expressed as a policy π θ (o i , s i ) → a i+1 , s i+1 , that maps an observation o i , and an internal state s i of the circuit, to an action a i+1 , and a new internal state s i+1 . This policy acts upon a possible stochastic environment Env(a i+1 ), that provides an observation o i+1 , and a reward, r i+1 . The stochastic return is given by R(θ) := T t=1 r t . Objective of the Reinforcement learning is to find a θ that maximizes E R(θ) .
Approaches to find such optimal θ, can be categorized into two major groups, based on how randomness is formulated for the environment explorations (Schulman et al., 2015;Salimans et al., 2017): I-Gradient-based and II-search-based methods. The principle of gradient-based RL is to perform random sampling for generating a i , and use the action's influence on the return value, to improve θ (Williams, 1992). Search-based methods directly randomly sample parame-ters and estimate how good these random parameters are, to update θ (Salimans et al., 2017;Szita & Lörincz, 2006). Here, we adopted such search-based optimization which can be applied in any control setting, regardless of the internal structure of the policy, (black-box optimization). One major obstacle for search-based optimization is the stochastic nature of the RL environment, which makes the objective function, f (θ) = E(R θ ), a probability distribution and the underlying optimization problem an instance of Stochastic Optimization (Spall, 2003). Samples of the objective distribution can be generated by running rollouts with π θ on the environment. A possible solution to overcome a high variance when estimating E(R θ ), is to rely on a very large number of samples (Salimans et al., 2017;Duan et al., 2016), which nonetheless comes with high computational costs. Our approach is based on a Random Search (RS) (Rastrigin, 1963) optimization, combined with an Objective Estimate (OE) as an objective function f : θ → R + .
The OE generates N rollouts with π θ on the environment and computes an estimate of E(R θ ) based on a filtering mechanism on these N samples. We compared two filtering strategies in this context; I-Taking the average of the N samples, and II: taking the average of the worst k samples out of N samples. The first strategy is equivalent to the Sample Mean estimator, whereas the second strategy aims to avoid getting mislead by outlying high samples of E(R θ ). Our objective for realizing the second strategy was the fact that a suitable parameter θ makes the policy π θ , control the environment in a decent way even in difficult situations (i.e. rollouts with the lowest return).
In both strategies, to ensure that a single outlying high OE for some θ does not hinder the algorithm to find a legitimately good parameter, the OE is reevaluated after it is utilized M times, within the algorithm. The full algorithm is outlined in Algorithm 1.

Experiments
The goal of our experiments is to answer the following questions: 1) How would a neural circuit policy perform in a basic standard control setting? 2) When possible, how would the performance of our learned circuit compare to the other methods? 3) Can we transfer a policy from a simulated environment to a real environment? 4) Can we interpret the behavior of the neural circuit policies?
We prepared five environmental settings for the TW sensory/motor neurons and then deployed our RL algorithm to learn the parameters of the TW circuit to realize the given control objective. Environments include I) Inverted pendulum of Roboschool (Schulman et al., 2017), II) Mountain car of OpenAI Gym, III) Mountain car of rllab, IV) cart-pole balancing of rllab and V) Parking a real rover robot from Motor outputs -Control +Control +" -" a learned policy in a simulated environment. The code is available online 1 .
The major constraint that prevents the TW neural circuit (shown in Figure 1) to be tested on arbitrary tasks, is its limited number of sensory and motor neurons. As the TW circuit allows us to incorporate only two input and one output variables, we selected tasks that can be solved with only two observations. We chose environments from different toolkits to evaluate our neural circuit policies on a variety of dynamics, interactions and reward settings. For instance, the two mountain car suites of OpenAI Gym and rllab realize different positive rewards functions as explained in section 4.3.

Inverted pendulum with the TW neural circuit
The TW neural circuit shown in Figure 1, contains four sensory neurons. It therefore, allows us to map the circuit to only two input variables. Note that as we discussed in section 2.2, a sensory component expressed in the form of Eq. (2) and Eq. (3), consists of two neurons for incorporating positive and negative values of the dynamic system variable. The inverted pendulum environment provides four observation variables as shown in Figure 2A. The position of the cart x, together with its velocityẋ, the angle of the pendulum ϕ 2 along with its angular velocityφ.
Since the main objective of the controller is to balance the pendulum in an upward position and make the car stay within the horizontal borders, we fed ϕ, and x, as the inputs to the sensors of the TW circuit, as illustrated in Figure 2B.
Control commands to the pendulum were originated by the abstract motor neuron classes, FWD and REV components. The activity of these components are governed by Eq. (4)  (5), respectively, and represented in Figure 2B, graphically.
We set up the search-based RL algorithm to optimize the neurons and synapses parameters ω,ω, σ, C m , V Leak and G Leak in the Roboschool RoboschoolInvertedPendulum-v1 environment with an slight modification (Schulman et al., 2017) in the reward calculation. It is desirable for the cart to stay in the center of the horizontal space. Therefore, we incorporated an additional axillary reward; if the pendulum is in the up-right position and in the center, an additional 20% reward is collected. This bonus linearly decreased. For instance the bonus reward is 10% if the cart is halfway to the end. Right at the borders the bonus reward vanishes. A video of different stages of the learned neuronal circuit policy for the inverted pendulum can be viewed at https://youtu.be/iOHeQ7DhQv8 Note that we were also able the train the TW circuit to solve original version of the inverted pendulum task when feeding ϕ, andφ into the circuit. However, we observed a slow drift of the pendulum toward one of the two ends in final policy, which we wished to eliminate.

The mountain car (OpenAI Gym) control with the TW neural circuit
In this experiment we trained the TW circuit to drive the car shown in Figure 3A uphill to the right-hand side, by generating gravitational momentum. The observation variables are the car's position (on the horizontal axis, x) together with its linear velocity. The control signal applies force to the car to move it to the right-and left-hand side, periodically, to finally bring the car up on the hill.
Similar to configuration of the inverted pendulum, we set the observational and motor variables for the tap withdrawal, as illustrated in Figure 3B. The circuit was then learned by the search-based RL algorithm. A video illustrating the control of the car at various episodes during the learning process can be viewed at https://youtu.be/lMrP1sXp3jk.

The mountain car (rllab) control with the TW neural circuit
The environmental observations and motor actions for the mountain car are the same in rllab and OpenAI Gym. Therefore, we set up the TW circuit the same way we did for the mountain car in the Gym.
The major difference of the two environments are in the way the reward is computed. The environment of rllab has a graded continuous reward which is directly associated with the position of the car, whereas in the OpenAI Gym implementation, reward is sparse and gets allocated only once the car reaches a certain altitude. Furthermore, in both frameworks, reward is subjected to a penalty as follows; in rllab the penalty is constant, whereas in the Gym the amount of the penalty varies depending on the amplitude of the performed action. Consequently, energy efficient solutions achieve a higher score, in the Gym environment, in contrast to the rllab version, where the highest scoring solutions bring the car uphill as fast as possible.

Cart-pole (rllab) control with the TW circuit
The Cart-pole environmental setting is substantially similar to that of the inverted pendulum. In contrast to the inverted pendulum experiment, here we mapped the observation variables, pole angle, ϕ and the pole's angular velocityφ to the sensor. As a result, the controller is not aware of the position of the cart, x, and whether the boundary of the movable space is nearby or not. An intuitive solution for handling more control variables would be to add sensory neurons to the TW circuit. However, in the present work we intended to maintain the structure of the TW circuit constant, and test its capabilities in control scenarios with the resulting partial observability. Environmental variables for the cart-pole suite were mapped to the circuit elements as denoted in Table 1.

Autonomous parking of the rover with the TW neural circuit
In this experiment, we generalized our TW neuronal circuit policy to a real-world control setting. We let the TW circuit learn to park a rover robot on a determined spot, given a set of checkpoints which form a trajectory, in a deterministic simulated environment. We then deployed the learned policy on a mobile robot in a real environment shown in Figure  4A. The key objective here was to show the capability of the method to perform well in a transformation from a simulated environment to a real setting. For doing this, we developed a custom deterministic simulated RL environment 1 .
The rover robot provides four observational variables (Starting signal, position of the rover, x, y and its angular pose, θ), together with two motor actions (linear and angular velocity, v and w). We mapped all four observatory variables, as illustrated in Figure 4B, to the sensors of the TW. Note that here the geometric reference of the surrounding space is set at the initial position of the robot. Therefore, observation variables are only positive.
We mapped the linear velocity (which is a positive variable throughout the parking task) to one motor neuron and the same variable to another motor neuron. We determined two motor neurons for the positive and negative angular velocity. The mapping details are provided in Table 1. This configu-1 https://github.com/mlech26l/neuronal_ circuit_policies ration implies that the command neuron, AVA, controls two motor neurons responsible for the turn-right and forward motion-primitives, and AVB to control the turn-left and forward motor neurons. Therefore, the TW circuit is able to govern the robot's locomotion in three different directions, forward, left-and right-turns, which are sufficient to perform a parking trajectory. Furthermore, each command neuron is able to move the rover forward and turn it to the left or right.

RL SETTING FOR THE PARKING TASK
A set of checkpoints on a pre-defined parking trajectory were determined in the custom simulated environment. For every checkpoint, a deadline was assigned. At each deadline a reward was given as the negative distance of the rover to the current checkpoint. The checkpoints are placed to resemble a real parking trajectory composed of a sequence of motion primitives: Forward, turn left, forward, turn right, forward and stop. We then learned the TW circuit, by the RL algorithm. The learned policy has been mounted on a Pioneer AT-3 mobile robot and performed a reasonable parking performance. The Video of the performance of the TW neuronal circuit policy on the parking task can be viewed at https://youtu.be/Vwydc2ez9Wc.

Experimental Evaluation
In this section, we thoroughly assess the results of our experimentation. We qualitatively and quantitatively explain the performance of our neuronal circuit policies. We then benchmark our results where possible, with the existing methods and describe the main attributes of our methodology. We finally illustrate how the activity of the learned neuronal circuits can be interpretable.

Performance
The training algorithm was able to solve all five tasks, after a reasonable number of iterations as shown in Figure 5. All learning curves reached a stable state after the given number of iterations. Jumps in the learning curves of the mountain car in OpenAI Gym ( Figure 5B) are the consequence of the sparse reward. The inverted pendulum of the Roboschool and the cartpole of rllab, realized substantially similar learning curves ( Figure 5A and 5D) . This indicates the robustness of the policy and the learning algorithm to different frameworks for a given control task. For the deterministic parking trajectory, the learning curve approaches a stable state, exponentially fast.
The final return values for the basic standard RL tasks (provided in Table 2), matches that of conventional policies (Heidrich-Meisner & Igel, 2008), and the state-of-the-art deep neural network policies learned by many RL algorithms (de Froissard de Broissia & Sigaud, 2016;Schulman et al., 2017;Berkenkamp et al., 2017). Table 2, also depicts the average return over the entire training iterations. The average return is not significant compared to the other algorithms applied to the artificial neural network policies , due to the smaller number of training iterations. Training curves in our case however, reaches an stable state reasonably fast in all tasks even with a fewer number of epochs.
To take advantage of parallel hardware resources and to further increase the performance of our training algorithm, we deployed an ensemble of TW agents which were trained independently in parallel. Depending on the difficulty of the

Effect of filter size on the training performance
Fig 2F shows how the choice of the objective-estimate, affects the training performance. The objective-estimate is defined as the mean of the k (=filter size) returns out of 20 rollouts. When k = 20, the estimation is equal to the sample mean. We tested two environments with different reward settings: The mean return of Mountaincar (OpenAI Gym) after 50,000 training iterations to represent a sparse reward scenario and the mean return of our modified inverted pendulum task after 15,000 training iterations as an example of a gradual reward setting. The results denote that in a sparse reward setting, filtering of the high-outliers tend to degrade the training performance while in a gradual reward setup, filtering of the outliers improves the performance. The reported values in Fig2F, correspond to the average, when running this experiment 10 times. Further discussions about this can be found in the supplementary materials Section 4.

Interpretability of the neuronal circuit policies
Interpretability of neural network policies is still a challenge to be solved (Heess et al., 2016;Bacon et al., 2017). Successful attempts on the development of interpretable representation learning have been provided where the latent space of the learned policies demonstrate interpretable skills Florensa et al., 2017). A neuronal circuit policy has the distinct attribute of being interpretable at the cell level. Here, we show how the activity of the learned TW neuronal policies, in different domains is auditable. Figure  6A to 6C represent the normalized membrane potential of neurons in learned policies for the inverted pendulum, the A B C Figure 6. Neuronal activity of the Learned neuronal policies. A) The inverted pendulum circuit B) The mountain car (OpenAI Gym) circuit C) The parking circuit. Individual neuron's resting potential, V leak , is mapped to 0, neuron's maximum potential is mapped to 1 and neuron's minimum potential is mapped to -1. mountain car and the parking task. We describe the global neuronal dynamics in these scenarios.
Inverted Pendulum -Activity of the learned Tapwithdrawal's neurons in a successful episode of the inverted pendulum control, is shown in Figure 6A. The learned circuit surprisingly realized a competitive behavior between two sub-circuits within the network, similar to the reflexive behavior observed in the worm (Wicks et al., 1996). Neurons AVM, AVD, AVA, and REV control the pendulum not to fall on the left side, whereas a circuit composed of PLM, PVC, AVB and FWD, controls the other side of the balance. The antagonistic behavior between the command neurons AVA and AVB balances the pendulum in the middle. Note that DVA neuron acts as a mediator which couples the kinetics of the two sub-circuits. Note that Neurons' dynamics are realized with a variety of time-constants during the balancing control episodes. For instance PVC modulates fast undulations while the motor neurons work with a slower dynamic.
Mountain Car -As illustrated in Figure 6B, a gradual increase in the amplitude of the overall periodic dynamic of the neurons is observed due to the alternation of the direction of movement to bring the car uphill. Sensory neurons linearly transform the environmental observations to the global activity of the system, forming a phase-shifted highly synchronized behavior. In every episode, by the rising of PDV (higher negative velocity) or PLM (be at a positive x), all neurons approach their resting potential. This feedback mechanism which prevents unstable behavior, originated from inhibition of PVC and AVD by the ALM sensory neuron. At every period in which the car reaches the maximum position on the left-hand side, a circuit composed of AVM, AVD, AVB and FWD applies a force-pulse which pushes the car to get closer to the target. AVA and AVB retained their objective as in the TW actual circuit, as the command neurons which enforce the network's decision to the motor neurons REV and FWF. However their activity is not fully antagonistic in this learned network. This is where the competition "A more active command neuron, gives rise to a more active downstream motor neuron, and as a results winning the competition", is still present.
Parking of the rover - Figure 6C shows the activity of the learned TW policy on the parking task. AVM receives the starting signal and accordingly, a left turn together with moving forward are initiated by the motor neuron FWD and LFT. This is governed by the command neuron AVB. The rover continues moving forward and initializes a right turn by the motor neurons FWD and RGT. This is controlled by AVA. AVD learned to control the turning-phases of the rover by moderating its state, (a higher potential during right turns and a lower membrane state during left turns). PVC and DVA function as the pre-command neurons which incorporate the sensory and the network inputs, into a readable action for the command neuron AVB.

Conclusions
In this work, we showed that a neuronal circuit policy can be adopted to function as a controller for standard control tasks expressing similar characteristics to the original purpose of the circuit. We illustrated the performance of such policies learned by a search-based RL algorithm, in standard basic RL domains. We also generalized the use of such policies in a real-domain robot control. More importantly, for various domains we showed that our learned policies developed interpretable skills at the neuron level.
Our neuronal policies are limited to the domains in which the sensory observations and motor actions are as many as the available sensory and motor neurons of the neural circuit. Implementation of larger neural circuits to build interpretable controllers for tasks with higher degrees of freedom will be the focus of our continued effort. Moreover, a policy gradient algorithm may significantly enhance the performance of our policies. We open-sourced our policies, to encourage other researchers to further explore the attributes of neuronal circuit policies and apply them to other control and reinforcement learning domains. 002  003  004  005  006  007  008  009  010  011  012  013  014  015  016  017  018  019  020  021  022  023  024  025  026  027  028  029  030  031  032  033  034  035  036  037  038  039  040  041  042  043  044  045  046  047  048  049  050  051  052  053  054 Supplementary Materials 1. Videos of the performance of the learned neuronal circuit policies Description URL TW circuit controls an inverted pendulum at different stages of the training process https://youtu.be/iOHeQ7DhQv8 TW circuit controls a mountain car at different stages of the training process https://youtu.be/lMrP1sXp3jk TW circuit performs the parking task https://youtu.be/Vwydc2ez9Wc Parking task with four degrees-of-freedom https://youtu.be/jVQqKoHopTU Table S1. Videos

Neural Circuit Implementation and Setup
In this section we describe how to integrate the neuron and synapse equations into a computational framework to build up the TW circuit. Due to non-linearity of the sigmoid function in Eq. (7)Main-text, the neuron's differential equation, Eq.
(1)Main-text, becomes non-linear. Unfortunately, there are no complete theory of solving problems of this class, explicitly (Gerald, 2012). Thus, for simulating neural networks composed of such dynamic neuron models, we adopted a numerical implicit solver.
When a network structure is dense and full of synaptic pathways, the set of ODEs (neuron potentials), defined in Eq.
(1)Main-text, becomes stiff . Therefore, in order to overcome stability issues we used an implicit derivation approximation method as follows : In this way, we discretize the time variable and transform the set of ODEs into a set of iterative equations.
For each neuron, Eq. (1)Main-text, exposed to chemical synaptic currents in the form of Eq. (7)Main-text, and gap junction currents in the form of Eq. (8)Main-text, if we apply approximation of the Eq. (1)supplementary and assume v pre (t) ≈ v pre (t − ∆ t ), we can show that the membrane potential of that neuron at time t, is computable by: In Eq.
(2)Supplementary, ω ex,i , ω inh,i , ω gj,i , respectively stand for the overall conductance of the excitatory synapse, inhibitory synapse and the gap junction, where ω ex,i = g ex,i (v pre ), ω inh,i = g inh,i (v pre ), and ω gj,i =ω. Variables together with their boundaries, and constants in Eq.

Sensory neuron mappings
As introduced in section 2.2, input and output values are mapped to the potential of sensory respectively motor neurons by an affine mapping. This affine mapping is defined by the minimum and maximum value of the particular input or output value. For each of the five RL environments we set these boundary values separately, according to the following  Table S5. Input and output boundary values used to define the affine sensory and motor mappings 4. Discussion on the influence of the filtering on the training performance We experienced that a large filter size, i.e an objective estimate, which filters out only few outlying samples, performs better when the reward is sparse. We observed that learning curve of such environments usually has the shape of a series of step functions. One example is the Mountaincar (OpenAI Gym) environment, where a positive reward is only given once at the end, when the entire task has been solved. A large filter size performs better, in such tasks, because the episodes that have been solved by luck, are not filtered out and instead, have a large effect on the estimate. This is crucial for the training in a sparse reward setting, since during the learning phase, we observed that the agent first is unable to solve the task, then is able to solve only a few cases (e.g. when the car starts somewhere uphill) and so on, until the final agent solves the task even in the difficult scenarios.
Furthermore, we observed that a small filter size, i.e. an objective-estimate which is computed only from the lowest samples, performs slightly better in tasks with gradually increasing reward. Learning curve of such environments are usually shaped smoother. An example of such task is the inverted pendulum (Roboschool), where a positive reward is given all the time when the pendulum is still facing upwards. Our filtering strategy performs slightly better here, because, as originally intended, the estimate is not influenced by outlying high samples. In the inverted pendulum task, such samples occur when the pendulum start in an almost straight pose and a high return can be collected without any specific action.

TW circuit can realize more degrees of freedom
In our first parking experiment, the TW circuit is able to make a turn left and move the rover forward with only one command neuron being active. This means that the circuit is able to solve the task with having binary activation states (active, not active) of the command neuron. To test the flexibility of the TW circuit and underlying neuron model, we set up a second parking experiment. In this experimental setup, we connected the command neuron AVA to two motor neurons responsible for turning right and moving backwards, and AVB to two motor neurons responsible for turning left and moving forward. In this setup, the controller is principally able to move the robot to 4 different directions: Forward, Backward, turn left


A framework for managing interest in technical debt: an industrial validation

Technical debt management entails the quantification of principal and interest. In our previous work we had introduced a framework for calculating the Technical Debt Breaking Point (TD-BP), which is a point in time where the accumulated interest becomes larger than the principal; thus the debt of the company is no longer sustainable after this point in time. In this paper, we instantiate this framework and validate its ability to assess the breaking point of source code modules in an industrial setting. The results of the validation suggest that the calculated TD-BP is strongly correlated to experts' opinion on the sustainability of modules, and that it can accurately rank components, based on their maintenance difficulty.

INTRODUCTION
In the last years, the technical debt research community has been spending increasingly large effort on the different activities of Technical Debt Management (TDM). To a large extent, this research is focusing on issues related to cost and benefit analysis of software quality management [3]. An overview of TDM research is presented in, among others, two literature reviews [4,5]. Although research on TDM has been intense in the last years, the community still faces major challenges, two of which are the focus of this paper: (a) a sound estimation of the amount of TD (i.e., the quantification of interest and principal [5]), and (b) the monitoring of the TD amount, as it increases due to the accumulation of interest [6]. In our previous work [6,7], we begun to address these challenges, by defining a theoretical framework, named FITTED, that can be used for the long-term management of Technical Debt. FITTED considers that interest accumulating during software evolution, can potentially outgrow the amount required for repaying the principal of TD. Therefore, it is critical for project managers to be able to estimate the point in time where the accumulated interest will be equal to the TD principal. This point in time is called the breaking point (TD-BP), in the sense that any benefit deriving from the decision to take on technical debt is being neutralized after that point, i.e. the cost becomes higher than the benefit [8]. The aim of this paper is twofold: (a) we instantiate the FITTED framework by providing details on the calculation of principal and interest, and (b) we validate the proposed instantiation of the FITTED framework in an industrial setting. We note that both the instantiation of the framework and its subsequent validation are at the source code level; instantiation could also take place for technical debt at other levels but that is out of the scope of this paper. The instantiation of the framework is supported by a software tool which can extract data on: (a) the principal of technical debt, (b) the interest accumulated in every version of the system, and (c) the occurrence of the breaking point for every version of the system. The validation of the framework was conducted in collaboration with a software development company. Specifically the tool was applied on two of their current software development projects, and its results were analyzed together with two software engineers. Compared to the state-of-research (see Section 2.1), this is the first automated approach that calculates the Breaking Point of Technical Debt and validates it within an industrial setting. The rest of the paper is structured as follows: In Section 2, the FITTED framework is presented. In Section 3, we present our approach for validating FITTED, and in Section 4, we explain the design of our study. Next, we present the results of our analysis, in Section 5, and we discuss them, in Section 6. In Section 7 we refer to the threats to validity, and finally, in Section 8, we present our conclusions and future work.  [10] and [18] or through references to economic theory, i.e., it is defined as the increase rate of the amount of TD [11], or through software engineering concepts, i.e., it is defined as a change in a design-time quality attribute e.g., [12] and [13]. Moreover, it is observed that almost 28% of primary studies recognize the effect of interest probability. Two of these studies [14,15] approach TDM with a financial risk management theory and define interest probability as the standard deviation of interest rate, i.e., they consider it as the probability of TD to occur. Additionally, in Ampatzoglou et al. [6] it is suggested that almost 21% of studies refer to the evolution of interest along time. Under this perspective, interest is characterized either as compound or continuously increasing. On the other hand, Chin et al. [16] suggest that there is one type of interest which is simple. In particular, they propose that the cost of the organization to restrict TD neither increases nor decreases, but it is stable across time. Finally, only 17% of the studies suggest a specific methodology for measuring interest. In most of the cases, the estimation is performed by using historical data, documentation, and maintenance.

FITTED: A Framework for Managing Interest in Technical Debt
To develop a theory on managing technical debt interest, we borrowed the rationale of the economic interest theories on the equilibrium achievement. More particularly, we adopt the idea of the breaking point at the money market, where money supply is equal to money demand. As long as accumulated interest is lower than the principal, the benefit derived by the initial decision to save effort is more than the cost generated by the extra effort to maintain the software. On the other hand, when the accumulation of interest overcomes the principal, then the cost needed for the system's maintenance becomes more than the money saved when technical debt incurred. Consequently, the point where the accumulated interest is equal to the principal (breaking point) is very critical in technical debt management and can help project managers in their decision making. Under this perspective, we map principal to money supply -since it represents the amount of money available -and the accumulated amount of interest with money demand -since it embodies the extra amount of money that will be necessary for future maintenance activities caused by the TD. Figure 1 represents the FITTED Interest Theory: the horizontal axis depicts time, while the vertical axis stands for money amount. The blue curve Σ(Ι) denotes the increasing cumulative interest, as the software develops. We consider cumulative interest as continuously increasing, because it consists of the accumulation of interest in every successive version of the project and because TD interest is compound [6]. Principal, on the other hand, is represented by the green line P and is also considered as increasing, in the sense that the introduction of new functionality to the software generates new TD items. Concerning principal, we have to point out that in our original analysis of the FITTED framework [6], we have defined it as an amount of effort saved once while taking on technical debt and considered it stable throughout software evolution. However, as software grows, technical debt keeps accumulating. Therefore, in this study we take into account the development of principal and we calculate it during software development, so that we are able to reach a more accurate assessment of the version where breaking point occurs.

Figure 1: FITTED Interest Theory
As described by the figure, until time stamp t0, interest is lower than the principal and taking on technical debt can be perceived as beneficial for the project. After time point t0, interest accumulates and becomes larger than the principal and extra maintenance costs exceed the initial money saving. Therefore, the equilibrium point E, denotes the point in time t0 where the complete amount of money saved by incurring TD (i.e., the principal) equals the money spent on extra maintenance activities caused by the incurred TD [8]. If the development team proceeds with some repayment activity, e.g. at timestamp tr, accumulated interest increases, because of the addition of the repayment effort, and line Σ(I) shifts upwards to Σ(I)΄. However, the slope of the line decreases, since lower maintenance activities are expected in the future due to the repayment. On the other hand, principal decreases at time stamp tr, because of the repayment activity, and then follows the course of line P΄. Consequently, the equilibrium point moves to the right E΄, and the benefit period increases to time stamp t0΄ [8].

PROPOSED APPROACH
In order to instantiate the aforementioned approach, we need to describe the way that the two components of TD-BP can be calculated. To this end, in Section 3.1 we describe the employed way for calculating principal at source code level, whereas in Section 3.2, we present the proposed approach for assessing interest on source code artifacts.

Principal Estimation
As proposed in literature, TD principal can be calculated as a function of three variables [17]. The first variable is the number of problems that must be fixed, the second one is the time required to fix each one of these problems, and the third one is the cost for fixing each problem. Regarding the number of must-fix problems. In our previous work [7] we have presumed that there is an actual design quality for any object-oriented software, which can be estimated by a proper fitness function. Under this perspective, the 'spread' between the optimal and the actual design, as it can be derived by the difference in their fitness function values, can be mapped to the principal, i.e., the effort needed to convert the actual system to the correspondent optimum one. This notion of 'distance' is depicted in Figure 2, as Effortr.
For the purpose of this study, we have decided to estimate principal, based on the computation of a widely used OSS platform, SonarQube-formerly known as Sonar, which is a state-of-the-art tool for calculating TD principal. According to its documentation 1 , SonarQube aims at the continuous evaluation of software quality. SonarQube can assess the quality of software on a plentitude of programming languages, generating documentation on quality measures and issues, such as coding rule violations. The analysis has been performed according to the platform's default format, without any further configuration. The platform algorithm is based upon an adopted version of the SQALE method proposed by Letouzey [18], in which a remediation index is obtained for requirements of an applicable Quality Model. For example, for a requirement stating that all files should have at least 70% code coverage, the corresponding remediation action is to write additional tests. A remediation function maps effort to each action, for example, 20 minutes per uncovered line of code. Finally, for each artifact, the remediation index relating to all the characteristics of the Quality Model is obtained by adding all remediation indices linked to all quality requirements. The resulting SQALE Quality Index is considered to represent the principal of the TD for the assessed source code.

Interest Estimation
In this section we discuss the way we calculate interest in the proposed approach. In particular, in Section 3.2.1 we present the theoretical background of the approach, whereas in Section 3.2.2, we present the metrics and the method that we have used for its instantiation at the source code level, also known as code TD.

FITTED Interest Theory
Assuming the optimal and actual systems of Figure 2, in the common case, maintaining the optimum system requires less effort than maintaining the actual system. As shown in the figure, adding a new feature A to the optimum system needs a certain effort, noted as Effort(optimum), whereas adding the same feature to the actual system necessitates a larger effort, noted as Effort(actual). The difference between these two efforts represents the interest that is accumulated during this maintenance activity, i.e., the addition of feature A. However, since the evolution of the software cannot be predicted, it is not possible to foresee what kind of modifications will be made in a system during future releases. Hence, we base our assessment of future maintenance effort on historic data, by considering past effort spent on maintenance activities. More specifically, although many measures can be used for the calculation of past effort, we have selected to use the average number of lines of code added between sequential releases. Added lines of code indicate the amount of effort required for the addition of new functionality and in a way the effort needed for changing existing modules. Supposing that an average of k lines of code are added in the transition from one version to another, then we can accept that adding k lines to a high design quality system (as denoted by the assigned fitness function value) is easier than making the same addition to a lower design quality system. At this point we suppose that maintenance effort is dependent on the design quality, as in (1): where c is an arbitrary constant. Based on the abovementioned approach, the fitness value for both the optimum and the actual systems can be calculated and therefore we can capture the proportion of the theoretical over the actual effort, as in (2) (2) Hence, if past maintenance effort (i.e., the number (k) of added lines) is used to define the actual maintenance effort, then the optimum effort can be directly obtained. Consequently, the amount of interest accumulated between any two sequential versions can be calculated as the distance between the optimum and the actual effort and is given by (3) and (4).
Since the optimum value for each one of the applied metrics could be either the minimum or the maximum value, interest is given by the actual effort (k) multiplied by 1 minus the ratio of the optimum fitness value to the actual fitness value, if the optimum value is the maximum oneas in (3). Otherwise, if the optimum value is the minimum one, interest is given by the actual effort (k) multiplied by 1 minus the ratio of the actual fitness value to the optimum fitness value-as in (4).

Interest in Source Code Debt
Given the fact that interest is closely related to the system's maintainability [19], our analysis concentrates on the estimation of interest, based on well-known maintainability models. Maintainability models are sets of low-level metrics that quantify design-time quality attributes, such as inheritance, cohesion, coupling, complexity and size, which are somehow aggregated in one maintainability index. Therefore, given the relation between maintainability and interest, in this study we define the fitness value used for the estimation of the interest, using the set of metrics that are accurate maintainability predictors. Riaz et al. [20], in their systematic review have studied all existing maintainability models, and reported on their accuracy. Based on the results of that systematic literature review, a list of ten metrics stand out as successful maintainability predictors (see Table 1)-and this is the set of metrics that we use in our study. Table 1 depicts these metrics and the quality attribute to which they refer, as presented by Arvanitou et al. [21]. Nevertheless, we need to note that in our case no aggregation function is used for these metrics, since it is out of the scope of this paper. More specifically, in order to assess the optimum design, our analysis consists of the following steps. Firstly, classes of high similarity level with the investigated class are found (see Section 3.4). The identification and comparison of a class with structurally similar ones or the calculation of the fitness function is crucial, in the sense that the maintainability of a class of tens of LoC cannot be comparable with one with hundreds of LoCs. Secondly, the optimum value for each one of the aforementioned metrics is identified, the difference between the actual and the optimum fitness values is calculated and the average distance between the actual and optimum design is assessed. Subsequently, the interest is calculated as described above, in equations (3) and (4). Finally, taking into account the number of LoC that can be added in a unit of time (according to past data) and the developer's hourly rate, interest is estimated in terms of money.

Breaking Point
According to the abovementioned analysis, an estimate of the interest in USD can be achieved, based on the difference between theoretical effort (maintenance performed on the optimum system) and the actual effort, on past data concerning the time required for the addition of lines of code, and the per hour cost of a developer's work. Therefore, given the principal, as it comes from SonarQube, and knowing the interest accumulated in one version of the software, the ratio of the principal over the interest gives the number of versions after which the company will reach the breaking point, as shown in (5).

Tool Support
To boost the applicability of the aforementioned methodology, and moreover to validate the FITTED framework, we created a desktop application, developed in Java, called Breaking Point Calculator (BPC). The procedure is separated into two phases, the analysis phase and the computation phase. During the analysis phase, the source code of the project is analyzed and the metrics needed for the calculation of the results are retrieved. Each project is analyzed with the aid of two software tools, SonarQube and Percerons Client: x SonarQube is used for the extraction of five metrics which are used as a factor of similarity and technical debt principal. The metrics used for identifying similar classes are depicted in Table 2, accompanied by their description. The analysis of a project by SonarQube can take several hours, thus it takes place on our server in UoM (University of Macedonia) where it is installed. After the analysis is complete the metrics are stored in SonarQube database, which we access to retrieve them. x Percerons Client is a toolset 2 developed to support empirical software engineering research. Percerons is able to calculate metrics for every class contained in a .JAR file and produces a .CSV output [22]. Percerons Client is used to extract the object-oriented metrics for each class, on every version of the investigated project. The output document is parsed by BPC and the metrics of each class are stored into a database.

Complexity
The complexity calculated based on the number of paths through the code.

Metric Description
Functions Number of functions.

NCLOC
Number of physical lines that contain at least one character which is neither a whitespace nor a tabulation nor part of a comment.
Sqale Index Effort needed to fix all maintainability issues.
Statements Number of statements.
Once the analysis phase is completed, we proceed with the computation of the breaking point, by selecting the project under investigation, along with the computation scope, i.e., whether we want to perform the computation on "class" or "package" level. Classes and packages selected for our scope are being referred to as artifacts. The goal of the computation phase is the calculation of the breaking point.
To this end, we retrieve all the artifacts for each version of the project selected, along with the similarity and objectoriented metrics that characterize each one. For every artifact under study, we detect the most similar ones by using the aforementioned similarity metrics. Thereafter, we calculate the optimal artifact by selecting the best values among the grouped artifacts.

Figure 3: Breaking Point Calculator
On the completion of this phase, we are able to proceed with the calculation of the fitness value, the principal, the interest and the breaking point. To sum up, by the time the computation phase is completed, the following information can be extracted for any artifact from any available project version: 1. The five most similar artifacts to the investigated one, along with the values of the similarity metrics and the similarity rate. 2. The values of the object-oriented metrics for the most similar artifacts and the investigated one.
3. The values of the quality metrics for the optimum artifact. 4. The principal in USD. 5. The interest in USD. 6. The breaking point in versions. Finally we are producing overview reports for the project in .CSV documents in order to get a more comprehensive picture of it, as well as to enable the comparison of multiple artifacts. Some screenshots of the tool are provided in Figure 3.

CASE STUDY DESIGN
To validate the proposed framework and its ability to estimate the TD-BP, we performed an industrial case study in a small-medium enterprise (SME) in Greece, which is active in the domain of mobile applications. The company wants to keep its anonymity, thus all records in the dataset have been anonymized, and no personalized information can be provided, either about the company and its projects, or the case study participants. The case study is designed according to the guidelines by Runeson et al. [23]. In the next sections we present the four parts of our research design, i.e., objective and research question (see Section 4.1), case selection and units of analysis (see Section 4.2), data collection (see Section 4.3), and analysis (see Section 4.4).

Objective and Research Question
The goal of the case study is to validate the FITTED approach, regarding its ability to assess the breaking point in an industrial environment. Therefore, our study focuses in the estimation of the accuracy of the framework in assessing the breaking point. To this end, we derived the following research question: RQ: Is the calculated breaking point metric correlated with the effort required for maintenance activities? This Research Question aims at validating the proposed instantiation method of FITTED, and in particular it assesses if the TD-BP is correlated with maintenance efforts.

Case Selection and Units of Analysis
To collect data for our case study, we executed the Breaking Point Calculator Tool on the source code of two projects of the collaborating SME: (a) a lottery engine, and (b) an online betting system. Since both systems are medium scale and a manual inspection of the code might be required from the participants, we preferred to scope our analysis at high-level packages. An additional reason for this decision is that by scaling up the level of granularity for our reporting, we come closer to the architectural debt notation that has higher impact than code TD, since larger amount of TD and interest are expected to occur. Therefore, the units of our analysis are 18 packages, contained in these two different android application projects. The projects have been developed in different context and under dissimilar circumstances; hence they are expected to have different levels of accumulated TD and interest. Moreover, prerequisites for the selection of the projects were that the projects contained adequate number of versions and that the developers would be available to answer a short questionnaire concerning the development of the projects.

Data Collection
To answer the aforementioned research question, we performed a three phase analysis: x TD Estimation. We applied the Breaking Point Tool on the source code of the two applications, as described in Section 3.4, and obtained data on 8 versions of the two projects. From this analysis, we recorded three main variables: interest (I), normalized principal per LoC (NrP), and breaking point (TD-BP).
x Extraction of Control Variables. We compiled a list of packages, in a random ordering, and asked the developers to assign a Likert-scale score (1)(2)(3)(4)(5) to each one of these packages, with respect to the effort required to maintain them, and the extent to which they consider them sustainable (i.e., need to completely re-write them, due to heavy maintenance effort). The data extraction method for this phase was a questionnaire-based approach.
x Discussion and Interpretation of Obtained Results.
During this phase we conducted a focus group with all developers (i.e., case study participants) and discussed their rationale for ranking specific packages with high sustainability scores, and others with a lower one.

Data Analysis
To answer the research question, we performed a Spearman Rank correlation, between the TD estimated and the control variable. The reason for performing a non-parametric test is that the dataset was not normally distributed, whereas the reason for performing rank correlation instead of actual values correlation is that we are not interested in the predictive power of the metrics (i.e., how well they can predict the values of the sustainability), but on the consistency of the proposed metric. According to the 1061-1998 IEEE Standard for Software Metrics [24], consistency assesses whether there is consistency between the ranks of the quality characteristic and the ranks of the metric under study. Consistency determines if a metric can accurately rank artifacts in terms of quality. The criterion is quantified by the coefficient of rank correlation.

RESULTS
In this section we present the results of the empirical study, through presenting some demographics and subsequently answering the research question.

Study Demographics
In Table 3, we present the average descriptive statistics for the TD estimation variables and the average number of lines per class per version, between versions (a proxy of maintenance effort per version). Furthermore, in Figure 4, we present the distribution of participants' responses with respect to the control variable, i.e., the sustainability of each examined packages. Finally, in Figure 5, we present the evolution of the three TD estimators. In order to fit all estimators in the same graph (since they have different range of values), we normalize all of them against their value in the first of the examined versions (for that version all values equal 1), and then plot in a line chart their evolution.
Based on the results of Figure 4, we observe that the majority of the examined packages have been assigned an average sustainability, whereas the packages with high sustainability are more than those with low sustainability. Another observation that can be made, based on the evolution analysis of TD estimators (see Figure 5) is that Principal is a more stable metric [14], i.e., provides less fluctuations, compared to interest (ranges from 100%-400% of the value of the first version) and the breaking point (ranges from 75% -138% of the first version). Also, we can observe that the interest which increased a lot in the 2 nd examined version is then decreasing along time, suggesting an improvement in the structural quality of the software. When developers were asked if they could explain the reason behind the enhancement of quality along evolution, one of them replied that: "the quality of the code is a top priority for us. We are trying to structure our code in a modular manner, applying well-known patterns for that, like Model-View-Presenter (MVP) 3 ".

Answering the Research Question
As mentioned in Section 4.4, the research question is answered using correlation analysis. The results of the analysis are presented in Table 4. In the table, statistically significant correlations are denoted with grey cell shading. The results of the correlation analysis suggest that both interest and normalized principal are negatively correlated with software artifacts sustainability which is considered an intuitive outcome. However, only the relation between interest and sustainability is strong and statistically significant. By assessing the correlation between TD breaking point and sustainability, we can observe that the relation is statistically significant and the strongest among the three examined TD estimators. This outcome, suggests that the compound metric that we have proposed in this study is effective. This is an interesting finding in the sense that although principal in isolation is not correlated to sustainability, its inclusion in the calculation of TD-BP has increased its assessing power. The main reasons that the developers used for explaining high and low values of sustainability are summarized below: x High Sustainability: 1. "Provision of limited functionality" 2. "Only shows some data that never change" 3. "Presents a static user interface as well" 4. "Low coupling class which manipulates small amount of data" 5. "Obeys to the Single responsibility" 6. "Used to present static details about the app" x Low Sustainability 1. "High Coupling" 2. "Handles feed from different data sources. Needs extra effort to be maintained" 3. "Fundamental class of the app which handles many data and has many responsibilities"

DISCUSSION
In this section, we interpret the case study results and discuss their implications for researchers and practitioners.

Explanation of Obtained Results
According to the results of the study interest (as calculated by the proposed approach) appears to be more sensitive than principal (as calculated by SonarQube) in the sense that it exhibits more fluctuations over the course of the project's evolution. This could be attributed to the fact that principal depends on the number of rule violations that SonarQube identifies, and these violations do not change drastically from one version to the other. On the other hand, interest as calculated by the proposed FITTED approach, takes indirectly into account many structural characteristics of software, as the 'optimum' design is assessed by identifying structurally similar classes. For example, even a change in the control flow structure of a method will be reflected on interest, whereas it could go unnoticed by the principal calculation. Thus, interest calculation is subject even to minor changes in the code leading to a more sensitive measure.
Considering the results of the correlation analysis between principal, interest, breaking point and software artifacts sustainability, it appears that the consideration of the structural characteristics on top of the TD issues identified by SonarQube has been successful. In other words, the combination of interest (which has a statistically significant correlation to sustainability) and principal for the extraction of the TD breaking point, leads to even stronger, and statistically significant correlation between the breaking point and sustainability. It should be noted that in this study sustainability is related to the difficulty of performing maintenance or conversely the need to completely re-write software modules. A module has low sustainability if it undergoes frequent and extensive maintenance and at the same time its TD hinders maintenance. According to the responses of the study typical symptoms of a poor design, such as high coupling and violation of the single responsibility principle contribute to low sustainability. The fact that the proposed TD breaking point captures to a sufficient extent the notion of sustainability as perceived by the developers implies that TD issues combined with structural properties of the code can serve as a valid indicator of maintainability.

Implications for Researchers
Considering the need to assess the sustainability of software modules in an objective and accurate manner, the promising results regarding the correlation between TD-BP and sustainability should be checked against more projects. Given that the case study has been executed on two projects from the mobile domain, extension of the study to projects with different characteristics in terms of application domain, programming language and development process can be very valuable for tuning the calculation of the breaking point. It should be noted that the proposed application of FITTED targets code TD; however, the underlying theory can be extended to other levels such as architecture. Considering the architectural TD has a larger impact on system maintainability than code TD, obtaining the interest of an architectural element based on the notion of 'distance' to similar, 'optimum' elements can remove the existing barriers of effectively calculating a pragmatic TD measure.

Implications for Practitioners
The results of the study and the developers' feedback on the causes of low and high sustainability indicate that the adoption of rather 'classic' well-known practices (such as the use of patterns, conformance to the Single Responsibility Principle, low coupling and high cohesion) can improve sustainability. FITTED confirms that addressing code and design inefficiencies can move the breaking point, i.e. the time at which increased maintenance costs exceed any short-term benefits, further away to the future. The proposed toolset in this study and especially the consideration of structurally similar modules that exhibit a higher quality can be of help to novice software engineers. Assessing principal, interest and breaking point in a relative manner that is by comparing a software module to that of peers can yield feasible opportunities for improvement.
Moreover, the proposed toolset offers a viable way of continuous monitoring of TD critical levels, so as to act as a Quality Gate in the context of contemporary continuous integration practices.

THREATS TO VALIDITY
The results regarding the correlation between interest and breaking point and sustainability are subject to external validity threats meaning that they cannot be generalized to other application domains, programming languages or project sizes. Furthermore, TD principal estimation with the use of SonarQube has to be reported as a possible validity threat. Since the tool does not consider architecture when calculating TD principal, this may impact the precision of the estimation. Moreover, it should be noted that the developers in the study might have perceived the concept of sustainability in different ways. Since sustainability is one of the main variables in the analysis, construct validity threats arise, although an attempt to clarify the concept prior to the collection of results has been made.

CONCLUSION
Principal and Interest are key concepts in the Technical Debt literature as they provide a monetary representation of detectable inefficiencies. However, their value has to be demonstrated by proving their correlation to external software qualities. In this paper we demonstrate the feasibility of instantiating a theoretical framework which allows the calculation of the breaking point, which is a time stamp in which the accumulated interest becomes larger than the principal. The results of a validation in an industrial setting are very promising since they suggest that breaking point is strongly correlated to experts' opinion on the sustainability of modules.