<jupyter_start><jupyter_text>LinksCallbacks- https://dash.plotly.com/sharing-data-between-callbacks- https://dash.plotly.com/advanced-callbacksHTML- https://dash.plotly.com/dash-core-components- https://dash.plotly.com/dash-html-componentsPlots- https://plotly.com/python/reference/scatter/scatter-   Fixme really:- The user can insert wrong units and does not know. and we also first know when trying to simulate a neuron. This error has to be catched and the original Brian message parsed to the user. This will happen if the user clicks on fit-model- if for some reason brian can not compute the traces, i.e. very large nan values occur or whatever. we have to know this and then say that soemthing gone wrong and ideally get the error message that brian had during computation. somehow need the logs brom brian (Tracegenerator has to collect them) to get infos. (actaully this si semething needed to be returend as an eeror fofrom trace gen so fitter is not called and fitter also has to check that)- the above mighjt be solved if the user choose another integration method, therefore build something that the user can specifie the integration method(already implemente din variable iunterpeter, just need to be userdefinable)- since no dynamic callbacks maybe one can just use **r to retun the whole array and have shorter return statements- inside the code are many fixmes :P<jupyter_code>None.strip()
type('a') is not str<jupyter_output><empty_output><jupyter_text>import timefrom jupyter_dash import JupyterDashimport dash_core_components as dccimport dashfrom dash.dependencies import Input, Output, Statefrom dash.exceptions import PreventUpdateimport plotly.express as pximport plotly.graph_objects as gofrom fitter.trace_generator import *from fitter.loihi_fitter import *from brian2 import * from brian2.units.allunits import *  get everythingfrom widget.html import *from widget.callbacks import *app_storage = {} Add brian units to storageapp_storage['brian_units'] = { unit: eval(unit) for unit in [*units.allunits.__all__, *units.__all__] }app = JupyterDash(    __name__,    external_stylesheets=[dbc.themes.BOOTSTRAP])app.layout = html.Div([    html.H1('Loihi Fitter'),    wrapper]) Add callbacks to appinit_callbacks(app, app_storage) Run serverapp.run_server(mode="jupyterlab", port=8899, debug=True)<jupyter_code>app._terminate_server_for_port("localhost", 8899)
#----------------------------------------------------------------
#                       Global Variables
#----------------------------------------------------------------
# generate dictionary with all brian units
unit_dict = {}
for unit in units.__all__:
    unit_dict[str(unit)] = eval(unit)
#for unit in units.allunits.__all__:
#    unit_dict[str(unit)] = eval(unit)


    
#----------------------------------------------------------------
#                      DashApplication
#----------------------------------------------------------------

from dash.html import *


# since dash has no dynamic callbacks the workaround is to define everything in the beginning, hide it and only make it visible if needed.
# idea is:
# define 3 divs: names, values, SIUnit
# update divs differently i.e. add new html for the names in the names div dynamically, unhide the other ones based on how many identifiers
n_identifiers = 10
html_identifiers= []
for i in range(n_identifiers):
    name = html.Div (id='identifier_' + str(i) + '_name', children=[])
    value= dcc.Input(id='identifier_' + str(i) + '_value', type='number')
    unit = dcc.Input(id='identifier_' + str(i) + '_unit', type='text')
    
    wrapper = html.Div(id='identifier_' + str(i) + '_wrapper',
                       children=[name, value, unit],
                       style={'display': 'none'})
    html_identifiers.append(wrapper)

# add a div which can be shown if there is an error whith parsing the equation
html_identifiers.append(html.Div (id='identifier_error', children=[]))

user_input_identifiers = html.Div(id='identifiers', className='wrapper', children=html_identifiers)

# ouput from the fiter will go here
output_fitter = html.Div(id='output', children=[
    html.Div(id='out-fitter-error', children=[]),
    dcc.Graph(id="out-fitter-plot"),
    html.Br(),
    html.Button('fit-model', id='fit-model', n_clicks=0, style={'display': 'none'})

])

# user specific storage
# FIXME:
# - Any given output can only have one callback that sets it.
#   To resolve this situation, try combining these into
#   one callback function, distinguishing the trigger
#   by using `dash.callback_context` if necessary.
# - currently just using two storages one for the eqaution and identifiers and one for the results of the trace generator and fitter
# only store in memory so if page reloads everything will be reset
app_storage_eq = dcc.Store(id='app_storage_eq', storage_type='memory')
app_storage_fit= dcc.Store(id='app_storage_fit', storage_type='memory')

#-----------------
#     dashApp
#-----------------
app = JupyterDash(__name__)
app.layout = html.Div([
    html.H1('<NAME>'),
    user_input_generell,
    user_input_equation,
    user_input_identifiers,
    output_fitter,
    app_storage_eq,
    app_storage_fit
])

#-----------------
#   callbacks
#-----------------

# parse equation callback
# actually I LMAO !!! this hack with the Outputs predefined and therefore the return values is the most ugly programming :D
# is this really the ONLY solution?????
@app.callback(
    ([Output("identifier_{}_wrapper".format(_), "style") for _ in range(n_identifiers)] + 
     [Output("identifier_{}_name".format(_), "children") for _ in range(n_identifiers)] + 
     [Output("fit-model", "style"), Output("identifier_error", "children")] + 
     [Output('app_storage_eq', 'data')])
    ,
    [
        Input("submit-equation", 'n_clicks'),
        State("equation", 'value'),
        State("output_variable", 'value'),
        State("output_variable_unit", 'value'),
        State("input_variable", 'value'),
        State("input_variable_unit", 'value'),
        State("resting_variable", 'value'),
        State("threshold", 'value'),
        State("threshold_unit", 'value'),
        State("weight", 'value'),
        State("weight_unit", 'value'),
        State("dt", 'value'),
        State("dt_unit", 'value'),
        State("runtime", 'value'),
        State("runtime_unit", 'value'),
        State("app_storage_eq", 'data')
    ]
)
def parse_equation(n_clicks,
                   equation,
                   output_variable,
                   output_variable_unit,
                   input_variable,
                   input_variable_unit,
                   resting_variable,
                   threshold,
                   threshold_unit,
                   weight,
                   weight_unit,
                   dt,
                   dt_unit,
                   runtime,
                   runtime_unit,
                   app_storage
                  ):
    # do nothing in the none callback
    if n_clicks == 0 or None:
        raise PreventUpdate
        #return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],r[21],app_storage
    # ge tapp_storage, default to {} if no data present
    app_storage = app_storage or {}
    
    # define the return variables as array to have it shorter
    output_style = [{'display': 'none'}]*n_identifiers
    output_name  = ['']*n_identifiers
    output_addi  = [{'display': 'none'}, ''] # this is additional output (fit_button_style, error)
    r = output_style + output_name + output_addi
    
    # parse input
    # equation
    if isinstance(equation, str) == False:
        return  r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'equation is not a String',app_storage
    
    # output variable
    #output_variable_tuple = 
    #if isinstance(output_variable, str) == False:
    #    return 'output variable is not a String'
    
    if str(output_variable_unit) not in unit_dict:
        return  r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'output variable unit is not valid. you entered {0}'.format(str(output_variable_unit)),app_storage
    app_storage['output_variable'] = (output_variable, output_variable_unit)
        
    # input variable
    #if isinstance(input_variable, str) == False:
    #    return 'input variable is not a String'
    
    if str(input_variable_unit) not in unit_dict:
        return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'input variable unit is not valid. you entered {0}'.format(str(input_variable_unit)),app_storage
    app_storage['input_variable'] = (input_variable, input_variable_unit) 
        
    # resting variable
    if isinstance(resting_variable, str) == False:
        return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'resting variable is not a String',app_storage
    app_storage['resting_variable'] = resting_variable
    
    # FIXME:
    # - check that all the following inputs can be casted to float and catch error if not i.e. prevent user to input a string (this should be done because of input_type=number?)
    #try:
    #    float(input)
    #except ValueError:
    #    return 'enter a float'
        
    # threshold
    if str(threshold_unit) not in unit_dict:
        return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'threshold unit is not valid. you entered {0}'.format(str(threshold_unit)),app_storage
    app_storage['threshold'] = (float(threshold), threshold_unit)
        
    # weight
    if str(weight_unit) not in unit_dict:
        return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'weight unit is not valid. you entered {0}'.format(str(weight_unit)),app_storage
    app_storage['weight'] =(float(weight), weight_unit) 
        
    # dt
    # TODO
    # - maybe pre-define a dt and leave this optional?
    if str(dt_unit) not in unit_dict:
        return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'dt unit is not valid. you entered {0}'.format(str(dt_unit)),app_storage
    app_storage['dt'] =(float(dt), dt_unit)
        
    # runtime
    # FIXME:
    # - prevent a runtime which is to long, how to define to long? 1000ms ?
    # - what if the decay on loihi is very very slow how long would a simulation take to go back to zero?
    # - maybe pre-define a runtime and leave this optional?
    if str(runtime_unit) not in unit_dict:
        return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'runtime unit is not valid. you entered {0}'.format(str(runtime_unit)),app_storage
    app_storage['runtime'] = (int(runtime), runtime_unit)
    
    
    # make variable interpreter and get identifiers
    variable_interpreter = VariableInterpreter(equation)
    
    identifiers = variable_interpreter.get_identifier_names()
    
    # check parsing success of eq, return error if
    if isinstance(identifiers, Exception):
        return  r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],str(identifiers),app_storage
    
    # make list
    identifiers = list(identifiers)
    
    # before storage check if the resting variable in identifiers
    if app_storage['resting_variable'] != '' and app_storage['resting_variable'] not in identifiers:
        return  r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],'resting variable '+str(app_storage['resting_variable'])+'not int '+str(identifiers),app_storage
    
    # update app storage, save identifiers, and equation, its clear now equation can be parsed.
    app_storage['equation'] = equation
    app_storage['identifiers'] = identifiers
    
    # everything allright update the return array r accordingly
    for i, ident in zip(range(len(identifiers)), identifiers):
        output_style[i] = {'display': 'inline'}
        output_name[i]  = ident
    
    # make fit model button visible
    output_addi  = [{'display': 'inline'}, '']
    r = output_style + output_name + output_addi # reconstruct r 
    
    return r[0],r[1],r[2],r[3],r[4],r[5],r[6],r[7],r[8],r[9],r[10],r[11],r[12],r[13],r[14],r[15],r[16],r[17],r[18],r[19],r[20],r[21],app_storage

# fit model callback
# FIXME: 
# - if this is called everything else on the webpage should be reset
@app.callback(
    [
        Output("out-fitter-error", 'children'),
        Output("out-fitter-plot", 'figure'),
        Output("app_storage_fit", 'data')
    ]
    ,
    ([Input("fit-model", 'n_clicks')] +
     [State("identifier_{}_value".format(_), "value") for _ in range(n_identifiers)] + 
     [State("identifier_{}_unit".format(_), "value") for _ in range(n_identifiers)] + 
     [State("app_storage_eq", 'data'), State("app_storage_fit", 'data')])
)
def fit_model(*vals):
    # prevent Non callbacks
    if vals[0] == 0 or vals[0] == None:
        raise PreventUpdate
        #return vals[0]
        
    # TODO shift app_storage before identifiers
    # - therefore also shift the indices of the read out of the values below! i.e. identifier_values = -.....
    # get app_storage, default to {} if no data present
    s_index = 1+n_identifiers*2 # the storage location in the inputs 1 = button, every identifier has value and unit => n_ident*2
    app_storage_eq = vals[s_index] or {}
    app_storage_fit = vals[s_index+1] or {}
    

    
    # check if we have necessary data
    if ('equation' not in app_storage_eq or
        'identifiers' not in app_storage_eq or
        'output_variable' not in app_storage_eq or
        'input_variable' not in app_storage_eq or
        'threshold' not in app_storage_eq or
        'weight' not in app_storage_eq or
        'dt' not in app_storage_eq or
        'runtime' not in app_storage_eq):
        return "No equation or missing identifiers or missing necessary variables, can't fit model.",None,app_storage_fit
    
    
    # get user inputs, this will include hidden, therefore empty inputs
    identifier_values= vals[1:n_identifiers+1] # first input is the button
    identifier_units = vals[1+n_identifiers:(n_identifiers*2)+1] # the button needs to be added
    
    # assuming identifiers having the same order in the input we can use the known identifier list for the names.
    eq_identifiers = list(app_storage_eq['identifiers'])
    ident_dict = {}
    for i, ident in zip(range(len(eq_identifiers)),eq_identifiers):
        # check if value provided
        if identifier_values[i] == '' or identifier_units[i] == '':
            return 'please enter values for all identifiers and their units',None,app_storage_fit
        
        # check whether we have a valid unit
        if str(identifier_units[i]) not in unit_dict:
            return str(ident) + ' has not a valid unit. Got ' + str(identifier_units[i]),None,app_storage_fit
        
        # add identifier with unit to the dict
        # FIXME:
        # - check that identifier_values is parsable to float
        ident_dict[str(ident)] = (float(identifier_values[i]), unit_dict[str(identifier_units[i])])
    
    # now for the fitting
    # setting up the variables
    variable_interpreter = VariableInterpreter(app_storage_eq['equation'])
    
    # these shall be ('var_name', SIUnit)
    # we get e.g {'v': (10, mV)} and have to translate the unit to a python object
    
    # TODO make this maybe shorter?
    #names = ['input_variable', 'output_variable', ...]
    #varss   = [(app_storage_eq[v][0], unit_dict[app_storage_eq[v][1]]) for v in names]
    
    input_variable   = (app_storage_eq['input_variable'][0], unit_dict[app_storage_eq['input_variable'][1]])
    output_variable  = (app_storage_eq['output_variable'][0], unit_dict[app_storage_eq['output_variable'][1]])
    resting_variable =  app_storage_eq['resting_variable'] # special is only string
    threshold        = (app_storage_eq['threshold'][0], unit_dict[app_storage_eq['threshold'][1]])
    weight           = (app_storage_eq['weight'][0], unit_dict[app_storage_eq['weight'][1]])         
    dt               = (app_storage_eq['dt'][0], unit_dict[app_storage_eq['dt'][1]])
    runtime          = (app_storage_eq['runtime'][0], unit_dict[app_storage_eq['runtime'][1]])
    variable_interpreter.set_variables(input_variable, output_variable, resting_variable, threshold, weight, dt, runtime)
    
    
    
    #set identifiers
    variable_interpreter.set_identifiers(ident_dict)
    
    
    # make trace generator and generate traces
    trace_generator = TraceGenerator(variable_interpreter)
    
    s_time = time.time()
    trace_generator.generate_traces()
    e_time = time.time()
    generate_traces_time = e_time-s_time

    # make Loihi fitter and fit model
    loihi_fitter = LoihiFitter(trace_generator)
    
    s_time = time.time()
    loihi_fitter.fit_model()
    loihi_fitter.fit_thres_weight()
    e_time = time.time()
    fit_model_time = e_time-s_time

    # TODO Make this maybe optional
    s_time = time.time()
    loihi_fitter.fit_mantissa_table()
    e_time = time.time()
    fit_mantissa_table_time = e_time-s_time
    
    # make plot
    x_axis = np.arange(trace_generator.get_simulation_steps())

    # normalize trace where threshold is fixed at 1, [trace]/threshold
    b_trace = trace_generator.threshold_normalize()  # get_trace
    l_trace = loihi_fitter.threshold_normalize()  # get_trace
    thresh  = np.array([1]*int(trace_generator.get_simulation_steps()))

    # FIXME:
    # - use good colors ;)
    # - make one line bigger then the other
    Loihi_color = '#1D33FF'
    Brian_color = '#FFD679'
    Thresh_color= '#8E00FF'
    t0 = go.Scatter(x=x_axis, y=b_trace, name='input_model', mode='lines', marker_color=Brian_color)
    t1 = go.Scatter(x=x_axis, y=l_trace, name='Loihi_fitted', mode='lines', marker_color=Loihi_color)
    t2 = go.Scatter(x=x_axis, y=thresh, name='Threshold', mode='lines', marker_color=Thresh_color)

    layout = go.Layout(
        paper_bgcolor='rgba(0,0,17,0)', # everything around the plot
        plot_bgcolor='rgba(0,0,13,0)'   # the plot itself
    )
    fig = go.Figure(layout=layout)

    fig.add_traces(t0)
    fig.add_traces(t1)
    fig.add_traces(t2)
    
    fig.update_layout(
        title="Threshold normalized traces",
        xaxis_title="runtime",
        yaxis_title="percent thresh",
        legend_title="Trace legend",
        font=dict(
            family="Courier New, monospace",
            size=14,
            color="RebeccaPurple"
        )
    )
    
    loihi_values = str(loihi_fitter)
    output_infos = 'generate traces: ' +   str(generate_traces_time)
    output_infos+= 'fit model: ' + str(fit_model_time)
    output_infos+= 'fit mant_table: ' + str(fit_mantissa_table_time)
    output_infos+= loihi_values
    
    # FIXME:
    # - save the results for further use in the dashboard etc.
    
    
    return output_infos,fig,app_storage_fit
    # get and set identifiers
    # make traces
    # fit model
    # plot result
    # return model fitted values
    
    
#app.run_server(mode="inline", port=9998, debug=True)

# only works in jupyterlab and only from time to tiime :P
app.run_server(mode="jupyterlab", port=8899, debug=True)

# Lett the dash app run in external mode (you can open it with the stated weAdress)
#app.run_server(mode="external", host='localhost', port=9999, debug=False)
app._terminate_server_for_port("localhost", 8899)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14856711915535445, 'fraction_numerical': 0.024635495223730517, 'mean_word_length': 3.4113994233754714, 'pattern_counts': {'":': 0, '<': 10, '<?xml version=': 0, '>': 11, 'https://': 5, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '269497', 'license': 'MIT', 'n_tokens_mistral': 6668, 'n_tokens_neox': 6433, 'n_words': 1581, 'path': 'Loihi_fitter_dash.ipynb', 'repo_name': 'sagacitysite/loihi-fitter-dashboard', 'size': '19896'}