<jupyter_start><jupyter_text>Public Demo of IncQuery Server Jupyter Client Extensions  Preliminaries  Setup IQS Connection Execute the following piece of code to connect to the public IncQuery Server demo instance.If you have non-guest privileges, you may specify your credentials here.<jupyter_code>import iqs_jupyter
iqs = iqs_jupyter.connect(
    address='https://openmbee.incquery.io',
    user='openmbeeguest',
    password='<PASSWORD>'
)<jupyter_output><empty_output><jupyter_text>Select MMS commit to consider Run the next code block to display the commit selector widget, and use it to browse around the MMS repository. When you've had your fun, make sure to leave it in a state where a commit is selected from the _IQS4MMS Demos_ org, as we have made sure to pre-index and load those commits in the IQS.<jupyter_code>commit_selector = iqs.jupyter_tools.mms_commit_selector_widget()<jupyter_output><empty_output><jupyter_text>The following piece of code assigns the Python name `model` to the MMS commit selected above, and checks whether the model is indeed indexed and loaded by IQS, which is required for the rest of the demo<jupyter_code>model = commit_selector.value().to_model_compartment()
if model.is_loaded_by_server(iqs):
    print("We may proceed.")
else: 
    print("Model is not indexed&loaded by IQS, so the next demo steps will not work.")
    print(" (Unfortunately, guest users are not allowed to control model indexing.)")
    print("Please select another model from the 'IQS4MMS Demos' org.")<jupyter_output><empty_output><jupyter_text>Model validation  Perform validation checks with custom validation rules (progress with indexed models only) Request and display a validation report for the selected model<jupyter_code>validation_report = iqs.validation.validate_model_compartment(model)
validation_report<jupyter_output><empty_output><jupyter_text>Visualize custom validation reports via Pandas dataframes and Plot.ly The following block renders results in Pandas dataframe format, ready for complex client-side manipulation:<jupyter_code>validation_report_df = validation_report.to_data_frame()<jupyter_output><empty_output><jupyter_text>Setting up Cufflinks for Pandas/Plot.ly visualization:<jupyter_code>import cufflinks as cf
cf.go_offline()<jupyter_output><empty_output><jupyter_text>Preprocess results using Pandas and visualize the output using Plot.ly; this is handy for generating many kinds of custom validation reports.First, here is a diagram showing the number of violations found for each validation rule:<jupyter_code>validation_report_rulewise = validation_report_df.groupby(by=['constraint_element_name','severity']).size().unstack(level=-1, fill_value=0)
validation_report_rulewise['total'] = validation_report_rulewise.apply(sum, axis=1)
validation_report_rulewise = validation_report_rulewise.sort_values(by='total', ascending=False).drop('total', axis=1)
validation_report_rulewise.iplot( 
    kind='bar', barmode='stack', colors = iqs_jupyter.validation_color_scale,
    filename='cufflinks/categorical-bar-chart/frequent_diagnostic_rules',
    yTitle='Number of Diagnostics per Diagnostic Rule', title='Diagnostic Report',
)<jupyter_output><empty_output><jupyter_text>Next, we do some precomputation with Pandas and plot a chart of violating elements against their total number of violations; if there are multiple severities (e.g. warnings and errors), the bars will be subdivided (stacked bar chart).<jupyter_code>validation_report_elementwise = validation_report_df.groupby(by=['matching_element_relative_id','severity']).size().unstack(level=-1, fill_value=0)
validation_report_elementwise['total'] = validation_report_elementwise.apply(sum, axis=1)
cols_sorted = ['total'] + [severity for severity in iqs_jupyter.validation_diagnostics_ranking if severity in validation_report_elementwise.columns.to_list()]
validation_report_elementwise = validation_report_elementwise.sort_values(by=cols_sorted, ascending=False).drop('total', axis=1)
validation_report_elementwise.iplot( 
    kind='bar', barmode='stack', colors = iqs_jupyter.validation_color_scale,
    filename='cufflinks/stacked-bar-chart/diagnostics_per_element',
    yTitle='Number of Diagnostics per Violating Element', title='Detailed Diagnostic Report',
)<jupyter_output><empty_output><jupyter_text>Coverage report using custom model queries  Define and register custom ad-hoc queries The custom model queries defined below will discover multiple forms of traceability to _Requirement_ elements, determine the coverage of _Block_ elements by such traceability links, and aggregate coverage metrics for _Package_s containing these _Block_s.<jupyter_code>coverage_query_package = "iqs4mms.demo.coverage"
coverage_query_main = "iqs4mms.demo.coverage.packageCoverage"
coverage_query_code = '''

// SECTION 1: main query

/* 
 * Associates a UML package with 
 *  - the total number of transitively contained SysML Blocks, and 
 *  - the number of strongly and weakly covered blocks among them
 */
pattern packageCoverage(
    pack: Package, 
    totalBlocks: java Integer, 
    stronglyCovered: java Integer, 
    weaklyCovered: java Integer
) {
    totalBlocks     == count find blockInPackage(_, pack);
    stronglyCovered == count find stronglyCoveredBlockInPackage(_, pack);
    weaklyCovered   == count find weaklyCoveredBlockInPackage(_, pack);
}


// SECTION 2: custom in-house definitions for strong and weak coverage

/* 
 * Identifies elements that are strongly covered by a requirement.
 * This definition may be customized according to in-house concept of strong coverage.
 */
pattern stronglyCovered(element: NamedElement) {
    find sysml.Requirement_SatisfiedBy(_, element);
} or {
    find sysml.Requirement_VerifiedBy(_, element);
} or {
    find sysml.Requirement_TracedTo(_, element);
}
/* 
 * Identifies elements that are NOT strongly covered, 
 * but are reachable from a strongly covered element 
 * using one or more custom propagation steps
 */
pattern weaklyCovered(element: NamedElement) {
    find stronglyCovered(otherElement);
    find coveragePropagates+(otherElement, element);
    neg find stronglyCovered(element);
}
/* 
 * Defines custom coverage propagation steps, where 
 * coverage of a 'from' element automatically implies weak coverage of a 'to' element.
 * This definition may be customized according to in-house propagation rules.
 */
pattern coveragePropagates(from: NamedElement, to: NamedElement) {
    // from container to contained part
    Property.owner(part, from);
    Property.aggregation(part, ::composite);
    TypedElement.type(part, to);
} or { 
    // from general superclassifier / block to specific block
    Generalization.general(gen, from);
    Generalization.specific(gen, to);
}

// SECTION 3: helper queries to assemble a report on (transitive) package contents

/* 
 * Associates SysML Blocks with UML packages directly or indirectly containing them.
 */
pattern blockInPackage(block: Class, pack: Package) {
    find sysml.Block(block, _);    
    Element.owner+(block, pack);
}
/* 
 * Associates strongly covered Blocks with UML packages directly or indirectly containing them.
 */
pattern stronglyCoveredBlockInPackage(block: Class, pack: Package) {
    find stronglyCovered(block);    
    find blockInPackage(block, pack);
}
/* 
 * Associates weakly covered Blocks with UML packages directly or indirectly containing them.
 */
pattern weaklyCoveredBlockInPackage(block: Class, pack: Package) {
    find weaklyCovered(block);    
    find blockInPackage(block, pack);
}


'''<jupyter_output><empty_output><jupyter_text>Queries need to be registered on the server before they can be evaluated. _Note: you need elevated privileges to be able to register new queries; for the guest user in this demo, the next part will only work if the query has already been registered; otherwise you will receive a '403 Forbidden' error._<jupyter_code>if coverage_query_main not in iqs.queries.list_queries().viatra.query_fq_ns: # skip if already registered
    try: 
        iqs.queries.register_queries_plain_text(coverage_query_code, query_package=coverage_query_package)
    except: 
        print("Query registration not available as guest user; please try again in a few minutes")
else:
    print("Query is already registered; proceed")<jupyter_output><empty_output><jupyter_text>To verify, see which queries are registered and ready for execution:<jupyter_code>iqs.queries.list_queries().viatra<jupyter_output><empty_output><jupyter_text>Execute pre-registered query and process results Before executing the query, we make sure that it is served as a standing query. This means that it is evaluated once, and then results are provided instantaneously for any subsequent requests.This might take a few seconds if you are the first user to run the following command on this IQS server instance and this model. For subsequent invocations, even by other users, this preparation will return immediately.<jupyter_code>from iqs_jupyter import schema
display(iqs.query_execution.prepare_standing_queries_on_model_compartment(
    schema.QueryFQNListWithModelCompartment(
      model_compartment = model,
      query_fq_ns = [coverage_query_main]
    )))<jupyter_output><empty_output><jupyter_text>We will now request the results of the previusly registered query.<jupyter_code>qResults = iqs.query_execution.execute_query_on_model_compartment(
    schema.ExecuteQueryOnCompartmentRequest(
      model_compartment = model,
      query_fqn = coverage_query_main,
      query_mode = "standing"
    ))
qResults<jupyter_output><empty_output><jupyter_text>Visualize query results using Pandas dataframes and Plot.ly Setting up Cufflinks for Pandas/Plot.ly visualization:<jupyter_code>import cufflinks as cf
cf.go_offline()<jupyter_output><empty_output><jupyter_text>The following block query renders results in Pandas dataframe format, ready for complex client-side preprocessing:<jupyter_code>qResults_df = qResults.to_data_frame().query('totalBlocks != 0')
qResults_df['pack'] = qResults_df['pack'].apply(lambda element: element.relative_element_id)
qResults_df.set_index('pack', inplace=True)
qResults_df['nonCovered'] = qResults_df['totalBlocks'] - qResults_df['stronglyCovered'] - qResults_df['weaklyCovered']<jupyter_output><empty_output><jupyter_text>Visualize the output using Plot.ly:<jupyter_code>qResults_df.sort_values(by='totalBlocks', ascending=False).drop('totalBlocks', axis=1).iplot(
    kind='bar', barmode='stack', filename='cufflinks/categorical-bar-chart/coverage',
    yTitle='Blocks in Package Traced to Requirements', title='Coverage Report',
)<jupyter_output><empty_output><jupyter_text>Extract individual model elements, execute queries with parameter bindings Descriptors of individual model elements can be extracted into Python variables from query results...<jupyter_code>first_result_element = qResults.to_list_of_matches()[0]['pack']
first_result_element<jupyter_output><empty_output><jupyter_text>...alternatively, model element descriptors can be directly constructed using element identifiers:<jupyter_code>some_element = model.get_element_in_compartment_by_id("_18_0_2_baa02e2_1421374069634_721435_78319") 
some_element<jupyter_output><empty_output><jupyter_text>Model elements or simple values can be used as parameter bindings to restrict the requested results:<jupyter_code>from iqs_jupyter import binding
qResults_restricted = iqs.query_execution.execute_query_on_model_compartment(
    schema.ExecuteQueryOnCompartmentRequest(
        model_compartment = model,
        query_fqn = coverage_query_main,
        query_mode = "standing",
        parameter_binding = binding(pack=first_result_element)
    ))
qResults_restricted<jupyter_output><empty_output><jupyter_text>Extended features via direct MMS connection First, connect to the MMS server.<jupyter_code>import iqs_jupyter # if not already imported
mms = iqs_jupyter.MMSClient(
    address = "https://mms.openmbee.org/alfresco/service",
    user = "openmbeeguest",
    password= "<PASSWORD>"
)<jupyter_output><empty_output><jupyter_text>Browse individual model elements and properties (requires connection to MMS) Display element information either for an element at specified commit and with specified id, or for a previously obtained model element handle. Browse through attributes and element references. Note: some element references are not shown due to MMS limitations.<jupyter_code>element_info_widget = mms.helpers.show_element_info_widget(
 org_id     = '9ff6af30-af8a-4f9d-a26b-499010ba5b6e',
 project_id = 'PROJECT-0e791c0e-16fe-422f-8f85-462ab035ce99',
 ref_id     = 'master',
 commit_id  = '0e4c90e2-ee9d-4d7e-9d90-e7fb7bb2c0c8',
 element    = '_18_0_2_baa02e2_1421374069634_721435_78319'
)
# alternatively, use the already available model element handle
# element_info_widget = mms.helpers.show_element_info_widget(some_element)
element_info_widget.display()<jupyter_output><empty_output><jupyter_text>The element currently selected ('opened') in the widget can be obtained for further processing.<jupyter_code>element_info_widget.selected_element()<jupyter_output><empty_output><jupyter_text>Generate HTML report and upload it to MMS Below we export an HTML report from the validation results table. Interactive plots can also be exported, though not usable on MMS due to Javascript restrictions.<jupyter_code># div_frequent_diagnostic_rules = validation_report_df.groupby(by=['constraint_element_name','severity']).size().unstack(level=-1).iplot( 
#     kind='bar', barmode='stack', colors = iqs_jupyter.validation_color_scale,
#     yTitle='Number of Diagnostics per Diagnostic Rule', title='Diagnostic Report',
#     asFigure=True
# ).to_html(full_html=False, include_plotlyjs = 'cdn')

validation_report_html = '''
<h4>Validation Report</h4>
<p>{}</p>
<p>{}</p>
'''.format(
    validation_report.diagnostics._repr_html_(),
    validation_report._repr_html_(),
)<jupyter_output><empty_output><jupyter_text>The generated HTML report can be directly uploaded to MMS. Here we are uploading it to a sandbox document that is publicly writeable.<jupyter_code>target_project = 'PROJECT-3cc8325d-f955-424f-8410-4dcdbcae621e'
target_ref = 'master'
target_element = '_hidden_MMS_1575982749811_a6fca1fb-afe8-4b4e-bfdf-67cefbf01aff_pei'
mms.element.post_elements(target_project, target_ref,{"elements": [
    {"id": target_element, "documentation": validation_report_html}
]})<jupyter_output><empty_output><jupyter_text>The uploaded report is available at [MMS (IQMAS-Jupyter-Example)](https://mms.openmbee.org/alfresco/mmsapp/mms.html/projects/PROJECT-3cc8325d-f955-424f-8410-4dcdbcae621e/master/documents/MMS_1575971180049_ea2f29db-359e-4a46-aa80-2f2fac3ea208/views/MMS_1575971180049_ea2f29db-359e-4a46-aa80-2f2fac3ea208).  Sandbox  Extra section for privileged usersDo not forget to specify your privileged credentials at the top of the notebook, in the first code cell  Repository management, indexing commits Force the server to refresh its knowledge of commits in the repository:<jupyter_code>iqs.mms_repository.update_mms_repository()<jupyter_output><empty_output><jupyter_text>Index another model from the repository, and then load the index into server memory:<jupyter_code>iqs.persistent_index.index_model_compartment(model)
iqs.in_memory_index.load_model_compartment(model)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08194056057748586, 'fraction_numerical': 0.01690836964297327, 'mean_word_length': 6.389716482460355, 'pattern_counts': {'":': 3, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '26', 'dir': 'jupyter-structured-clean-dedup', 'id': '231954', 'license': 'Apache-2.0', 'n_tokens_mistral': 4821, 'n_tokens_neox': 4550, 'n_words': 1273, 'path': 'example-notebook-home/iqs-demo-mms.ipynb', 'repo_name': 'IncQueryLabs/incquery-server-jupyter', 'size': '15367'}