<jupyter_start><jupyter_text>CWPK \60: Adding a SPARQL Endpoint - Part II=======================================Finally Getting a Working Instance--------------------------Yesterday's installment of [*Cooking with Python and KBpedia*](https://www.mkbergman.com/cooking-with-python-and-kbpedia/) presented the first part of this two-part series on developing a [SPARQL](https://en.wikipedia.org/wiki/SPARQL) endpoint for KBpedia on a remote server. This concluding part picks up with **step 7** in the stepwise approach I took to complete this task. At the outset I thought it would progress rapidly: After all, is not SPARQL a proven query language with central importance to [knowledge graphs](https://en.wikipedia.org/wiki/Knowledge_graph)? But, possibly because our focus in the series is [Python](https://en.wikipedia.org/wiki/Python_(programming_language)), or perhaps for other reasons, I have found a dearth of examples to follow regarding setting up a Python SPARQL endpoint (there are some resources available related to [REST](https://en.wikipedia.org/wiki/Representational_state_transfer) [APIs](https://en.wikipedia.org/wiki/API)).The first six steps in yesterday's installment covered getting our environment set up on the remote Linux server, including installing the Web framework [Flask](https://en.wikipedia.org/wiki/Flask_(web_framework)) and creating a virtual environment. We also presented the Web page form design and template for our SPARQL query form. This second part covers the steps of tieing this template form into actual endpoint code, which proved to be simple in presentation but exceeding difficult to formulate and debug. Once this working endpoint is in hand, I next cover the steps of giving the site an external URL address, starting and stopping the service on the server, and packaging the code for GitHub distribution. I conclude this two-part series with some lessons learned, some comments on the use and relevance of [linked data](https://en.wikipedia.org/wiki/Linked_data), and point to additional documentation.  Step-wise Approach (con't)We pick up our step-wise approach here.**7. Tie SPARQL Form to Local Instance**So far, we have a local instance that works from the command line and an empty SPARQL form. We need to relate thest two pieces together. In the last installment, I noted two SPARQL-related efforts, [pyLDAPI](https://github.com/RDFLib/pyLDAPI) (and its [GNAF](https://github.com/CSIRO-enviro-informatics/gnaf-dataset/blob/master/view/templates/page_sparql.html) example) and [adhs](https://github.com/nareike/adhs/blob/master/templates/sparql.html). I could not find working examples for either, but I did consult their code frequently while testing various options.Thus, unlike many areas throughout this **CWPK** series, I really had no working examples from which to build or modify our current SPARQL endpoint needs. While the related efforts above and other examples could provide single functions or small snippets, possibly as use for guidance or some modification, it was pretty clear I was going to need to build up the code step-by-step, in a similar stepwise manner to what I was following for the entire endpoint. Fortunately, as described in **step 6**, I did have a starting point for the Web page template using the GNAF example.From a code standpoint, the first area we need to address is to convert our example start-up stub, what was called test_sparql.py in the [**CWPK 58**](https://www.mkbergman.com/2407/cwpk-58-setting-up-a-remote-instance-and-web-page-server/) installment, to our main application for this endpoint. We choose to call it cowpoke-endpoint.py in keeping with its role. We will build on the earlier stub by adding most of our import and Flask-routing ('@app.route("/")', for example) statements, as well as the initialization code for the endpoint functions. We will call out some specific aspects of this file as we build it.The second coding area we need to address is how to tie the text areas in our Web form template to the actual Python code. We will put some of that code in the template and some of that code in cowpoke-endpoint.py governing getting and processing the SPARQL query. There is a useful pattern for how to relate templates to Python code via what might be entered into a text area from [StackOverflow](https://stackoverflow.com/questions/37345215/retrieve-text-from-textarea-in-flask). Here is the code example that should be put within the governing template, using the important {{ url_for('submit') }}:&lt;form action="{{ url_for('submit') }}" method="post">    &lt;textarea name="text">    &lt;input type="submit">&lt;/form>and here is the matching code that needs to go into the governing Python file:from flask import Flask, request, render_templateapp = Flask(__name__)@app.route('/')def index():    return render_template('form.html')@app.route('/submit', methods=['POST'])def submit():    return 'You entered: {}'.format(request.form['text'])Note that file names, form names and routes all need to be properly identified and matched. Also note that imports need to be complete. Further notice in the file listing below that we modify the return statement. We also repeat this form related to the SPARQL results text area.One of the challenging needs in the code development was working with a remote instance, as opposed to local code. I was also now dealing with a Linux environment, not my local Windows one. After much trial-and-error, which I'm sure is quite familiar to professional developers working in a client-server framework, I learned some valuable (essential!) lessons:1. First, with my miniconda approach and its minimal starting Python basis, I needed to check every new package import required by the code and check whether it was already in the remote instance. The conda list command is important here to first check whether the package is already in the Python environment or not. If not, I would need to find the proper repository for the package and install it per the instructions in [**CWPK 58**](https://www.mkbergman.com/2407/cwpk-58-setting-up-a-remote-instance-and-web-page-server/)2. I needed to make sure that the permission (Linux chmod) and ownership (Linux chown settings were properly set on the target directories for the remote instance such that I could use my SSH-based file transfer program ([WinSCP](xxx) in my case; [Filezilla](xxx) is another leading option). I simply do not do enough Linux work to be comfortable with remote editors. SSH transfer would enable me to work on the developing code in my local Windows instance3. I needed to get basic templates working early, since I needed Web page targets for where the outputs or traces of the running code would display 4. I needed to restart the Apache2 server whenever there was a new code update to be tested. This resulted in a fairly set workflow of edit &rarr; upload &rarr; re-start Apache &rarr; call up remote Web template form (e.g., http://xx.xxx.xxx.xxx/sparql) &rarr; inspect trace or logs &rarr; rinse and repeat5. Be attentive to and properly [set content types](https://stackoverflow.com/questions/11773348/python-flask-how-to-set-content-type), since we are moving data and results from Web forms to code and back again. Content header information can be tricky, and one needs to use [cURL](https://en.wikipedia.org/wiki/CURL) or [wget](https://en.wikipedia.org/wiki/Wget) (or [Postman](https://www.postman.com/), which is often referenced, but I did not use). One way to inspect headers and content types is in the output Web page templates, using this code:  req = request.form  print(req) 6. In HTML forms, use the &lt; code for the left angle bracket symbol (used in SPARQL queries to denote a URI link), otherwise the link will not display on the Web page since this character is reserved7. Used the standard [W3C validator](https://validator.w3.org/i18n-checker/check) when needing to check encodings and Web addresses8. Be **extremely attentive** to the use of tabs v white spaces in your Python code. Get in the habit of using spaces only, and not tabbing for indents. Editors are more forgiving in a Windows development environment; Linux ones are not.The reason I began assembling these lessons arose from the frustrations I had in early code development. Since I was getting small pieces of the functionality running directly in Python from the command line, some of which is shown in the prior two installments, my initial failures to import these routines in a code file (*.py) and get them to work had me pulling my hair out. I simply could not understand why routines that worked directly from the command line did not work once embedded into a code file.One discovery is that Flask does not play well with the Python list command. If one inspects prior SPARQL examples in this series (for example, [**CWPK 25**](https://www.mkbergman.com/2358/cwpk-25-querying-kbpedia-with-sparql/)), one can see that this construct is common with the standard query code. One adjustment, therefore, was to remove the list generator, and install a looping function for the query output. This applied to both RDFLib and owlready2.Besides the lessons presented above, some of the hypotheses I tested to get things to work included the use of CDATA (which only applies to XML), pasting to or saving and retrieving from intermediate text files, changing content-type or mimetype, treatment of the Python multi-line convention ("""), possible use of JavaScript, and more. Probably the major issue I needed to overcome was turning on space and tab display in my local editor to remove their mixed use. This experience really brought home to me the fundamental adherence to indentation in the Python language.Nonetheless, by following these guidelines and with eventual multiple tries, I was finally able to get a basic code block working, as documented under the next step. **8. Create and validate an external SPARQL query using SPARQLwrapper to this endpoint.**Since the approach that worked above got closer to the standard RDFLib approach, I decided to expand the query form to allow for external searches as well. Besides modifications to the Web page template, the use of external sources also invokes the [SPARQLwrapper](https://sparqlwrapper.readthedocs.io/en/latest/main.html) extension to RDFLib. Though its results presentation is a bit different, and we now have a requirement to also input and retrieve the URL of the external SPARQL endpoint, we were able to add this capability fairly easily.The resulting code is actually quite simple, though the path to get there was anything but. I present below the eventual code file so developed, with code notes following the listing. You will see that, aside from the Flask code conventions and decorators, that our code file is quite similar to others developed throughout *cowpoke*:<jupyter_code>from flask import Flask, Response, request, render_template            # Note 1
from owlready2 import *
import rdflib
from rdflib import Graph
import json
from SPARQLWrapper import SPARQLWrapper, JSON, XML

# load knowledge graph files
main = '/var/data/kbpedia/kbpedia_reference_concepts.owl'              # Note 2
skos_file = 'http://www.w3.org/2004/02/skos/core' 
kko_file = '/var/data/kbpedia/kko.owl'

# set up scopes and namespaces
world = World()                                                        # Note 2 
kb = world.get_ontology(main).load()
rc = kb.get_namespace('http://kbpedia.org/kko/rc/')
skos = world.get_ontology(skos_file).load()
kb.imported_ontologies.append(skos)
kko = world.get_ontology(kko_file).load()
kb.imported_ontologies.append(kko)

graph = world.as_rdflib_graph()

# set up Flask microservice
app = Flask(__name__)                                                  # Note 3

@app.route("/")
def sparql_form():
    return render_template('sparql_page.html')

# set up route for submitting query, receiving results 
@app.route('/submit', methods=['POST'])                                # Note 4
def submit():
#    if request.method == 'POST':
    q_submit = None
    results = ''
    if request.form['q_submit'] is None or len(request.form['q_submit']) < 5:
        return Response(
        'Your request to the SPARQL endpoint must contain a \'query\'.',
        mimetype = 'text/plain'
        )
    else:
        data = request.form['q_submit']                                # Note 5
        source = request.form['selectSource']
        format = request.form['selectFormat']
        q_url = request.values.get('q_url')
        try:                                                           # Note 6
            if source == 'kbpedia' and format == 'owlready':           # Note 7
                q_query = graph.query_owlready(data)                   # Note 8
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = results.replace(']', ']\n')
            elif source == 'kbpedia' and format == 'rdflib':
                q_query = graph.query(data)                            # Note 8
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = results.replace('))', '))\n')
            elif source == 'kbpedia' and format == 'xml':
                q_query = graph.query(data)
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = q_query.serialize(format='xml')
                results = str(results)
                results = results.replace('<result>', '\n<result>')
            elif source == 'kbpedia' and format == 'json':
                q_query = graph.query(data)
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = q_query.serialize(format='json')
                results = str(results)
                results = results.replace('}}, ', '}}, \n')
            elif source == 'kbpedia' and format == 'html':             #Note 9
                q_query = graph.query(data)
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = q_query.serialize(format='csv')
                results = str(results)
                results = results.readlines()
#                table = '<html><table>'
                for row in results:
#                     row = str(row)                    
                     result = row[0]
#                    row = row.replace('\r\n', '')
#                    row = row.replace(',', '</td><td>')
#                    table += '<tr><td>' + row + '</td></tr>' + '\n'
#                table += '</table><br></html>' 
#                results = table
                return result
            elif source == 'kbpedia' and format == 'txt':
                q_query = graph.query(data)
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = q_query.serialize(format='txt')
            elif source == 'kbpedia' and format == 'csv':
                q_query = graph.query(data)
                for row in q_query:
                    row = str(row)
                    results = results + row
                results = q_query.serialize(format='csv')
            elif source == 'external' and format == 'rdfxml':
                q_url = str(q_url)
                results = q_url
            elif source == 'external' and format == 'xml':
                sparql = SPARQLWrapper(q_url)
                data = data.replace('\r', '')
                sparql.setQuery(data)
                results = sparql.query()
            elif source == 'external' and format == 'json':            #Note 10
                sparql = SPARQLWrapper(q_url)
                data = data.replace('\r', '')
#                data = data.replace("\n", "\n' + '")
#                data = '"' + data + '"'
                sparql.setQuery(data)
                sparql.setReturnFormat(JSON)                           #Note 10
                results = sparql.queryAndConvert()
#                q_sparql = str(sparql)
#                results = q_sparql
            else:                                                      #Note 11
                results = ('This combination of Source + Format is not available. Here are the possible combinations:\n\n' + 
                           '    Kbpedia:   owlready2:    Formats:  owlready2\n' + 
                           '                  rdflib:                 rdflib\n' +
                           '                                             xml\n' +
                           '                                            json\n' +
                           '                                           *html\n' +
                           '                                            text\n' +
                           '                                             csv\n' +
                           '   External:  as entered:                rdf/xml\n' +
                           '                                           *json\n\n' +
                           '            * combo still buggy')
            if format == 'html':
                return Response(results, mimetype='text/html')         # Note 9, 12
            else:
                return Response(results, mimetype='text/plain')
        except Exception as e:                                         # Note 6
            return Response(
            'Error(s) found in query: ' + str(e),
            mimetype = 'text/plain'
            )

if __name__ == "__main__":
    app.run(debug=true)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08327710301394511, 'fraction_numerical': 0.0052294197031039135, 'mean_word_length': 2.4433688286544046, 'pattern_counts': {'":': 1, '<': 19, '<?xml version=': 0, '>': 22, 'https://': 21, 'lorem ipsum': 0, 'www.': 6, 'xml': 6}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '635582', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 4507, 'n_tokens_neox': 4270, 'n_words': 1972, 'path': 'cwpk-60-adding-sparql-endpoint-2.ipynb', 'repo_name': 'Cognonto/CWPK', 'size': '17784'}