<jupyter_start><jupyter_text>DataDictionary_RDF_Data_Cube2This Notebook steps through the development of a method to convert a UKDS DataDictionary .rtf file to a RDF file using the Data Cube vocabulary https://www.w3.org/TR/vocab-data-cube/  Initial setup  Import packages<jupyter_code>import os, ukds
import pandas as pd
from fairly import CreateRDF, Fuseki<jupyter_output><empty_output><jupyter_text>Set filepathsThis sets a filepath to an example data dictionary on a local file system, in this case the 'uktus15_household_ukda_data_dictionary.rtf' file.<jupyter_code>base_dir=os.path.join(*[os.pardir]*4,r'_Data\United_Kingdom_Time_Use_Survey_2014-2015\UKDA-8128-tab')
dd_fp=os.path.join(base_dir,r'mrdoc\allissue\uktus15_household_ukda_data_dictionary.rtf')<jupyter_output><empty_output><jupyter_text>Create DataDictionaryA ukds.DataDictionary instance is created and the .rtf file is read into it.<jupyter_code>dd=ukds.DataDictionary()
dd.read_rtf(dd_fp)
dd.variable_list[0:1]<jupyter_output><empty_output><jupyter_text>Discussion  AimThe aim of this notebook is to develop a method to convert the information in UKDS data dictionary files into RDF format based on RDF Data Cube vocabulary https://www.w3.org/TR/vocab-data-cube/schemes.Once converted this RDF data can be combined with RDF Data Cube for the UKDS data table files.  Sample callSample code could look like:```pythont=dd.to_rdf_data_cube()  dd is a DataDictionary instance```where t is a string of a turtle file.  Method development  Setup<jupyter_code>c=CreateRDF()
c.add_data_cube_prefixes()
c.add_skos_prefixes()
c.add_prefix('ukds8128','<http://purl.org/berg/ukds8128/>')
c.add_prefix('ukds8128-concept','<http://purl.org/berg/ukds8128/concept/>')<jupyter_output><empty_output><jupyter_text>Inputs<jupyter_code>row_dimension_property_uri='ukds:household'
row_dimension_property_label='"the Household serial number"'
row_concept_uri='ukds8128-concept:household'
row_concept_prefLabel='"A household"'<jupyter_output><empty_output><jupyter_text>DimensionsIn Data Cube, each observation has an associated set of dimensions and measures.Here, each single value (or cell) in a UKDS data table is converted to a single qb:Observation. This is the 'measure dimension' appraoch.What are the dimensions and measures for each observation?- one dimension is the table row. In UKDS data tables each row normally represents an entity (such as individual, household etc). The entity can be specified with a uri and the observation linked to it.- a second dimension is the table column. In UKDS data tables the columns are the type of observation made, i.e. what is measured. This can correspond to a measure dimension for each observation.  Table row dimensionIf each row is an entity of a class (such as individuals or households), then this class should be defined as a skos:Concept.In our example, each row of the 'uktus15_household.tab' data table is a household.<jupyter_code>def add_skos_table_row_concept(self,
                               c,
                               row_concept_uri,
                               row_concept_prefLabel):
    """Adds a skos:Concept for the table row dimension
    """
    return c.add_skos_concept(
                         concept_uri=row_concept_uri,
                         topConceptOf_uri=None,
                         prefLabel=row_concept_prefLabel,
                         notation=None,
                         inScheme_uri=None,
                         predicate_object_list=None
                         )

print(add_skos_table_row_concept(dd,c,row_concept_uri,row_concept_prefLabel))<jupyter_output>ukds8128-concept:Household a skos:Concept ;
	skos:prefLabel "A household" .<jupyter_text>This concept is then used as a reference for the dimension property<jupyter_code>def add_data_cube_table_row_dimension_property(self,
                                               c,
                                               row_dimension_property_uri,
                                               row_dimension_property_label,
                                               row_concept_uri,
                                               
                                              ):
    """Adds a qb:DimensionProperty for the tabel row dimension
    """
    return c.add_data_cube_dimension_property(
                                   dimension_property_uri=row_dimension_property_uri,
                                   label=row_dimension_property_label,
                                   subPropertyOf_uri=None,
                                   range_uri=None,
                                   concept_uri=row_concept_uri,
                                   code_list_uri=None,
                                   predicate_object_list=None)
    
print(add_data_cube_table_row_dimension_property(dd,
                                           c,
                                           row_dimension_property_uri,
                                           row_dimension_property_label,
                                           row_concept_uri))<jupyter_output>ukds:household a qb:DimensionProperty ;
	a rdf:Property ;
	rdfs:label "the Household serial number" ;
	qb:concept ukds8128-concept:Household .<jupyter_text>Table column dimensions (measure dimension)<jupyter_code>def add_skos_table_column_concept(self,
                                  c,
                                  column_concept_uri,
                                  column_concept_prefLabel,
                                  column_concept_notation,
                                  column_concept_comment
                                 ):
    """Adds a skos:Concept for the table row dimension
    """
    return c.add_skos_concept(
                         concept_uri=column_concept_uri,
                         topConceptOf_uri=None,
                         prefLabel=column_concept_prefLabel,
                         notation=column_concept_notation,
                         inScheme_uri=None,
                         predicate_object_list=[('rdfs:comment',column_concept_comment)]
                         )

column_concept_uri='ukds-concept:serial'
column_concept_prefLabel='"Household number"'
column_concept_notation='"serial"'
column_concept_comment='"%s"' % str({'pos': '1',
                            'variable': 'serial',
                            'variable_label': 'Household number',
                            'variable_type': 'numeric',
                            'SPSS_measurement_level': 'SCALE',
                            'SPSS_user_missing_values': '',
                            'value_labels': ''})
print(add_skos_table_column_concept(dd,
                                    c,
                                    column_concept_uri,
                                    column_concept_prefLabel,
                                    column_concept_notation,
                                    column_concept_comment))<jupyter_output>ukds-concept:serial a skos:Concept ;
	skos:prefLabel "Household number" ;
	skos:notation "serial" ;
	rdfs:comment "{'pos': '1', 'variable': 'serial', 'variable_label': 'Household number', 'variable_type': 'numeric', 'SPSS_measurement_level': 'SCALE', 'SPSS_user_missing_values': '', 'value_labels': ''}" .<jupyter_text>Sample RDF fileThe RDF Data Cube webpage gives, in Example 14, an example of a concept (or variable) and it's code list using the `skos` vocabulary.The proposal is the RDF file would look as below. This shows the data in turtle (.ttl) format for the variable *serial*:```turtle@prefix qb:	 .@prefix skos:  .@prefix rdf:  .@prefix rdfs:  .@prefix xsd:  .@prefix ukds8128-code:  .  a prefix for the UKDS Time Use Survey 2014-2015 dataset@prefix ukds8128-measure:  .ukds8128-measure:serial a rdf:Property, qb:MeasureProperty ;    rdfs:label "serial"@en ;    rdfs:subPropertyOf sdmx-measure:obsValue ;    rdfs:range xsd:decimal . ukds8128-code:serial a skos:ConceptScheme ;    skos:prefLabel "serial"@en ;  the 'variable' value    rdfs:label "serial"@en ;  the 'variable' value    skos:notation "serial" ;  the 'variable' value    skos:note "Household number."@en ;  the 'variable_label' value    skos:definition  ;   a uri based on the file name    .``` This shows the data in turtle (.ttl) format for the variable *strata*:```turtle@prefix qb:	 .@prefix skos:  .@prefix rdfs:  .@prefix ukds8128:  .ukds8128-code:strata a skos:ConceptScheme ;    skos:prefLabel "strata"@en ;    rdfs:label "strata"@en ;    skos:notation "strata" ;    skos:note "Strata"@en ;    skos:definition  ;    rdfs:seeAlso ukds8128-code:Strata ;    skos:hasTopConcept ukds8128-code:strata_code_-2.0 .ukds8128-code:Strata a rdfs:Class, owl:Class;    rdfs:subClassOf skos:Concept ;    rdfs:label "strata"@en;    rdfs:comment "Strata"@en;    rdfs:seeAlso ukds8128-code:strata .ukds8128-code:strata_code_-2.0 a skos:Concept, ukds8128-code:Strata;    skos:topConceptOf ukds8128-code:strata;    skos:prefLabel "Schedule not applicable"@en ;    skos:notation -2.0 ;    skos:inScheme ukds8128-code:strata .```Here this includes a single 'value label' code.  Developing the method  to_data_structure_definition<jupyter_code>def to_data_structure_definition(self,prefix,dataset_name):
    """Returns a RDF Turtle string of the qb:DataStructureDefinition using the Data Cube and skos vocabulary
    
    Arguments:
        - self: the DataDictionary instance
        - prefix (str): the prefix to use for the data dictionary uris   
        - dataset_name (str): the name of the dataset
    
    """
    
    l=['%s:%s-dsd a qb:DataStructureDefinition' % (prefix,dataset_name)]
        
    for i,variable in enumerate(dd.get_variable_names()):
        l.append('qb:component [ qb:measure %s-measure:%s; qb:order %s ]' % (prefix,variable,i+1))
        
    st=' ;\n\t'.join(l) + ' .\n\n'
    
    return st

print(to_data_structure_definition(dd,'ukds8128','uktus15_household'))<jupyter_output>ukds8128:uktus15_household-dsd a qb:DataStructureDefinition ;
	qb:component [ qb:measure ukds8128-measure:serial; qb:order 1 ] ;
	qb:component [ qb:measure ukds8128-measure:strata; qb:order 2 ] ;
	qb:component [ qb:measure ukds8128-measure:psu; qb:order 3 ] ;
	qb:component [ qb:measure ukds8128-measure:HhOut; qb:order 4 ] ;
	qb:component [ qb:measure ukds8128-measure:hh_wt; qb:order 5 ] ;
	qb:component [ qb:measure ukds8128-measure:IMonth; qb:order 6 ] ;
	qb:component [ qb:measure ukds8128-measure:IYear; qb:order 7 ] ;
	qb:component [ qb:measure ukds8128-measure:DM014; qb:order 8 ] ;
	qb:component [ qb:measure ukds8128-measure:DM016; qb:order 9 ] ;
	qb:component [ qb:measure ukds8128-measure:DM510; qb:order 10 ] ;
	qb:component [ qb:measure ukds8128-measure:DM1115; qb:order 11 ] ;
	qb:component [ qb:measure ukds8128-measure:DM1619; qb:order 12 ] ;
	qb:component [ qb:measure ukds8128-measure:NumAdult; qb:order 13 ] ;
	qb:component [ qb:measure ukds8128-measure:NumChild; qb:order 14 ] ;
[...]<jupyter_text>to_measure_property<jupyter_code>def to_measure_property(self,variable,prefix,):
    """Returns a RDF Turtle string of the qb:MeaureProperty using the Data Cube and skos vocabulary
    
    Arguments:
        - self: the DataDictionary instance
        - variable (str): the variable to convert to RDF
        - prefix (str): the prefix to use for the data dictionary uris   
    
    """
    d=self.get_variable_dict(variable)
    
    if d['value_labels']:
        x=', qb:CodedProperty'
    else:
        x=''
    
    l=[
        'ukds8128-measure:%s a rdf:Property, qb:MeasureProperty%s' % (variable,x) ,
        'rdfs:label "serial"@en' ,
        'rdfs:subPropertyOf sdmx-measure:obsValue',
    ]
    
    if d['value_labels']:
        l.append('qb:CodeList %s-code:%s' % (prefix,variable))
        l.append('rdfs:range %s-code:%s' % (prefix,variable[0].upper()+variable[1:]))
    else:
        if d['variable_type']=='numeric':
            l.append('rdfs:range xsd:decimal')
        
    st=' ;\n\t'.join(l) + ' .\n\n'
    
    return st

print(to_measure_property(dd,'serial','ukds8128')) 
print(to_measure_property(dd,'strata','ukds8128'))<jupyter_output>ukds8128-measure:serial a rdf:Property, qb:MeasureProperty ;
	rdfs:label "serial"@en ;
	rdfs:subPropertyOf sdmx-measure:obsValue ;
	rdfs:range xsd:decimal .


ukds8128-measure:strata a rdf:Property, qb:MeasureProperty, qb:CodedProperty ;
	rdfs:label "serial"@en ;
	rdfs:subPropertyOf sdmx-measure:obsValue ;
	qb:CodeList ukds8128-code:strata ;
	rdfs:range ukds8128-code:Strata .<jupyter_text>to_codelist<jupyter_code>def to_codelist(self,variable,prefix,filename_no_ext):
    """Returns a RDF Turtle string of the codelist using the Data Cube and skos vocabulary
    
    Arguments:
        - self: the DataDictionary instance
        - variable (str): the variable to convert to RDF
        - filename_no_ext (str): the filename of the Data Dictionary file with no extension included.
        - prefix (str): the prefix to use for the data dictionary uris
    
    """
    d=self.get_variable_dict(variable)
    variable_lower=d['variable'][0].lower() + d['variable'][1:]
    variable_upper=d['variable'][0].upper() + d['variable'][1:]
    
    # ConceptScheme
    l=[]
    l+=[
        '%s-code:%s a skos:ConceptScheme' % (prefix,variable_lower),
        'skos:prefLabel "%s"@en' % d['variable'],
        'rdfs:label "%s"@en' % d['variable'],
        'skos:notation "%s"' % d['variable'],
        'skos:note "%s"@en' % d['variable_label'],
        'skos:definition <%s:%s>' % (prefix,filename_no_ext),
        ]
    
    if d['value_labels']:
        
        l.append('rdfs:seeAlso %s-code:%s' % (prefix,variable_upper))
        for k,v in d['value_labels'].items():
            l.append('skos:hasTopConcept %s:%s_code_%s' % (prefix,variable_lower,k))
    
    st=' ;\n\t'.join(l) + ' .\n\n'
    
    # Code
    
    if d['value_labels']:
        
        l=[
            '%s-code:%s a rdfs:Class, owl:Class ' % (prefix,variable_upper),
            'rdfs:subClassOf skos:Concept ',
            'rdfs:label "%s"@en ' % d['variable'],
            'rdfs:comment "%s"@en ' % d['variable_label'],
            'rdfs:seeAlso %s-code:%s ' % (prefix,variable_lower),
        ]
        st+=' ;\n\t'.join(l) + ' .\n\n'
        
        for k,v in d['value_labels'].items():
            l=[
                '%s-code:%s_code_%s a skos:Concept, %s-code:%s' % (prefix,variable_lower,k,prefix,variable_upper),
                'skos:topConceptOf %s-code:%s' % (prefix,variable_lower),
                'skos:prefLabel "%s"@en' % v,
                'skos:notation %s' % k,
                'skos:inScheme %s-code:%s'  % (prefix,variable_lower),
            ]
            st+=' ;\n\t'.join(l) + ' .\n\n'
            
    return st
    
print(to_codelist(dd,'serial','ukds8128','uktus15_household_ukda_data_dictionary'))
print(to_codelist(dd,'strata','ukds8128','uktus15_household_ukda_data_dictionary'))
def to_rdf_data_cube(self,prefix,base_uri,filename_no_ext,dataset_name):
    """Returns a RDF Turtle string using the Data Cube vocabulary
    
    Arguments:
        - self: the DataDictionary instance
        - filename_no_ext (str): the filename of the Data Dictionary file with no extension included.
        - prefix (str): the prefix to use for the data dictionary base uri
        - base_uri (str): the data dictionary base uri
        - dataset_name (str): the name of the dataset
    
    """
    
    st="""
@prefix qb:    <http://purl.org/linked-data/cube#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix %s: %s> .
@prefix %s-code: %scode#> .
@prefix %s-measure: %smeasure#> .

""" % (prefix,base_uri,prefix,base_uri,prefix,base_uri)
    
    st+=to_data_structure_definition(self,prefix,dataset_name)
    
    for variable in dd.get_variable_names():
        st+=to_measure_property(self,variable,prefix)
        st+=to_codelist(self,variable,prefix,filename_no_ext)
        break
    return st
print(to_rdf_data_cube(dd,'ukds8128','<http://purl.org/berg/ukds8128/','uktus15_household_ukda_data_dictionary','uktus15_household'))<jupyter_output>@prefix qb:    <http://purl.org/linked-data/cube#> .
@prefix skos: <http://www.w3.org/2004/02/skos/core#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix ukds8128: <http://purl.org/berg/ukds8128/> .
@prefix ukds8128-code: <http://purl.org/berg/ukds8128/code#> .
@prefix ukds8128-measure: <http://purl.org/berg/ukds8128/measure#> .

ukds8128:uktus15_household-dsd a qb:DataStructureDefinition ;
	qb:component [ qb:measure ukds8128-measure:serial; qb:order 1 ] ;
	qb:component [ qb:measure ukds8128-measure:strata; qb:order 2 ] ;
	qb:component [ qb:measure ukds8128-measure:psu; qb:order 3 ] ;
	qb:component [ qb:measure ukds8128-measure:HhOut; qb:order 4 ] ;
	qb:component [ qb:measure ukds8128-measure:hh_wt; qb:order 5 ] ;
	qb:component [ qb:measure ukds8128-measure:IMonth; qb:order 6 ] ;
	qb:component [ qb:measure ukds8128-measure:IYear; qb:order 7 ] ;
	qb:componen[...]
{'fraction_non_alphanumeric': 0.11323376172282042, 'fraction_numerical': 0.02373509320365868, 'mean_word_length': 2.8474387527839644, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 58, 'https://': 2, 'lorem ipsum': 0, 'www.': 10, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 38, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '513309', 'license': 'MIT', 'n_tokens_mistral': 5992, 'n_tokens_neox': 5596, 'n_words': 1169, 'path': 'development/DataDictionary_RDF_Data_Cube2.ipynb', 'repo_name': 'building-energy/ukds', 'size': '17274'}