<commit_before>---
layout: documentation
title: The Case for Semantic Modeling
---

You might be wondering why we need a semantic model at all.  Wouldn't it be easier to
just work with Avro schema and skip the semantic model altogether?  

The short answer is that Avro schemas are brittle because they express a physical model for data on the wire in 
one particular format.  But a given resource can have more than one description; there may be multiple
*views* of a given resource.  While Avro supports schema evolution to some extent, it does not offer a
mechanism for stitching together alternative views of a resource.  

Let's look at a concrete example.  Suppose you decide to use the 
[W3C Activity Streams Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) to report
social interactions.  For instance, whenever someone "likes" a blog post, you might publish a
record like the one shown below, where `endTime` is the time of the event expressed in
milliseconds since the beginning of the epoch (00:00:00 January 1, 1970).

**Sample Activity Record: Version 1**

```json
  {
    "type" : "as:Like",
    "actor" : "http://example.com/person/alice",
    "object" : "http://example.com/person/bob/post/5",
    "endTime" : 1461753982
  }
```

Here's the schema for this record expressed in [Avro IDL](https://avro.apache.org/docs/1.8.0/idl.html)
syntax:

**Schema for Activity Records: Version 1**

```java
@namespace("com.example.analytics.v1")
protocol Analytics {

  record Like {
    string type;
    string actor;
    string object;
    long endTime;
  }
}
```

Given a stream of such records you can compute some interesting metrics including:

*  The number of times that a given social media posting has been liked
*  The number of times that a given actor has liked something

You might produce reports that aggregate these metrics by day, week and month.  The product manager is delighted with the result.  

A couple months later, she asks that you slice the metrics based on the educational organization to
which the actor belongs.

To support this requirement, you decide to enrich the activity record so that it includes information
about the actor's affiliation with an educational organization as shown below.

**Sample Activity Record: Version 2**

```json
<commit_msg>Updated The Case for Semantic Modeling<commit_after>---
layout: documentation
title: The Case for Semantic Modeling
---

You might be wondering why we need a semantic model at all.  Wouldn't it be easier to
just work with Avro schema and skip the semantic model altogether?  

The short answer is that Avro schemas are brittle because they express a physical model for data on the wire in 
one particular format.  But a given resource can have more than one description; there may be multiple
*views* of a given resource.  While Avro supports schema evolution to some extent, it does not offer a
mechanism for stitching together alternative views of a resource.  

Let's look at a concrete example.  Suppose you decide to use the 
[W3C Activity Streams Vocabulary](https://www.w3.org/TR/activitystreams-vocabulary/) to report
social interactions.  For instance, whenever someone "likes" a blog post, you might publish a
record like the one shown below, where `endTime` is the time of the event expressed in
milliseconds since the beginning of the epoch (00:00:00 January 1, 1970).

**Sample Activity Record: Version 1**

```json
  {
    "type" : "as:Like",
    "actor" : "http://example.com/person/alice",
    "object" : "http://example.com/person/bob/post/5",
    "endTime" : 1461753982
  }
```

Here's the schema for this record expressed in [Avro IDL](https://avro.apache.org/docs/1.8.0/idl.html)
syntax:

**Schema for Activity Records: Version 1**

```java
@namespace("com.example.analytics.v1")
protocol Analytics {

  record Like {
    string type;
    string actor;
    string object;
    long endTime;
  }
}
```

Given a stream of such records you can compute some interesting metrics including:

*  The number of times that a given social media posting has been liked
*  The number of times that a given actor has liked something

Suppose you produce reports that aggregate these metrics by day, week and month.  Your product manager is delighted with the result.  

A couple months later, she asks that you slice the metrics based on the educational organization to
which the actor belongs.

To support this requirement, you decide to enrich the activity record so that it includes information
about the actor's affiliation with an educational organization as shown below.

**Sample Activity Record: Version 2**

```json

{'fraction_non_alphanumeric': 0.06991799741044454, 'fraction_numerical': 0.013379369874838154, 'mean_word_length': 4.060043668122271, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 4, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '352216', 'n_tokens_mistral': 1293, 'n_tokens_neox': 1217, 'n_words': 641}