<jupyter_start><jupyter_text><jupyter_code>%%capture
!pip install git+git://github.com/apmoore1/target-extraction.git@v0.0.2#egg=target_extraction<jupyter_output><empty_output><jupyter_text>SemEval 2015 and 2016 datasets do the texts contain the same aspect more than once?The title is the question we are going to answer here. We are only going to look at sentences that contain (target, aspect, sentiment) where aspect can also be an (entity, aspect) pair. The reason why an aspect or (entity, aspect) pair can occur more than once in a text is because it is attached to a target that is within the text. Thus there could be the following case:```The CPU memory is great, but the RAM is not```Given this text we could have the following annotations (memory, MEMORY, positive) and (RAM, MEMORY, negative) where the following represent (target, aspect, sentiment). Thus in this case the MEMORY aspect has occured twice in the same text. The reason we want to see if this occurs or not in the SemEval datasets is to see if we could treat the target sentiment problem as just a text/sentence level aspect based sentiment analysis task. Further whether when trying to identify aspects whether this has to be done at the target level or if it could be done at the text/sentence level. As shown by this example if it was treated a text level aspect based sentiment problem it would have to get one of the two samples wrong as the aspect occurs twice with two different sentiments.To analysis this we first need to load the datasets. The datasets that have the following annotations (target, aspect, sentiment) are:1. [Restaurant SemEval 2015 Train.](http://alt.qcri.org/semeval2015/task12/index.php?id=data-and-tools)2. [Restaurant SemEval 2015 Test.](http://alt.qcri.org/semeval2015/task12/index.php?id=data-and-tools)3. [Restaurant SemEval 2016 Train.](http://alt.qcri.org/semeval2016/task5/index.php?id=data-and-tools)4. [Restaurant SemEval 2016 Test.](http://alt.qcri.org/semeval2016/task5/index.php?id=data-and-tools)There are more SemEval datasets from the 2016 edition that contian this annotation format. However for now we are only going to look at these datasets.<jupyter_code>from collections import Counter
from pathlib import Path

from google.colab import files
from target_extraction.dataset_parsers import semeval_2016
from target_extraction.data_types import TargetTextCollection

# SemEval 2014 Laptop and Restaurant
semeval_dataset_fp = {'Restaurant Train 2015': Path('ABSA-15_Restaurants_Train_Final.xml'),
                      'Restaurant Test 2015': Path('ABSA15_Restaurants_Test.xml'),
                      'Restaurant Train 2016': Path('ABSA16_Restaurants_Train_SB1_v2.xml'),
                      'Restaurant Test 2016': Path('EN_REST_SB1_TEST_2016.xml')}
semeval_dataset = {}
for dataset_name, fp in semeval_dataset_fp.items():
  if not fp.exists():
    print(f'Upload {dataset_name}')
    files.upload()
  semeval_dataset[dataset_name] = semeval_2016(fp, conflict=True)

# SemEval Restaurant
restaurant_train = semeval_dataset['Restaurant Train 2015']
restaurant_test = semeval_dataset['Restaurant Test 2015']
restaurant_combined_2015 = TargetTextCollection.combine(restaurant_train, 
                                                        restaurant_test)
restaurant_combined_2015.name = 'Restaurant 2015'

restaurant_train = semeval_dataset['Restaurant Train 2016']
restaurant_test = semeval_dataset['Restaurant Test 2016']
restaurant_combined_2016 = TargetTextCollection.combine(restaurant_train, 
                                                        restaurant_test)
restaurant_combined_2016.name = 'Restaurant 2016'<jupyter_output><empty_output><jupyter_text>To confirm that we have the data correct we shall check that the total number of aspects/categories for the datasets matches those that are in the papers:1. For Restaurant 2015 we should have 2499 according to [Pontiki et al. 2015](https://www.aclweb.org/anthology/S15-2082.pdf)2. For Restaurant 2016 we should have 3366 according to [Pontiki et al. 2016](https://www.aclweb.org/anthology/S16-1002.pdf)<jupyter_code>print(f'Number of aspects in Restaurant 2015 '
      f'{restaurant_combined_2015.number_categories()}')
print(f'Number of aspects in Restaurant 2016 '
      f'{restaurant_combined_2016.number_categories()}')<jupyter_output>Number of aspects in Restaurant 2015 2499
Number of aspects in Restaurant 2016 3366<jupyter_text>It would appear from above that we have parsed that dataset correctly as we match the number of aspects in the original paper.We now move on to see how many sentences and the number of aspects that are affected if we treat the task as a text level aspect task instead of taking into the target:<jupyter_code>def aspects_affected(dataset: TargetTextCollection) -> None:
  number_texts_wrong = 0
  number_wrong = 0
  number_text_wrong_diff = 0
  number_diff_wrong = 0
  for key, value in dataset.items():
    aspects_in_text = value['categories']
    all_aspect_sentiments = value['target_sentiments']
    if aspects_in_text is None:
      assert value['targets'] is None
      continue
    number_aspects = len(aspects_in_text)
    aspect_count = Counter(aspects_in_text)
    aspect_count_diff = number_aspects - len(aspect_count)
    if aspect_count_diff != 0:
      number_texts_wrong += 1
      number_wrong += aspect_count_diff

      # This is a more fine grained analysis that looks at if the aspects that 
      # do occur more than once in a text, occur more than once with a 
      # different sentiment in the text. As if they all have the same sentiment 
      # to some degree you can just get anyway with not duplicating the aspect
      # even though the aspects should be attached to a target.
      number_text_wrong_diff_bool = False
      aspects_occur_more_than_once = [aspect for aspect, count in aspect_count.items() 
                                      if count > 1]
      for aspect in aspects_occur_more_than_once:
        aspect_sentiments = set()
        for index, sentiment in enumerate(all_aspect_sentiments):
          if aspects_in_text[index] == aspect:
            aspect_sentiments.add(sentiment)
        if len(aspect_sentiments) > 1:
          number_text_wrong_diff_bool = True
          number_diff_wrong += aspect_count[aspect]
      if number_text_wrong_diff_bool:
        number_text_wrong_diff += 1
      #print(aspects_occur_more_than_once)
      #print(aspect_count)
  number_samples = dataset.number_categories()
  percent_wrong = round((number_wrong / float(number_samples)) * 100, 2)
  percent_text_wrong = round((number_texts_wrong / float(number_samples)) * 100, 2)
  
  percent_diff_wrong = round((number_diff_wrong / float(number_samples)) * 100, 2)
  percent_diff_text_wrong = round((number_text_wrong_diff / float(number_samples)) * 100, 2)
  print(f'For the dataset {dataset.name} which contains {number_samples} '
        f'samples and {len(dataset)} texts\n{number_texts_wrong}'
        f'({percent_text_wrong}%) texts and {number_wrong}'
        f'({percent_wrong}%) samples are affected.\n{number_text_wrong_diff}'
        f'({percent_diff_text_wrong}%) texts and {number_diff_wrong}'
        f'({percent_diff_wrong}%) samples are affected with respect to having a '
        'different sentiment.')
  
aspects_affected(restaurant_combined_2015)
aspects_affected(restaurant_combined_2016)<jupyter_output>For the dataset Restaurant 2015 which contains 2499 samples and 2000 texts
188(7.52%) texts and 246(9.84%) samples are affected.
41(1.64%) texts and 92(3.68%) samples are affected with respect to having a different sentiment.
For the dataset Restaurant 2016 which contains 3366 samples and 2676 texts
255(7.58%) texts and 365(10.84%) samples are affected.
50(1.49%) texts and 110(3.27%) samples are affected with respect to having a different sentiment.<jupyter_text>As we can see from the two datasets it shows that up to 11% of samples are affected and up to 8% of texts are affected, whereby the texts can contain more than one of the same aspect. However only up to 4% of samples and up to 2% of texts contain more than one of the same aspect, but where those same aspects have different sentiments like the example given at the start of the notebook on the aspect MEMORY. This shows that in the majority of cases when an aspect does occur more than once for the same text it does so with the same sentiment, thus allowing you to some degree to get anyway with using aspect based text level methods and ignoring the target. However by doing so you will not be able to accurately classify up to 4% of samples that do contain texts with multiple of the same aspects with different sentiments. How many implicit targets are there?In these datasets some of the targets are implicit. This can be found when the target value is `None` while the aspect exisits. An example of this can be seen below:<jupyter_code>for key, value in restaurant_combined_2015['P#3:3'].items():
  print(f'{key}: {value}')<jupyter_output>text: Cool atmosphere, the fire place in the back really ads to it but needs a bit more heat throughout on a cold night.
text_id: P#3:3
targets: ['atmosphere', 'fire place', None]
spans: [Span(start=5, end=15), Span(start=21, end=31), Span(start=0, end=0)]
target_sentiments: ['positive', 'positive', 'negative']
categories: ['AMBIENCE#GENERAL', 'AMBIENCE#GENERAL', 'AMBIENCE#GENERAL']
category_sentiments: None<jupyter_text>This particular example shows how difficult it would be to identify these implicit examples as the aspect already appears explictly in the text. However we are here to answer the question of how many implicit targets there are:<jupyter_code>def implicit_targets(dataset: TargetTextCollection) -> None:
  number_implicit_texts = 0
  number_implicit_targets = 0
  for key, value in dataset.items():
    targets = value['targets']
    aspects = value['categories']
    if targets is None:
      continue
    text_is_implicit = False
    for target, aspect in zip(targets, aspects):
      if target is None:
        assert not aspect is None
        number_implicit_targets += 1
        text_is_implicit = True
    if text_is_implicit:
      number_implicit_texts += 1
  number_samples = dataset.number_categories()
  percent_texts = round((number_implicit_texts / float(number_samples)) * 100, 2)
  percent_targets = round((number_implicit_targets / float(number_samples)) * 100, 2)
  print(f'For the dataset {dataset.name} which contains {number_samples} '
        f'samples and {len(dataset)} texts\n{number_implicit_texts}'
        f'({percent_texts}%) texts and {number_implicit_targets}'
        f'({percent_targets}%) samples are implicit.')
  
implicit_targets(restaurant_combined_2015)
implicit_targets(restaurant_combined_2016)<jupyter_output>For the dataset Restaurant 2015 which contains 2499 samples and 2000 texts
566(22.65%) texts and 621(24.85%) samples are implicit.
For the dataset Restaurant 2016 which contains 3366 samples and 2676 texts
766(22.76%) texts and 832(24.72%) samples are implicit.
{'fraction_non_alphanumeric': 0.07320166696865374, 'fraction_numerical': 0.03687262185178474, 'mean_word_length': 4.398044009779951, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 2, 'xml': 4}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '513076', 'license': 'Apache-2.0', 'n_tokens_mistral': 3490, 'n_tokens_neox': 3053, 'n_words': 1180, 'path': 'texts_contain_the_same_aspect_more_than_once.ipynb', 'repo_name': 'apmoore1/target_aspect_unique', 'size': '11038'}