<commit_before># ShapeWorld

### Getting started

```bash
git clone --recursive https://github.com/AlexKuhnle/ShapeWorld.git
pip3 install -e ShapeWorld
```


### Recently added features

- Generator option `-H` to create an HTML file displaying the generated data (e.g. see in (examples))
- Various new and extended caption agreement datasets


### Table of content

- [About ShapeWorld](#about-shapeworld)
- [Example data](#example-data)
- [Integration into Python code](#integration-into-python-code)
- [Stand-alone data generation](#stand-alone-data-generation)
- [Loading extracted data](#loading-extracted-data)
- [CLEVR and NLVR interface](#clevr-and-nlvr-interface)
- [Evaluation and example models](#evaluation-and-example-models)



## About ShapeWorld

ShapeWorld is a framework which allows to specify generators for abstract, visually grounded language data (or just visual data).

The main motivation behind ShapeWorld is to provide a new testbed and evaluation methodology for visually grounded language understanding, particularly aimed at deep learning models. It differs from standard evaluation datasets in two ways: Firstly, data is randomly sampled during training and evaluation according to constraints specified by the experimenter. Secondly, its focus of evaluation is on linguistic understanding capabilities of the type investigated by formal semantics. In this context, the ShapeWorld tasks can be thought of as unit-testing multimodal models for specific linguistic generalization abilities -- similar to, for instance, the [bAbI tasks](https://research.fb.com/projects/babi/) of [Weston et al. (2015)](https://arxiv.org/abs/1502.05698) for text-only understanding.

The code is written in Python 3 (but should be compatible to Python 2). The data can either be obtained within a Python module as [NumPy](http://www.numpy.org/) arrays, and hence integrates into deep learning projects based on common frameworks like [TensorFlow](https://www.tensorflow.org/), [PyTorch](http://pytorch.org/) or [Theano](http://deeplearning.net/software/theano/), or it can be extracted into separate files. Both options are described further below. For language generation, the Python package [pydmrs](https://github.com/delph-in/pydmrs) [(Copestake et al., 2016)](http://www.lrec-conf.org/proceedings/lrec2016/pdf/634_Paper.pdf) is required.

**The ShapeWorld framework is still under active development.**

I am interested in hearing about any applications you plan to use the ShapeWorld data for. In particular, let me know if you have a great idea in mind that you are interested in investigating with such abstract data, but which the current setup does not allow to realize -- I am happy to collaboratively find a way to make it happen.

Contact: aok25 (at) cam.ac.uk



## Example data

#### Caption agreement datasets

- [Oneshape](examples/agreement/oneshape/data.html)
- [Oneshape (simple)](examples/agreement/oneshape_simple/data.html)
- [Multishape](examples/agreement/multishape/data.html)
- [Multishape (simple)](examples/agreement/multishape_simple/data.html)
- [Spatial](examples/agreement/spatial/data.html)
- [Spatial (simple)](examples/agreement/spatial_simple/data.html)
- [Relational](examples/agreement/relational/data.html)
- [Counting](examples/agreement/counting/data.html)
- [Counting (simple)](examples/agreement/counting_simple/data.html)
- [Quantification](examples/agreement/quantification/data.html)
- [Quantification (simple)](examples/agreement/quantification_simple/data.html)
- [Combination](examples/agreement/combination/data.html)

#### Classification datasets

- [Oneshape](examples/classification/oneshape/data.html)
- [Multishape](examples/classification/multishape/data.html)
- [Countshape](examples/classification/countshape/data.html)



## Integration into Python code

The easiest way to use the ShapeWorld data in your Python project is to directly call it from the code. Whenever a batch of training/evaluation instances is required, `dataset.generate(...)` is called with the respective arguments. This means that generation happens simultaneously to training/testing. Below an example of how to generate a batch of 128 training instances. See also the example models below.

```python
from shapeworld import dataset

dataset = dataset(dtype='agreement', name='multishape')
generated = dataset.generate(n=128, mode='train', noise_range=0.1, include_model=True)

# given to the image caption agreement system
batch = (generated['world'], generated['caption'], generated['agreement'])

# can be used for more specific evaluation
world_model = generated['world_model']
caption_model = generated['caption_model']
```
<commit_msg>readme: added links to html<commit_after># ShapeWorld

### Getting started

```bash
git clone --recursive https://github.com/AlexKuhnle/ShapeWorld.git
pip3 install -e ShapeWorld
```


### Recently added features

- Generator option `-H` to create an HTML file displaying the generated data (e.g. see in (examples))
- Various new and extended caption agreement datasets


### Table of content

- [About ShapeWorld](#about-shapeworld)
- [Example data](#example-data)
- [Integration into Python code](#integration-into-python-code)
- [Stand-alone data generation](#stand-alone-data-generation)
- [Loading extracted data](#loading-extracted-data)
- [CLEVR and NLVR interface](#clevr-and-nlvr-interface)
- [Evaluation and example models](#evaluation-and-example-models)



## About ShapeWorld

ShapeWorld is a framework which allows to specify generators for abstract, visually grounded language data (or just visual data).

The main motivation behind ShapeWorld is to provide a new testbed and evaluation methodology for visually grounded language understanding, particularly aimed at deep learning models. It differs from standard evaluation datasets in two ways: Firstly, data is randomly sampled during training and evaluation according to constraints specified by the experimenter. Secondly, its focus of evaluation is on linguistic understanding capabilities of the type investigated by formal semantics. In this context, the ShapeWorld tasks can be thought of as unit-testing multimodal models for specific linguistic generalization abilities -- similar to, for instance, the [bAbI tasks](https://research.fb.com/projects/babi/) of [Weston et al. (2015)](https://arxiv.org/abs/1502.05698) for text-only understanding.

The code is written in Python 3 (but should be compatible to Python 2). The data can either be obtained within a Python module as [NumPy](http://www.numpy.org/) arrays, and hence integrates into deep learning projects based on common frameworks like [TensorFlow](https://www.tensorflow.org/), [PyTorch](http://pytorch.org/) or [Theano](http://deeplearning.net/software/theano/), or it can be extracted into separate files. Both options are described further below. For language generation, the Python package [pydmrs](https://github.com/delph-in/pydmrs) [(Copestake et al., 2016)](http://www.lrec-conf.org/proceedings/lrec2016/pdf/634_Paper.pdf) is required.

**The ShapeWorld framework is still under active development.**

I am interested in hearing about any applications you plan to use the ShapeWorld data for. In particular, let me know if you have a great idea in mind that you are interested in investigating with such abstract data, but which the current setup does not allow to realize -- I am happy to collaboratively find a way to make it happen.

Contact: aok25 (at) cam.ac.uk



## Example data

#### Caption agreement datasets

- [Oneshape](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/oneshape/data.html)
- [Oneshape (simple)](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/oneshape_simple/data.html)
- [Multishape](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/multishape/data.html)
- [Multishape (simple)](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/multishape_simple/data.html)
- [Spatial](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/spatial/data.html)
- [Spatial (simple)](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/spatial_simple/data.html)
- [Relational](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/relational/data.html)
- [Counting](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/counting/data.html)
- [Counting (simple)](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/counting_simple/data.html)
- [Quantification](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/quantification/data.html)
- [Quantification (simple)](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/quantification_simple/data.html)
- [Combination](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/agreement/combination/data.html)

#### Classification datasets

- [Oneshape](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/classification/oneshape/data.html)
- [Multishape](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/classification/multishape/data.html)
- [Countshape](https://cdn.rawgit.com/AlexKuhnle/ShapeWorld/2932f4dd/examples/classification/countshape/data.html)



## Integration into Python code

The easiest way to use the ShapeWorld data in your Python project is to directly call it from the code. Whenever a batch of training/evaluation instances is required, `dataset.generate(...)` is called with the respective arguments. This means that generation happens simultaneously to training/testing. Below an example of how to generate a batch of 128 training instances. See also the example models below.

```python
from shapeworld import dataset

dataset = dataset(dtype='agreement', name='multishape')
generated = dataset.generate(n=128, mode='train', noise_range=0.1, include_model=True)

# given to the image caption agreement system
batch = (generated['world'], generated['caption'], generated['agreement'])

# can be used for more specific evaluation
world_model = generated['world_model']
caption_model = generated['caption_model']
```

{'fraction_non_alphanumeric': 0.10633706101628408, 'fraction_numerical': 0.014616441043751226, 'mean_word_length': 5.440303221730891, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 25, 'lorem ipsum': 0, 'www.': 6, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6355991', 'n_tokens_mistral': 3072, 'n_tokens_neox': 2758, 'n_words': 918}