language:en license:mit region:us
---
license: mit
language:
- en
---

Microlang was designed to test text generation architectures

It consists of 16 tokens:
  1. special (implicit):

    <> 

  2. noun:
  
    bob, tom, bike, speech
  
  3. transitive verb:
  
    take, use
  
  4. intransitive verb: 

    talk, go
  
  5. adjective:
  
    good, active
  
  6. adverb:
  
    not
  
  7. conjunction:
  
    and, then, but
    
  8. punctuation:
 
    .


The tokenizer can be found on `umarzein/microlang-utils` and can be loaded this:

```python
import transformers
tokenizer = transformers.PreTrainedTokenizerFast.from_pretrained("umarzein/microlang-utils")
```

