license:apache-2.0 arxiv:2305.08144 region:us
---
license: apache-2.0
---

# WikiHow Task Set

WikiHow task set is an InfoUI interaction task set based on
[Mobile-Env](https://github.com/X-LANCE/Mobile-Env) proposed in [*Mobile-Env:
An Evaluation Platform and Benchmark for Interactive Agents in LLM
Era*](https://arxiv.org/abs/2305.08144).
[WikiHow](https://www.wikihow.com/Main-Page) is a collaborative wiki site about
various real-life tips with more than 340,000 online articles. To construct the
task set, 107,448 pages are crawled, and the dumped website data occupy about
88 GiB totally.

Several task definition templates are designed according to the functions of
WikiHow app and 5,522 task definitions are instantiated through the template
toolkit in Mobile-Env. This task set is named the *extended set*
(`wikihow-extended.tar.xz`). There may be several faults that may make the
system or the task fail in the auto-generated tasks. Therefore, 178 tasks are
sampled from the extended set and have been verified by human beings to ensure
correctness and stability, which is named the *canonical set*
(`wikihow-canonical.tar.xz`). Owing to the limit of the budgets, only 70 tasks
are tested using the proposed LLM-based agent in the corresponding pager.
These 70 tasks are given in `wikihow-microcanon.tar.xz`. We call it the
*canonical subset* or the *micro canonical set*.

### Website Data Replay

The replay script for [mitmproxy](https://mitmproxy.org/) is given as
`replay_url.py`. To use this replay script, the information retrieval tool
[Pyserini](https://github.com/castorini/pyserini/) is required. Four parameters
are expected to be assigned in the script:

+ The crawled data from WikiHow website (`dumps` in `wikihow.data.tar.xz`)
+ The HTML templates used to mock the search result page (`templates` in
  `wikihow.data.tar.xz`)
+ The indices for the search engine based on Pyserini (`indices-t/indices` in
  `wikihow.data.tar.xz`)
+ The metadata of the crawled articles (`indices-t/docs/doc_meta.csv` in
  `wikihow.data.tar.xz`)

All the required data are offered in `wikihow.data.tar.xz`. (The archive is
about 78 GiB. And the decompressed data are about 88 GiB.) The archive is split
into two pieces (`wikihow.data.tar.xz.00` and `wikihow.data.tar.xz.01`). You
can use `cat` to concatenate them:

```sh
cat wikihow.data.tar.xz.00 wikihow.data.tar.xz.01 >wikihow.data.tar.xz
```

The SHA256 checksums are provided in `wikihow.data.tar.xz.sha256` to check the
integrity.

To run the script:

```sh
mitmproxy --showhost -s replay_url.py
```

### Certificate Unpinning Plan

The `syscert` plan proposed by Mobile-Env works for WikiHow app. You can
complete the config according to the [guideline of
Mobile-Env](https://github.com/X-LANCE/Mobile-Env/blob/master/docs/dynamic-app-en.md).
The available APK package from [APKCombo](https://apkcombo.com/) is provided.
And note to use the AVD image of version Android 11.0 (API Level 30) (Google
APIs) to obtain the best compatibility and the root-enabled ADBD.

### Human-Rewritten Instructions

Human-rewritten instructions for the *canonical set* are release under
`instruction_rewriting/`. An AndroidEnv wrapper `InstructionRewritingWrapper`
is provided to load the rewritten instructions (`merged_doccano.json`) and
public patterns (`pattern-*.txt`). The annotations are collected via
[doccano](https://doccano.github.io/doccano/). The patterns are parsed by
[`sentence_pattern.py`](instruction_rewriting/sentence_pattern.py).
