<commit_before>Test data syntax
----------------

This section covers Robot Framework's overall test data
syntax. The following sections will explain how to actually create test
cases, test suites and so on.

.. contents:: 
   :depth: 2
   :local: 


Files and directories
~~~~~~~~~~~~~~~~~~~~~

The hierarchical structure for arranging test cases is built as follows:

- Test cases are created in `test case files`_.
- A test case file automatically creates a `test suite`_ containing
  the test cases in that file.
- A directory containing test case files forms a higher-level test
  suite. Such a `test suite directory`_ has suites created from test
  case files as its sub test suites.
- A test suite directory can also contain other test suite directories,
  and this hierarchical structure can be as deeply nested as needed.
- Test suite directories can have a special `initialization file`_.

In addition to this, there are:

- `Test libraries`_ containing the lowest-level keywords
- `Resource files`_ with variables_ and higher-level `user keywords`_
- `Variable files`_ to provide more flexible ways to create variables
  than resource files 


Supported file formats
~~~~~~~~~~~~~~~~~~~~~~

Robot Framework test data is defined in tabular format, using either
the HTML (hypertext markup language) or TSV (tab-separated values)
format. Robot Framework selects a parser for the test data based on
the file extension. The extension is case-insensitive, and the
recognized extensions are :path:`.html`, :path:`.htm` and
:path:`.xhtml` for HTML, and :path:`.tsv` for TSV.

Different `test data templates`_ are available to make it easier to
get started writing tests.


HTML format
'''''''''''

In HTML files, the test data is defined in separate tables (see the
example below). Robot Framework recognizes these `test data tables`_
based on the text in their first cell. Everything outside recognized
tables is ignored.

.. table:: Using the HTML format
   :class: example

   ============  ================  =======  =======
      Setting          Value        Value    Value 
   ============  ================  =======  =======
   Library       OperatingSystem   \        \
   \             \                 \        \
   ============  ================  =======  =======

.. table::
   :class: example

   ============  ================  =======  =======
     Variable        Value          Value    Value
   ============  ================  =======  =======
   ${MESSAGE}    Hello, world!     \        \
   \             \                 \        \
   ============  ================  =======  =======

.. table::
   :class: example

   ============  ===================  ============  =============
    Test Case           Action          Argument      Argument 
   ============  ===================  ============  =============
   My Test       [Documentation]      Example test  \
   \             Log                  ${MESSAGE}    \
   \             My Keyword           /tmp          \
   \             \                    \             \
   Another Test  Should Be Equal      ${MESSAGE}    Hello, world!
   ============  ===================  ============  =============
   
.. table::
   :class: example

   ============  ======================  ============  ==========
     Keyword            Action             Argument     Argument
   ============  ======================  ============  ==========
   My Keyword    [Argument]              ${path}       \
   \             Directory Should Exist  ${path}       \
   ============  ======================  ============  ==========


Test data in HTML files can be edited with whichever editor you
prefer, but a graphic editor, where you can actually see the tables,
is recommended.  There is also a tool called `Robot IDE`_
available that is actually designed for editing the test data.

HTML entity references (for example, :code:`&auml;`) are
supported. Additionally, any encoding can be used, assuming that it is
specified in the data file. Normal HTML files must use the META
element as in the example below::

 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

XHTML files should use the XML preamble as in this example::
 
 <?xml version="1.0" encoding="Big5"?>

If no encoding is specified, Robot Framework uses ISO-8859-1 by default. 

.. Note:: Before Robot Framework version 1.8.4, the HTML test data was
          always assumed to be in ASCII format.


TSV format
''''''''''

In a TSV file, all the data is in one large table. `Test data tables`_
are recognized from an asterisk (*), followed by a normal table
name. Robot Framework ignores any other asterisks in the name, so it
can also be prefixed with another asterisk as in the example below.
In TSV test data, everything before the first recognized table is
ignored in Robot Framework.

.. table:: Using the TSV format
   :class: tsv-example

   ============  =======================  =============  =============
   \*Setting*    \*Value*                 \*Value*       \*Value*
   Library       OperatingSystem          \              \
   \             \                        \              \ 
   \             \                        \              \
   \*Variable*   \*Value*                 \*Value*       \*Value*
   ${MESSAGE}    Hello, world!            \              \
   \             \                        \              \
   \             \                        \              \
   \*Test Case*  \*Action*                \*Argument*    \*Argument*
   My Test       [Documentation]          Example test   \
   \             Log                      ${MESSAGE}     \
   \             My Keyword               /tmp           \
   \             \                        \              \
   Another Test  Should Be Equal          ${MESSAGE}     Hello, world!
   \             \                        \              \ 
   \             \                        \              \
   \*Keyword*    \*Action*                \*Argument*    \*Argument*
   My Keyword    [Argument]               ${path}        \
   \             Directory Should Exist   ${path}        \
   ============  =======================  =============  =============

You can create and edit TSV files in any spreadsheet program, such as
Microsoft Excel. Select the tab-separated format when you save the
file and remember to set the file extension to :path:`.tsv`. The TSV
format can be used in Robot Framework's test data for all the same
purposes as HTML.

Robot Framework parses TSV data by first splitting all the content
into rows and then rows into cells on the basis of the tabular
characters. Spreadsheet programs sometimes surround cells with quotes
(for example, :code:`"my value"`) and Robot Framework removes
them. Possible quotes inside the data are doubled (for example,
:code:`"my ""quoted"" value"`) and Robot Framework handles also
them. If you are using a spreadsheet program to create TSV data, you
do not need to pay attention to that, but if you create data
programmatically, you have to add the same quotes as spreadsheets.


Test data tables
~~~~~~~~~~~~~~~~

Test data is structured in four types of tables listed below. These
test data tables are identified by the first cell of the table, and
the last column in the table below lists different aliases that can be
used as a table name.

.. table:: Different test data tables
   :class: tabular<commit_msg>1) added explanation that tsv files are expected to use utf-8 (issue 211), 2) added subtitles, 3) some cleanup<commit_after>Test data syntax
----------------

This section covers Robot Framework's overall test data
syntax. The following sections will explain how to actually create test
cases, test suites and so on.

.. contents:: 
   :depth: 2
   :local: 


Files and directories
~~~~~~~~~~~~~~~~~~~~~

The hierarchical structure for arranging test cases is built as follows:

- Test cases are created in `test case files`_.
- A test case file automatically creates a `test suite`_ containing
  the test cases in that file.
- A directory containing test case files forms a higher-level test
  suite. Such a `test suite directory`_ has suites created from test
  case files as its sub test suites.
- A test suite directory can also contain other test suite directories,
  and this hierarchical structure can be as deeply nested as needed.
- Test suite directories can have a special `initialization file`_.

In addition to this, there are:

- `Test libraries`_ containing the lowest-level keywords
- `Resource files`_ with variables_ and higher-level `user keywords`_
- `Variable files`_ to provide more flexible ways to create variables
  than resource files 


Supported file formats
~~~~~~~~~~~~~~~~~~~~~~

Robot Framework test data is defined in tabular format, using either
the HTML (hypertext markup language) or TSV (tab-separated values)
format. Robot Framework selects a parser for the test data based on
the file extension. The extension is case-insensitive, and the
recognized extensions are :path:`.html`, :path:`.htm` and
:path:`.xhtml` for HTML, and :path:`.tsv` for TSV.

Different `test data templates`_ are available to make it easier to
get started writing tests.


HTML format
'''''''''''

In HTML files, the test data is defined in separate tables (see the
example below). Robot Framework recognizes these `test data tables`_
based on the text in their first cell. Everything outside recognized
tables is ignored.

.. table:: Using the HTML format
   :class: example

   ============  ================  =======  =======
      Setting          Value        Value    Value 
   ============  ================  =======  =======
   Library       OperatingSystem   \        \
   \             \                 \        \
   ============  ================  =======  =======

.. table::
   :class: example

   ============  ================  =======  =======
     Variable        Value          Value    Value
   ============  ================  =======  =======
   ${MESSAGE}    Hello, world!     \        \
   \             \                 \        \
   ============  ================  =======  =======

.. table::
   :class: example

   ============  ===================  ============  =============
    Test Case           Action          Argument      Argument 
   ============  ===================  ============  =============
   My Test       [Documentation]      Example test  \
   \             Log                  ${MESSAGE}    \
   \             My Keyword           /tmp          \
   \             \                    \             \
   Another Test  Should Be Equal      ${MESSAGE}    Hello, world!
   ============  ===================  ============  =============
   
.. table::
   :class: example

   ============  ======================  ============  ==========
     Keyword            Action             Argument     Argument
   ============  ======================  ============  ==========
   My Keyword    [Argument]              ${path}       \
   \             Directory Should Exist  ${path}       \
   ============  ======================  ============  ==========


Editing test data
`````````````````

Test data in HTML files can be edited with whichever editor you
prefer, but a graphic editor, where you can actually see the tables,
is recommended.  There is also a tool called `Robot IDE`_
available that is actually designed for editing the test data.


Encoding and entity references
``````````````````````````````

HTML entity references (for example, :code:`&auml;`) are
supported. Additionally, any encoding can be used, assuming that it is
specified in the data file. Normal HTML files must use the META
element as in the example below::

  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">

XHTML files should use the XML preamble as in this example::
 
  <?xml version="1.0" encoding="Big5"?>

If no encoding is specified, Robot Framework uses ISO-8859-1 by default. 


TSV format
''''''''''

The TSV format can be used in Robot Framework's test data for all the
same purposes as HTML. In a TSV file, all the data is in one large
table. `Test data tables`_ are recognized from one or more asterisks
(:code:`*`), followed by a normal table name and an optional closing
asterisks.  Everything before the first recognized table is ignored
similarly as data outside tables in HTML data.

.. table:: Using the TSV format
   :class: tsv-example

   ============  =======================  =============  =============
   \*Setting*    \*Value*                 \*Value*       \*Value*
   Library       OperatingSystem          \              \
   \             \                        \              \ 
   \             \                        \              \
   \*Variable*   \*Value*                 \*Value*       \*Value*
   ${MESSAGE}    Hello, world!            \              \
   \             \                        \              \
   \             \                        \              \
   \*Test Case*  \*Action*                \*Argument*    \*Argument*
   My Test       [Documentation]          Example test   \
   \             Log                      ${MESSAGE}     \
   \             My Keyword               /tmp           \
   \             \                        \              \
   Another Test  Should Be Equal          ${MESSAGE}     Hello, world!
   \             \                        \              \ 
   \             \                        \              \
   \*Keyword*    \*Action*                \*Argument*    \*Argument*
   My Keyword    [Argument]               ${path}        \
   \             Directory Should Exist   ${path}        \
   ============  =======================  =============  =============


Editing test data
`````````````````

You can create and edit TSV files in any spreadsheet program, such as
Microsoft Excel. Select the tab-separated format when you save the
file and remember to set the file extension to :path:`.tsv`. It is
also a good idea to turn all automatic corrections off and configure
the tool to treat all values in the file as plain text. The TSV format
is also relatively easy to edit with any text editor, especially if
the editor supports visually separating tabs from spaces.

Robot Framework parses TSV data by first splitting all the content
into rows and then rows into cells on the basis of the tabular
characters. Spreadsheet programs sometimes surround cells with quotes
(for example, :code:`"my value"`) and Robot Framework removes
them. Possible quotes inside the data are doubled (for example,
:code:`"my ""quoted"" value"`) and also this is handled correctly.  If
you are using a spreadsheet program to create TSV data, you should not
need to pay attention to this, but if you create data
programmatically, you have to follow the same quoting conventions as
spreadsheets.


Encoding
````````

TSV files are always expected to use UTF-8 encoding. Because ASCII is
a subset of UTF-8, plain ASCII is naturally supported too.


Test data tables
~~~~~~~~~~~~~~~~

Test data is structured in four types of tables listed below. These
test data tables are identified by the first cell of the table, and
the last column in the table below lists different aliases that can be
used as a table name.

.. table:: Different test data tables
   :class: tabular
{'fraction_non_alphanumeric': 0.163852122318576, 'fraction_numerical': 0.00208645758622938, 'mean_word_length': 2.1716294458229943, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 2, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 2}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 42, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '975131', 'n_tokens_mistral': 3978, 'n_tokens_neox': 3490, 'n_words': 1632}