<commit_before>.\" Copyright (c) 1989, 1991, 1993
.\"	The Regents of the University of California.  All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\" 3. All advertising materials mentioning features or use of this software
.\"    must display the following acknowledgement:
.\"	This product includes software developed by the University of
.\"	California, Berkeley and its contributors.
.\" 4. Neither the name of the University nor the names of its contributors
.\"    may be used to endorse or promote products derived from this software
.\"    without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\"     From: @(#)vis.3	8.1 (Berkeley) 6/9/93
.\" $FreeBSD$
.\"
.Dd July 25, 1996
.Dt VIS 3
.Os
.Sh NAME
.Nm vis
.Nd visually encode characters
.Sh LIBRARY
.Lb libc
.Sh SYNOPSIS
.Fd #include <vis.h>
.Ft char *
.Fn vis "char *dst" "int c" "int flag" "int nextc"
.Ft int
.Fn strvis "char *dst" "const char *src" "int flag"
.Ft int
.Fn strvisx "char *dst" "const char *src" "size_t len" "int flag"
.Sh DESCRIPTION
The
.Fn vis
function
copies into
.Fa dst
a string which represents the character
.Fa c .
If
.Fa c
needs no encoding, it is copied in unaltered.  The string is
null terminated, and a pointer to the end of the string is
returned.  The maximum length of any encoding is four
characters (not including the trailing
.Dv NUL ) ;
thus, when
encoding a set of characters into a buffer, the size of the buffer should
be four times the number of characters encoded, plus one for the trailing
.Dv NUL .
The flag parameter is used for altering the default range of
characters considered for encoding and for altering the visual
representation.
The additional character,
.Fa nextc ,
is only used when selecting the
.Dv VIS_CSTYLE
encoding format (explained below).
.Pp
The
.Fn strvis
and
.Fn strvisx
functions copy into
.Fa dst
a visual representation of
the string
.Fa src .
The
.Fn strvis
function encodes characters from
.Fa src
up to the
first
.Dv NUL .
The
.Fn strvisx
function encodes exactly
.Fa len
characters from
.Fa src
(this
is useful for encoding a block of data that may contain
.Dv NUL Ns 's).
Both forms
.Dv NUL
terminate
.Fa dst .
The size of
.Fa dst
must be four times the number
of characters encoded from
.Fa src
(plus one for the
.Dv NUL ) .
Both
forms return the number of characters in dst (not including
the trailing
.Dv NUL ) .
.Pp
The encoding is a unique, invertible representation composed entirely of
graphic characters; it can be decoded back into the original form using
the
.Xr unvis 3
or
.Xr strunvis 3
functions.
.Pp
There are two parameters that can be controlled: the range of
characters that are encoded, and the type
of representation used.
By default, all non-graphic characters.
except space, tab, and newline are encoded.
(See
.Xr isgraph 3 . )
The following flags
alter this:
.Bl -tag -width VIS_WHITEX
.It Dv VIS_SP
Also encode space.
.It Dv VIS_TAB
Also encode tab.
.It Dv VIS_NL
Also encode newline.
.It Dv VIS_WHITE
Synonym for
.Dv VIS_SP
\&|
.Dv VIS_TAB
\&|
.Dv VIS_NL .
.It Dv VIS_SAFE
Only encode "unsafe" characters.  Unsafe means control
characters which may cause common terminals to perform
unexpected functions.  Currently this form allows space,
tab, newline, backspace, bell, and return - in addition
to all graphic characters - unencoded.
.El
.Pp
There are three forms of encoding.
All forms use the backslash character
.Ql \e
to introduce a special
sequence; two backslashes are used to represent a real backslash.
These are the visual formats:
.Bl -tag -width VIS_HTTPSTYLE
.It (default)
Use an
.Ql M
to represent meta characters (characters with the 8th
bit set), and use carat<commit_msg>There are four types of encoding now, not three.  Most of them use the backslash as a special char, but not all.<commit_after>.\" Copyright (c) 1989, 1991, 1993
.\"	The Regents of the University of California.  All rights reserved.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\" 3. All advertising materials mentioning features or use of this software
.\"    must display the following acknowledgement:
.\"	This product includes software developed by the University of
.\"	California, Berkeley and its contributors.
.\" 4. Neither the name of the University nor the names of its contributors
.\"    may be used to endorse or promote products derived from this software
.\"    without specific prior written permission.
.\"
.\" THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND
.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE
.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
.\" SUCH DAMAGE.
.\"
.\"     From: @(#)vis.3	8.1 (Berkeley) 6/9/93
.\" $FreeBSD$
.\"
.Dd July 25, 1996
.Dt VIS 3
.Os
.Sh NAME
.Nm vis
.Nd visually encode characters
.Sh LIBRARY
.Lb libc
.Sh SYNOPSIS
.Fd #include <vis.h>
.Ft char *
.Fn vis "char *dst" "int c" "int flag" "int nextc"
.Ft int
.Fn strvis "char *dst" "const char *src" "int flag"
.Ft int
.Fn strvisx "char *dst" "const char *src" "size_t len" "int flag"
.Sh DESCRIPTION
The
.Fn vis
function
copies into
.Fa dst
a string which represents the character
.Fa c .
If
.Fa c
needs no encoding, it is copied in unaltered.  The string is
null terminated, and a pointer to the end of the string is
returned.  The maximum length of any encoding is four
characters (not including the trailing
.Dv NUL ) ;
thus, when
encoding a set of characters into a buffer, the size of the buffer should
be four times the number of characters encoded, plus one for the trailing
.Dv NUL .
The flag parameter is used for altering the default range of
characters considered for encoding and for altering the visual
representation.
The additional character,
.Fa nextc ,
is only used when selecting the
.Dv VIS_CSTYLE
encoding format (explained below).
.Pp
The
.Fn strvis
and
.Fn strvisx
functions copy into
.Fa dst
a visual representation of
the string
.Fa src .
The
.Fn strvis
function encodes characters from
.Fa src
up to the
first
.Dv NUL .
The
.Fn strvisx
function encodes exactly
.Fa len
characters from
.Fa src
(this
is useful for encoding a block of data that may contain
.Dv NUL Ns 's).
Both forms
.Dv NUL
terminate
.Fa dst .
The size of
.Fa dst
must be four times the number
of characters encoded from
.Fa src
(plus one for the
.Dv NUL ) .
Both
forms return the number of characters in dst (not including
the trailing
.Dv NUL ) .
.Pp
The encoding is a unique, invertible representation composed entirely of
graphic characters; it can be decoded back into the original form using
the
.Xr unvis 3
or
.Xr strunvis 3
functions.
.Pp
There are two parameters that can be controlled: the range of
characters that are encoded, and the type
of representation used.
By default, all non-graphic characters.
except space, tab, and newline are encoded.
(See
.Xr isgraph 3 . )
The following flags
alter this:
.Bl -tag -width VIS_WHITEX
.It Dv VIS_SP
Also encode space.
.It Dv VIS_TAB
Also encode tab.
.It Dv VIS_NL
Also encode newline.
.It Dv VIS_WHITE
Synonym for
.Dv VIS_SP
\&|
.Dv VIS_TAB
\&|
.Dv VIS_NL .
.It Dv VIS_SAFE
Only encode "unsafe" characters.  Unsafe means control
characters which may cause common terminals to perform
unexpected functions.  Currently this form allows space,
tab, newline, backspace, bell, and return - in addition
to all graphic characters - unencoded.
.El
.Pp
There are four forms of encoding.
Most forms use the backslash character
.Ql \e
to introduce a special
sequence; two backslashes are used to represent a real backslash.
These are the visual formats:
.Bl -tag -width VIS_HTTPSTYLE
.It (default)
Use an
.Ql M
to represent meta characters (characters with the 8th
bit set), and use carat
{'fraction_non_alphanumeric': 0.07014370415033665, 'fraction_numerical': 0.006833484071952568, 'mean_word_length': 3.7640019147917663, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '2084605', 'n_tokens_mistral': 3119, 'n_tokens_neox': 2806, 'n_words': 1554}