<commit_before>.\"
.\" FreeBSD pkg - a next generation package for the installation and
.\" maintenance of non-core utilities.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\"
.\"     @(#)pkg-repository.5
.\" $FreeBSD$
.\"
.Dd November 18, 2014
.Dt PKG-REPOSITORY 5
.Os
.Sh NAME
.Nm "package repository"
.Nd format and operation of package repositories used by
.Xr pkg 8 .
.Sh DESCRIPTION
.Nm "Package repositories"
used by the
.Xr pkg 8
program consist of one or more collections of
.Dq package tarballs
together with package catalogues and optionally various other
collected package metadata.
.Pp
Each collection consists of packages suitable for installation on a
specific system
.Sy ABI :
a combination of operating system, CPU architecture, OS version, word
size, and for certain processors endianness or similar attributes.
.Pp
The package collections are typically made available to users for
download via a web or FTP server although various other means of access
may be employed.
Encoding the
.Sy ABI
value into the repository URL allows
.Nm pkg
to automatically select the correct package collection by expanding the
special token
.Cm ${ABI}
in
.Pa pkg.conf .
.Pp
Repositories may be mirrored over several sites:
.Nm pkg
has built-in support for discovering available mirrors dynamically
given a common URL by several mechanisms.
.Sh FILESYSTEM ORGANIZATION
Only very minimal constraints on repository layout are prescribed by
.Nm pkg .
The following constraints are all that must be met:
.Bl -bullet
.It
A repository may contain several package collections with parallel
.Cm REPOSITORY_ROOTs
in order to support diverse system
.Cm ABIs .
.It
All of the content for one
.Sy ABI
should be accessible in a filesystem or URL hierarchy beneath the
.Cm REPOSITORY_ROOT .
.It
All packages available beneath one
.Cm REPOSITORY_ROOT
should be binary compatible with a specific system
.Cm ABI .
.It
The repository catalogue is located at the apex of the
repository, at a specific location relative to the
.Cm REPOSITORY_ROOT .
.El
.Pp
Package catalogues contain the paths relative to the
.Cm REPOSITORY_ROOT
for each package, allowing the full URL for downloading the
package to be constructed.
.Pp
Where a package may be applicable to more than one
.Sy ABI
(e.g., it contains only text files) symbolic or hard links, URL mappings
or other techniques may be utilised to avoid duplication of storage.
.Pp
Although no specific filesystem organization is required, the usual
convention (inherited from
.Xr pkg-install 8 )
is to create a filesystem hierarchy thus:
.Bl -tag -width "REPOSITORY"
.It Pa $REPOSITORY_ROOT/All
One directory that contains every package available from the
repository for that
.Sy ABI .
Packages are stored as package tarballs identified by name and
version.
This directory may contain several different versions of each package
accumulated over time, but the repository catalogue will only record
the latest version for each distinct package origin.
.It Pa $REPOSITORY_ROOT/Latest/
May contains symbolic links to the latest versions of packages in the
.Pa All
directory.
Symbolic links contain a
.Sq latest link
style name only, without version.
As the whole
.Sq latest link
concept is rendered obsolete by
.Nm pkg ,
this will usually contain only the
.Pa pkg.txz
link, used for bootstrapping
.Nm pkg
itself on a new system.
.It Pa $REPOSITORY_ROOT/digests.txz
Package checksum data and optionally package signatures.
.It Pa $REPOSITORY_ROOT/packagesite.txz
Contains one
.Cm YAML
document, which is the concatenation of the
.Pa +MANIFEST
files from each of the packages in the repository.
This is used by
.Nm pkg-1.1
or later.
.It Pa $REPOSITORY_ROOT/repo.txz
(Deprecated).
Contains the package manifest data as above, but pre-loaded into
an SQLite database.
This is supplied for backwards compatibility with
.Nm pkg-1.0 .
.It Pa $REPOSITORY_ROOT/filesite.txz
(Optional).
Contains a YAML document listing all of the files contained in all
of the packages within the repository.
.Pp
The repository may optionally contain sub-directories corresponding to
the package origins within the
.Os
ports tree.
.El
.Pp
Each of the packages listed in the repository catalogue must have a
unique
.Cm origin .
There are no other constraints: package sets are not required to be
either complete (i.e., with all dependencies satisfied) or
self-consistent within a single repository.
.Sh REPOSITORY ACCESS METHODS<commit_msg>Remove documentation of digest.txz which does not exists anymore and remove bogus reference to YAML<commit_after>.\"
.\" FreeBSD pkg - a next generation package for the installation and
.\" maintenance of non-core utilities.
.\"
.\" Redistribution and use in source and binary forms, with or without
.\" modification, are permitted provided that the following conditions
.\" are met:
.\" 1. Redistributions of source code must retain the above copyright
.\"    notice, this list of conditions and the following disclaimer.
.\" 2. Redistributions in binary form must reproduce the above copyright
.\"    notice, this list of conditions and the following disclaimer in the
.\"    documentation and/or other materials provided with the distribution.
.\"
.\"
.\"     @(#)pkg-repository.5
.\" $FreeBSD$
.\"
.Dd December 29, 2014
.Dt PKG-REPOSITORY 5
.Os
.Sh NAME
.Nm "package repository"
.Nd format and operation of package repositories used by
.Xr pkg 8 .
.Sh DESCRIPTION
.Nm "Package repositories"
used by the
.Xr pkg 8
program consist of one or more collections of
.Dq package tarballs
together with package catalogues and optionally various other
collected package metadata.
.Pp
Each collection consists of packages suitable for installation on a
specific system
.Sy ABI :
a combination of operating system, CPU architecture, OS version, word
size, and for certain processors endianness or similar attributes.
.Pp
The package collections are typically made available to users for
download via a web or FTP server although various other means of access
may be employed.
Encoding the
.Sy ABI
value into the repository URL allows
.Nm pkg
to automatically select the correct package collection by expanding the
special token
.Cm ${ABI}
in
.Pa pkg.conf .
.Pp
Repositories may be mirrored over several sites:
.Nm pkg
has built-in support for discovering available mirrors dynamically
given a common URL by several mechanisms.
.Sh FILESYSTEM ORGANIZATION
Only very minimal constraints on repository layout are prescribed by
.Nm pkg .
The following constraints are all that must be met:
.Bl -bullet
.It
A repository may contain several package collections with parallel
.Cm REPOSITORY_ROOTs
in order to support diverse system
.Cm ABIs .
.It
All of the content for one
.Sy ABI
should be accessible in a filesystem or URL hierarchy beneath the
.Cm REPOSITORY_ROOT .
.It
All packages available beneath one
.Cm REPOSITORY_ROOT
should be binary compatible with a specific system
.Cm ABI .
.It
The repository catalogue is located at the apex of the
repository, at a specific location relative to the
.Cm REPOSITORY_ROOT .
.El
.Pp
Package catalogues contain the paths relative to the
.Cm REPOSITORY_ROOT
for each package, allowing the full URL for downloading the
package to be constructed.
.Pp
Where a package may be applicable to more than one
.Sy ABI
(e.g., it contains only text files) symbolic or hard links, URL mappings
or other techniques may be utilised to avoid duplication of storage.
.Pp
Although no specific filesystem organization is required, the usual
convention (inherited from
.Xr pkg-install 8 )
is to create a filesystem hierarchy thus:
.Bl -tag -width "REPOSITORY"
.It Pa $REPOSITORY_ROOT/All
One directory that contains every package available from the
repository for that
.Sy ABI .
Packages are stored as package tarballs identified by name and
version.
This directory may contain several different versions of each package
accumulated over time, but the repository catalogue will only record
the latest version for each distinct package origin.
.It Pa $REPOSITORY_ROOT/Latest/
May contains symbolic links to the latest versions of packages in the
.Pa All
directory.
Symbolic links contain a
.Sq latest link
style name only, without version.
As the whole
.Sq latest link
concept is rendered obsolete by
.Nm pkg ,
this will usually contain only the
.Pa pkg.txz
link, used for bootstrapping
.Nm pkg
itself on a new system.
.It Pa $REPOSITORY_ROOT/packagesite.txz
Contains one
.Cm JSON
document, which is the concatenation of the
.Pa +MANIFEST
files from each of the packages in the repository.
This is used by
.Nm pkg-1.1
or later.
.It Pa $REPOSITORY_ROOT/repo.txz
(Deprecated).
Contains the package manifest data as above, but pre-loaded into
an SQLite database.
This is supplied for backwards compatibility with
.Nm pkg-1.0 .
.It Pa $REPOSITORY_ROOT/filesite.txz
(Optional).
Contains a YAML document listing all of the files contained in all
of the packages within the repository.
.Pp
The repository may optionally contain sub-directories corresponding to
the package origins within the
.Os
ports tree.
.El
.Pp
Each of the packages listed in the repository catalogue must have a
unique
.Cm origin .
There are no other constraints: package sets are not required to be
either complete (i.e., with all dependencies satisfied) or
self-consistent within a single repository.
.Sh REPOSITORY ACCESS METHODS
{'fraction_non_alphanumeric': 0.050106935533150015, 'fraction_numerical': 0.0034626744067623996, 'mean_word_length': 4.228966986155484, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1510201', 'n_tokens_mistral': 2752, 'n_tokens_neox': 2549, 'n_words': 1457}