<commit_before>.\" Copyright (c) 2001, <NAME>.  Terms and conditions are those of
.\" the BSD Copyright as specified in the file "/usr/src/COPYRIGHT" in
.\" the source tree.
.\"
.\" $FreeBSD$
.\"
.Dd June 25, 2002
.Dt TUNING 7
.Os
.Sh NAME
.Nm tuning
.Nd performance tuning under FreeBSD
.Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP
When using
.Xr bsdlabel 8
or
.Xr sysinstall 8
to lay out your file systems on a hard disk it is important to remember
that hard drives can transfer data much more quickly from outer tracks
than they can from inner tracks.
To take advantage of this you should
try to pack your smaller file systems and swap closer to the outer tracks,
follow with the larger file systems, and end with the largest file systems.
It is also important to size system standard file systems such that you
will not be forced to resize them later as you scale the machine up.
I usually create, in order, a 128M root, 1G swap, 128M
.Pa /var ,
128M
.Pa /var/tmp ,
3G
.Pa /usr ,
and use any remaining space for
.Pa /home .
.Pp
You should typically size your swap space to approximately 2x main memory.
If you do not have a lot of RAM, though, you will generally want a lot
more swap.
It is not recommended that you configure any less than
256M of swap on a system and you should keep in mind future memory
expansion when sizing the swap partition.
The kernel's VM paging algorithms are tuned to perform best when there is
at least 2x swap versus main memory.
Configuring too little swap can lead
to inefficiencies in the VM page scanning code as well as create issues
later on if you add more memory to your machine.
Finally, on larger systems
with multiple SCSI disks (or multiple IDE disks operating on different
controllers), we strongly recommend that you configure swap on each drive.
The swap partitions on the drives should be approximately the same size.
The kernel can handle arbitrary sizes but
internal data structures scale to 4 times the largest swap partition.
Keeping
the swap partitions near the same size will allow the kernel to optimally
stripe swap space across the N disks.
Do not worry about overdoing it a
little, swap space is the saving grace of
.Ux
and even if you do not normally use much swap, it can give you more time to
recover from a runaway program before being forced to reboot.
.Pp
How you size your
.Pa /var
partition depends heavily on what you intend to use the machine for.
This
partition is primarily used to hold mailboxes, the print spool, and log
files.
Some people even make
.Pa /var/log
its own partition (but except for extreme cases it is not worth the waste
of a partition ID).
If your machine is intended to act as a mail
or print server,
or you are running a heavily visited web server, you should consider
creating a much larger partition \(en perhaps a gig or more.
It is very easy
to underestimate log file storage requirements.
.Pp
Sizing
.Pa /var/tmp
depends on the kind of temporary file usage you think you will need.
128M is
the minimum we recommend.
Also note that sysinstall will create a
.Pa /tmp
directory.
Dedicating a partition for temporary file storage is important for
two reasons: first, it reduces the possibility of file system corruption
in a crash, and second it reduces the chance of a runaway process that
fills up
.Oo Pa /var Oc Ns Pa /tmp
from blowing up more critical subsystems (mail,
logging, etc).
Filling up
.Oo Pa /var Oc Ns Pa /tmp
is a very common problem to have.
.Pp
In the old days there were differences between
.Pa /tmp
and
.Pa /var/tmp ,
but the introduction of
.Pa /var
(and
.Pa /var/tmp )
led to massive confusion
by program writers so today programs haphazardly use one or the
other and thus no real distinction can be made between the two.
So it makes sense to have just one temporary directory and
softlink to it from the other
.Pa tmp
directory locations.
However you handle
.Pa /tmp ,
the one thing you do not want to do is leave it sitting
on the root partition where it might cause root to fill up or possibly
corrupt root in a crash/reboot situation.
.Pp
The
.Pa /usr
partition holds the bulk of the files required to support the system and
a subdirectory within it called
.Pa /usr/local
holds the bulk of the files installed from the
.Xr ports 7
hierarchy.
If you do not use ports all that much and do not intend to keep
system source
.Pq Pa /usr/src
on the machine, you can get away with
a 1 gigabyte
.Pa /usr
partition.
However, if you install a lot of ports
(especially window managers and Linux-emulated binaries), we recommend
at least a 2 gigabyte
.Pa /usr
and if you also intend to keep system source
on the machine, we recommend a 3 gigabyte
.Pa /usr .
Do not underestimate the
amount of space you will need in this partition, it can creep up and
surprise you!
.Pp
The
.Pa /home
partition is typically used to hold user-specific data.
I usually size it to the remainder of the disk.
.Pp
Why partition at all?
Why not create one big
.Pa /
partition and be done with it?
Then I do not have to worry about undersizing things!
Well, there are several reasons this is not a good idea.
First,
each partition has different operational characteristics and separating them
allows the file system to tune itself to those characteristics.
For example,
the root and
.Pa /usr
partitions are read-mostly, with very little writing, while
a lot of reading and writing could occur in
.Pa /var
and
.Pa /var/tmp .
By properly
partitioning your system fragmentation introduced in the smaller more
heavily write-loaded partitions will not bleed over into the mostly-read
partitions.
Additionally, keeping the write-loaded partitions closer to
the edge of the disk (i.e., before the really big partitions instead of after
in the partition table) will increase I/O performance in the partitions
where you need it the most.
Now it is true that you might also need I/O
performance in the larger partitions, but they are so large that shifting
them more towards the edge of the disk will not lead to a significant
performance improvement whereas moving
.Pa /var
to the edge can have a huge impact.
Finally, there are safety concerns.
Having a small neat root partition that
is essentially read-only gives it a greater chance of surviving a bad crash
intact.
.Pp
Properly partitioning your system also allows you to tune
.Xr newfs 8 ,
and
.Xr tunefs 8
parameters.
Tuning
.Xr newfs 8
requires more experience but can lead to significant improvements in
performance.
There are three parameters that are relatively safe to tune:
.Em blocksize , bytes/i-node ,
and
.Em cylinders/group .
.Pp
.Fx
performs best when using 8K or 16K file system block sizes.
The default file system block size is 16K,
which provides best performance for most applications,
with the exception of those that perform random access on large files
(such as database server software).
Such applications tend to perform better with a smaller block size,
although modern disk characteristics are such that the performance
gain from using a smaller block size may not be worth consideration.
Using a block size larger than 16K
can cause fragmentation of the buffer cache and
lead to lower performance.
.Pp
The defaults may be unsuitable
for a file system that requires a very large number of i-nodes
or is intended to hold a large number of very small files.
Such a file system should be created with an 8K or 4K block size.
This also requires you to specify a smaller
fragment size.
We recommend always using a fragment size that is 1/8
the block size (less testing has been done on other fragment size factors).
The
.Xr newfs 8
options for this would be
.Dq Li "newfs -f 1024 -b 8192 ..." .
.Pp
If a large partition is intended to be used to hold fewer, larger files, such
as database files, you can increase the
.Em bytes/i-node
ratio which reduces the number of i-nodes (maximum number of files and
directories that can be created) for that partition.
Decreasing the number
of i-nodes in a file system can greatly reduce
.Xr fsck 8
recovery times after a crash.
Do not use this option
unless you are actually storing large files on the partition, because if you
overcompensate you can wind up with a file system that has lots of free
space remaining but cannot accommodate any more files.
Using 32768, 65536, or 262144 bytes/i-node is recommended.
You can go higher but
it will have only incremental effects on
.Xr fsck 8
recovery times.
For example,
.Dq Li "newfs -i 32768 ..." .
.Pp
.Xr tunefs 8
may be used to further tune a file system.
This command can be run in
single-user mode without having to reformat the file system.
However, this is possibly the most abused program in the system.
Many people attempt to
increase available file system space by setting the min-free percentage to 0.
This can lead to severe file system fragmentation and we do not recommend
that you do this.
Really the only
.Xr tunefs 8
option worthwhile here is turning on
.Em softupdates
with
.Dq Li "tunefs -n enable /filesystem" .
(Note: in
.Fx 4.5
and later, softupdates can be turned on using the
.Fl U
option to
.Xr newfs 8 ,
and
.Xr sysinstall 8
will typically enable softupdates automatically for non-root file systems).
Softupdates drastically improves meta-data performance, mainly file
creation and deletion.
We recommend enabling softupdates on most file systems; however, there
are two limitations to softupdates that you should be aware of when
determining whether to use it on a file system.
First, softupdates guarantees file system consistency in the
case of a crash but could very easily be several seconds (even a minute!\&)
behind on pending write to the physical disk.
If you crash you may lose more work
than otherwise.
Secondly, softupdates delays the freeing of file system
blocks.
If you have a file system (such as the root file system) which is
close to full, doing a major update of it, e.g.\&
.Dq Li "make installworld" ,
can run it out of space and cause the update to fail.
For this reason, softupdates will not be enabled on the root file system
during a typical install.
There is no loss of performance since the root
file system is rarely written to.
.Pp
A number of run-time
.Xr mount 8
options exist that can help you tune the system.
The most obvious and most dangerous one is
.Cm async .
Do not ever use it; it is far too dangerous.
A less dangerous and more
useful
.Xr mount 8
option is called
.Cm noatime .
.Ux
file systems normally update the last-accessed time of a file or
directory whenever it is accessed.
This operation is handled in
.Fx
with a delayed write and normally does not create a burden on the system.
However, if your system is accessing a huge number of files on a continuing
basis the buffer cache can wind up getting polluted with atime updates,
creating a burden on the system.
For example, if you are running a heavily
loaded web site, or a news server with lots of readers, you might want to
consider turning off atime updates on your larger partitions with this
.Xr mount 8
option.
However, you should not gratuitously turn off atime
updates everywhere.
For example, the
.Pa /var
file system customarily
holds mailboxes, and atime (in combination with mtime) is used to
determine whether a mailbox has new mail.
You might as well leave
atime turned on for mostly read-only partitions such as
.Pa /
and
.Pa /usr
as well.
This is especially useful for
.Pa /
since some system utilities
use the atime field for reporting.
.Sh STRIPING DISKS
In larger systems you can stripe partitions from several drives together
to create a much larger overall partition.
Striping can also improve
the performance of a file system by splitting I/O operations across two
or more disks.
The
.Xr vinum 8
and
.Xr ccdconfig 8
utilities may be used to create simple striped file systems.
Generally
speaking, striping smaller partitions such as the root and
.Pa /var/tmp ,
or essentially read-only partitions such as
.Pa /usr
is a complete waste of time.
You should only stripe partitions that require serious I/O performance,
typically
.Pa /var , /home ,
or custom partitions used to hold databases and web pages.
Choosing the proper stripe size is also
important.
File systems tend to store meta-data on power-of-2 boundaries
and you usually want to reduce seeking rather than increase seeking.
This
means you want to use a large off-center stripe size such as 1152 sectors
so sequential I/O does not seek both disks and so meta-data is distributed
across both disks rather than concentrated on a single disk.
If
you really need to get sophisticated, we recommend using a real hardware
RAID controller from the list of
.Fx
supported controllers.
.Sh SYSCTL TUNING
.Xr sysctl 8
variables permit system behavior to be monitored and controlled at
run-time.
Some sysctls simply report on the behavior of the system; others allow
the system behavior to be modified;
some may be set at boot time using
.Xr rc.conf 5 ,
but most will be set via
.Xr sysctl.conf 5 .
There are several hundred sysctls in the system, including many that appear
to be candidates for tuning but actually are not.
In this document we will only cover the ones that have the greatest effect
on the system.
.Pp
The
.Va kern.ipc.shm_use_phys
sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on).
Setting
this parameter to 1 will cause all System V shared memory segments to be
mapped to unpageable physical RAM.
This feature only has an effect if you
are either (A) mapping small amounts of shared memory across many (hundreds)
of processes, or (B) mapping large amounts of shared memory across any
number of processes.
This feature allows the kernel to remove a great deal
of internal memory management page-tracking overhead at the cost of wiring
the shared memory into core, making it unswappable.
.Pp
The
.Va vfs.vmiodirenable
sysctl defaults to 1 (on).
This parameter controls how directories are cached
by the system.
Most directories are small and use but a single fragment
(typically 1K) in the file system and even less (typically 512 bytes) in
the buffer cache.
However, when operating in the default mode the buffer
cache will only cache a fixed number of directories even if you have a huge
amount of memory.
Turning on this sysctl allows the buffer cache to use
the VM Page Cache to cache the directories.
The advantage is that all of
memory is now available for caching directories.
The disadvantage is that
the minimum in-core memory used to cache a directory is the physical page
size (typically 4K) rather than 512 bytes.
We recommend turning this option off in memory-constrained environments;
however, when on, it will substantially improve the performance of services
that manipulate a large number of files.
Such services can include web caches, large mail systems, and news systems.
Turning on this option will generally not reduce performance even with the
wasted memory but you should experiment to find out.
.Pp
The
.Va vfs.write_behind
sysctl defaults to 1 (on).
This tells the file system to issue media
writes as full clusters are collected, which typically occurs when writing
large sequential files.
The idea is to avoid saturating the buffer
cache with dirty buffers when it would not benefit I/O performance.
However,
this may stall processes and under certain circumstances you may wish to turn
it off.
.Pp
The
.Va vfs.hirunningspace
sysctl determines how much outstanding write I/O may be queued to
disk controllers system-wide at any given instance.
The default is
usually sufficient but on machines with lots of disks you may want to bump
it up to four or five megabytes.
Note that setting too high a value
(exceeding the buffer cache's write threshold) can lead to extremely
bad clustering performance.
Do not set this value arbitrarily high!
Also,
higher write queueing values may add latency to reads occurring at the same
time.
.Pp
There are various other buffer-cache and VM page cache related sysctls.
We do not recommend modifying these values.
As of
.Fx 4.3 ,
the VM system does an extremely good job tuning itself.
.Pp
The
.Va net.inet.tcp.sendspace
and
.Va net.inet.tcp.recvspace
sysctls are of particular interest if you are running network intensive
applications.
They control the amount of send and receive buffer space
allowed for any given TCP connection.
The default sending buffer is 32K; the default receiving buffer
is 64K.
You can often
improve bandwidth utilization by increasing the default at the cost of
eating up more kernel memory for each connection.
We do not recommend
increasing the defaults if you are serving hundreds or thousands of
simultaneous connections because it is possible to quickly run the system
out of memory due to stalled connections building up.
But if you need
high bandwidth over a fewer number of connections, especially if you have
gigabit Ethernet, increasing these defaults can make a huge difference.
You can adjust the buffer size for incoming and outgoing data separately.
For example, if your machine is primarily doing web serving you may want
to decrease the recvspace in order to be able to increase the
sendspace without eating too much kernel memory.
Note that the routing table (see
.Xr route 8 )
can be used to introduce route-specific send and receive buffer size
defaults.
.Pp
As an additional management tool you can use pipes in your
firewall rules (see
.Xr ipfw 8 )
to limit the bandwidth going to or from particular IP blocks or ports.
For example, if you have a T1 you might want to limit your web traffic
to 70% of the T1's bandwidth in order to leave the remainder available
for mail and interactive use.
Normally a heavily loaded web server
will not introduce significant latencies into other services even if
the network link is maxed out, but enforcing a limit can smooth things
out and lead to longer term stability.
Many people also enforce artificial
bandwidth limitations in order to ensure that they are not charged for
using too much bandwidth.
.Pp
Setting the send or receive TCP buffer to values larger than 65535 will result
in a marginal performance improvement unless both hosts support the window
scaling extension of the TCP protocol, which is controlled by the
.Va net.inet.tcp.rfc1323
sysctl.
These extensions should be enabled and the TCP buffer size should be set
to a value larger than 65536 in order to obtain good performance from
certain types of network links; specifically, gigabit WAN links and
high-latency satellite links.
RFC1323 support is enabled by default.
.Pp
The
.Va net.inet.tcp.always_keepalive
sysctl determines whether or not the TCP implementation should attempt
to detect dead TCP connections by intermittently delivering
.Dq keepalives
on the connection.
By default, this is enabled for all applications; by setting this
sysctl to 0, only applications that specifically request keepalives
will use them.
In most environments, TCP keepalives will improve the management of
system state by expiring dead TCP connections, particularly for
systems serving dialup users who may not always terminate individual
TCP connections before disconnecting from the network.
However, in some environments, temporary network outages may be
incorrectly identified as dead sessions, resulting in unexpectedly
terminated TCP connections.
In such environments, setting the sysctl to 0 may reduce the occurrence of
TCP session disconnections.
.Pp
The
.Va net.inet.tcp.delayed_ack
TCP feature is largely misunderstood.
Historically speaking, this feature
was designed to allow the acknowledgement to transmitted data to be returned
along with the response.
For example, when you type over a remote shell,
the acknowledgement to the character you send can be returned along with the
data representing the echo of the character.
With delayed acks turned off,
the acknowledgement may be sent in its own packet, before the remote service
has a chance to echo the data it just received.
This same concept also
applies to any interactive protocol (e.g.\& SMTP, WWW, POP3), and can cut the
number of tiny packets flowing across the network in half.
The
.Fx
delayed ACK implementation also follows the TCP protocol rule that
at least every other packet be acknowledged even if the standard 100ms
timeout has not yet passed.
Normally the worst a delayed ACK can do is
slightly delay the teardown of a connection, or slightly delay the ramp-up
of a slow-start TCP connection.
While we are not sure we believe that
the several FAQs related to packages such as SAMBA and SQUID which advise
turning off delayed acks may be referring to the slow-start issue.
In
.Fx ,
it would be more beneficial to increase the slow-start flightsize via
the
.Va net.inet.tcp.slowstart_flightsize
sysctl rather than disable delayed acks.
.Pp
The
.Va net.inet.tcp.inflight_enable
sysctl turns on bandwidth delay product limiting for all TCP connections.
The system will attempt to calculate the bandwidth delay product for each
connection and limit the amount of data queued to the network to just the
amount required to maintain optimum throughput.
This feature is useful
if you are serving data over modems, GigE, or high speed WAN links (or
any other link with a high bandwidth*delay product), especially if you are
also using window scaling or have configured a large send window.
If you enable this option, you should also be sure to set
.Va net.inet.tcp.inflight_debug
to 0 (disable debugging), and for production use setting
.Va net.inet.tcp.inflight_min
to at least 6144 may be beneficial.
Note however, that setting high
minimums may effectively disable bandwidth limiting depending on the link.
The limiting feature reduces the amount of data built up in intermediate
router and switch packet queues as well as reduces the amount of data built
up in the local host's interface queue.
With fewer packets queued up,
interactive connections, especially over slow modems, will also be able
to operate with lower round trip times.
However, note that this feature
only affects data transmission (uploading / server-side).
It does not
affect data reception (downloading).
.Pp
Adjusting
.Va net.inet.tcp.inflight_stab
is not recommended.
This parameter defaults to 20, representing 2 maximal packets added
to the bandwidth delay product window calculation.
The additional
window is required to stabilize the algorithm and improve responsiveness
to changing conditions, but it can also result in higher ping times
over slow links (though still much lower than you would get without
the inflight algorithm).
In such cases you may
wish to try reducing this parameter to 15, 10, or 5, and you may also
have to reduce
.Va net.inet.tcp.inflight_min
(for example, to 3500) to get the desired effect.
Reducing these parameters
should be done as a last resort only.
.Pp
The
.Va net.inet.ip.portrange.*
sysctls control the port number ranges automatically bound to TCP and UDP
sockets.
There are three ranges: a low range, a default range, and a
high range, selectable via the
.Dv IP_PORTRANGE
.Xr setsockopt 2
call.
Most
network programs use the default range which is controlled by
.Va net.inet.ip.portrange.first
and
.Va net.inet.ip.portrange.last ,
which default to 1024 and 5000, respectively.<commit_msg>The inflight sysctls have moved to net.inet.tcp.inflight.*, so adjust documentation accordingly.<commit_after>.\" Copyright (c) 2001, <NAME>.  Terms and conditions are those of
.\" the BSD Copyright as specified in the file "/usr/src/COPYRIGHT" in
.\" the source tree.
.\"
.\" $FreeBSD$
.\"
.Dd June 25, 2002
.Dt TUNING 7
.Os
.Sh NAME
.Nm tuning
.Nd performance tuning under FreeBSD
.Sh SYSTEM SETUP - DISKLABEL, NEWFS, TUNEFS, SWAP
When using
.Xr bsdlabel 8
or
.Xr sysinstall 8
to lay out your file systems on a hard disk it is important to remember
that hard drives can transfer data much more quickly from outer tracks
than they can from inner tracks.
To take advantage of this you should
try to pack your smaller file systems and swap closer to the outer tracks,
follow with the larger file systems, and end with the largest file systems.
It is also important to size system standard file systems such that you
will not be forced to resize them later as you scale the machine up.
I usually create, in order, a 128M root, 1G swap, 128M
.Pa /var ,
128M
.Pa /var/tmp ,
3G
.Pa /usr ,
and use any remaining space for
.Pa /home .
.Pp
You should typically size your swap space to approximately 2x main memory.
If you do not have a lot of RAM, though, you will generally want a lot
more swap.
It is not recommended that you configure any less than
256M of swap on a system and you should keep in mind future memory
expansion when sizing the swap partition.
The kernel's VM paging algorithms are tuned to perform best when there is
at least 2x swap versus main memory.
Configuring too little swap can lead
to inefficiencies in the VM page scanning code as well as create issues
later on if you add more memory to your machine.
Finally, on larger systems
with multiple SCSI disks (or multiple IDE disks operating on different
controllers), we strongly recommend that you configure swap on each drive.
The swap partitions on the drives should be approximately the same size.
The kernel can handle arbitrary sizes but
internal data structures scale to 4 times the largest swap partition.
Keeping
the swap partitions near the same size will allow the kernel to optimally
stripe swap space across the N disks.
Do not worry about overdoing it a
little, swap space is the saving grace of
.Ux
and even if you do not normally use much swap, it can give you more time to
recover from a runaway program before being forced to reboot.
.Pp
How you size your
.Pa /var
partition depends heavily on what you intend to use the machine for.
This
partition is primarily used to hold mailboxes, the print spool, and log
files.
Some people even make
.Pa /var/log
its own partition (but except for extreme cases it is not worth the waste
of a partition ID).
If your machine is intended to act as a mail
or print server,
or you are running a heavily visited web server, you should consider
creating a much larger partition \(en perhaps a gig or more.
It is very easy
to underestimate log file storage requirements.
.Pp
Sizing
.Pa /var/tmp
depends on the kind of temporary file usage you think you will need.
128M is
the minimum we recommend.
Also note that sysinstall will create a
.Pa /tmp
directory.
Dedicating a partition for temporary file storage is important for
two reasons: first, it reduces the possibility of file system corruption
in a crash, and second it reduces the chance of a runaway process that
fills up
.Oo Pa /var Oc Ns Pa /tmp
from blowing up more critical subsystems (mail,
logging, etc).
Filling up
.Oo Pa /var Oc Ns Pa /tmp
is a very common problem to have.
.Pp
In the old days there were differences between
.Pa /tmp
and
.Pa /var/tmp ,
but the introduction of
.Pa /var
(and
.Pa /var/tmp )
led to massive confusion
by program writers so today programs haphazardly use one or the
other and thus no real distinction can be made between the two.
So it makes sense to have just one temporary directory and
softlink to it from the other
.Pa tmp
directory locations.
However you handle
.Pa /tmp ,
the one thing you do not want to do is leave it sitting
on the root partition where it might cause root to fill up or possibly
corrupt root in a crash/reboot situation.
.Pp
The
.Pa /usr
partition holds the bulk of the files required to support the system and
a subdirectory within it called
.Pa /usr/local
holds the bulk of the files installed from the
.Xr ports 7
hierarchy.
If you do not use ports all that much and do not intend to keep
system source
.Pq Pa /usr/src
on the machine, you can get away with
a 1 gigabyte
.Pa /usr
partition.
However, if you install a lot of ports
(especially window managers and Linux-emulated binaries), we recommend
at least a 2 gigabyte
.Pa /usr
and if you also intend to keep system source
on the machine, we recommend a 3 gigabyte
.Pa /usr .
Do not underestimate the
amount of space you will need in this partition, it can creep up and
surprise you!
.Pp
The
.Pa /home
partition is typically used to hold user-specific data.
I usually size it to the remainder of the disk.
.Pp
Why partition at all?
Why not create one big
.Pa /
partition and be done with it?
Then I do not have to worry about undersizing things!
Well, there are several reasons this is not a good idea.
First,
each partition has different operational characteristics and separating them
allows the file system to tune itself to those characteristics.
For example,
the root and
.Pa /usr
partitions are read-mostly, with very little writing, while
a lot of reading and writing could occur in
.Pa /var
and
.Pa /var/tmp .
By properly
partitioning your system fragmentation introduced in the smaller more
heavily write-loaded partitions will not bleed over into the mostly-read
partitions.
Additionally, keeping the write-loaded partitions closer to
the edge of the disk (i.e., before the really big partitions instead of after
in the partition table) will increase I/O performance in the partitions
where you need it the most.
Now it is true that you might also need I/O
performance in the larger partitions, but they are so large that shifting
them more towards the edge of the disk will not lead to a significant
performance improvement whereas moving
.Pa /var
to the edge can have a huge impact.
Finally, there are safety concerns.
Having a small neat root partition that
is essentially read-only gives it a greater chance of surviving a bad crash
intact.
.Pp
Properly partitioning your system also allows you to tune
.Xr newfs 8 ,
and
.Xr tunefs 8
parameters.
Tuning
.Xr newfs 8
requires more experience but can lead to significant improvements in
performance.
There are three parameters that are relatively safe to tune:
.Em blocksize , bytes/i-node ,
and
.Em cylinders/group .
.Pp
.Fx
performs best when using 8K or 16K file system block sizes.
The default file system block size is 16K,
which provides best performance for most applications,
with the exception of those that perform random access on large files
(such as database server software).
Such applications tend to perform better with a smaller block size,
although modern disk characteristics are such that the performance
gain from using a smaller block size may not be worth consideration.
Using a block size larger than 16K
can cause fragmentation of the buffer cache and
lead to lower performance.
.Pp
The defaults may be unsuitable
for a file system that requires a very large number of i-nodes
or is intended to hold a large number of very small files.
Such a file system should be created with an 8K or 4K block size.
This also requires you to specify a smaller
fragment size.
We recommend always using a fragment size that is 1/8
the block size (less testing has been done on other fragment size factors).
The
.Xr newfs 8
options for this would be
.Dq Li "newfs -f 1024 -b 8192 ..." .
.Pp
If a large partition is intended to be used to hold fewer, larger files, such
as database files, you can increase the
.Em bytes/i-node
ratio which reduces the number of i-nodes (maximum number of files and
directories that can be created) for that partition.
Decreasing the number
of i-nodes in a file system can greatly reduce
.Xr fsck 8
recovery times after a crash.
Do not use this option
unless you are actually storing large files on the partition, because if you
overcompensate you can wind up with a file system that has lots of free
space remaining but cannot accommodate any more files.
Using 32768, 65536, or 262144 bytes/i-node is recommended.
You can go higher but
it will have only incremental effects on
.Xr fsck 8
recovery times.
For example,
.Dq Li "newfs -i 32768 ..." .
.Pp
.Xr tunefs 8
may be used to further tune a file system.
This command can be run in
single-user mode without having to reformat the file system.
However, this is possibly the most abused program in the system.
Many people attempt to
increase available file system space by setting the min-free percentage to 0.
This can lead to severe file system fragmentation and we do not recommend
that you do this.
Really the only
.Xr tunefs 8
option worthwhile here is turning on
.Em softupdates
with
.Dq Li "tunefs -n enable /filesystem" .
(Note: in
.Fx 4.5
and later, softupdates can be turned on using the
.Fl U
option to
.Xr newfs 8 ,
and
.Xr sysinstall 8
will typically enable softupdates automatically for non-root file systems).
Softupdates drastically improves meta-data performance, mainly file
creation and deletion.
We recommend enabling softupdates on most file systems; however, there
are two limitations to softupdates that you should be aware of when
determining whether to use it on a file system.
First, softupdates guarantees file system consistency in the
case of a crash but could very easily be several seconds (even a minute!\&)
behind on pending write to the physical disk.
If you crash you may lose more work
than otherwise.
Secondly, softupdates delays the freeing of file system
blocks.
If you have a file system (such as the root file system) which is
close to full, doing a major update of it, e.g.\&
.Dq Li "make installworld" ,
can run it out of space and cause the update to fail.
For this reason, softupdates will not be enabled on the root file system
during a typical install.
There is no loss of performance since the root
file system is rarely written to.
.Pp
A number of run-time
.Xr mount 8
options exist that can help you tune the system.
The most obvious and most dangerous one is
.Cm async .
Do not ever use it; it is far too dangerous.
A less dangerous and more
useful
.Xr mount 8
option is called
.Cm noatime .
.Ux
file systems normally update the last-accessed time of a file or
directory whenever it is accessed.
This operation is handled in
.Fx
with a delayed write and normally does not create a burden on the system.
However, if your system is accessing a huge number of files on a continuing
basis the buffer cache can wind up getting polluted with atime updates,
creating a burden on the system.
For example, if you are running a heavily
loaded web site, or a news server with lots of readers, you might want to
consider turning off atime updates on your larger partitions with this
.Xr mount 8
option.
However, you should not gratuitously turn off atime
updates everywhere.
For example, the
.Pa /var
file system customarily
holds mailboxes, and atime (in combination with mtime) is used to
determine whether a mailbox has new mail.
You might as well leave
atime turned on for mostly read-only partitions such as
.Pa /
and
.Pa /usr
as well.
This is especially useful for
.Pa /
since some system utilities
use the atime field for reporting.
.Sh STRIPING DISKS
In larger systems you can stripe partitions from several drives together
to create a much larger overall partition.
Striping can also improve
the performance of a file system by splitting I/O operations across two
or more disks.
The
.Xr vinum 8
and
.Xr ccdconfig 8
utilities may be used to create simple striped file systems.
Generally
speaking, striping smaller partitions such as the root and
.Pa /var/tmp ,
or essentially read-only partitions such as
.Pa /usr
is a complete waste of time.
You should only stripe partitions that require serious I/O performance,
typically
.Pa /var , /home ,
or custom partitions used to hold databases and web pages.
Choosing the proper stripe size is also
important.
File systems tend to store meta-data on power-of-2 boundaries
and you usually want to reduce seeking rather than increase seeking.
This
means you want to use a large off-center stripe size such as 1152 sectors
so sequential I/O does not seek both disks and so meta-data is distributed
across both disks rather than concentrated on a single disk.
If
you really need to get sophisticated, we recommend using a real hardware
RAID controller from the list of
.Fx
supported controllers.
.Sh SYSCTL TUNING
.Xr sysctl 8
variables permit system behavior to be monitored and controlled at
run-time.
Some sysctls simply report on the behavior of the system; others allow
the system behavior to be modified;
some may be set at boot time using
.Xr rc.conf 5 ,
but most will be set via
.Xr sysctl.conf 5 .
There are several hundred sysctls in the system, including many that appear
to be candidates for tuning but actually are not.
In this document we will only cover the ones that have the greatest effect
on the system.
.Pp
The
.Va kern.ipc.shm_use_phys
sysctl defaults to 0 (off) and may be set to 0 (off) or 1 (on).
Setting
this parameter to 1 will cause all System V shared memory segments to be
mapped to unpageable physical RAM.
This feature only has an effect if you
are either (A) mapping small amounts of shared memory across many (hundreds)
of processes, or (B) mapping large amounts of shared memory across any
number of processes.
This feature allows the kernel to remove a great deal
of internal memory management page-tracking overhead at the cost of wiring
the shared memory into core, making it unswappable.
.Pp
The
.Va vfs.vmiodirenable
sysctl defaults to 1 (on).
This parameter controls how directories are cached
by the system.
Most directories are small and use but a single fragment
(typically 1K) in the file system and even less (typically 512 bytes) in
the buffer cache.
However, when operating in the default mode the buffer
cache will only cache a fixed number of directories even if you have a huge
amount of memory.
Turning on this sysctl allows the buffer cache to use
the VM Page Cache to cache the directories.
The advantage is that all of
memory is now available for caching directories.
The disadvantage is that
the minimum in-core memory used to cache a directory is the physical page
size (typically 4K) rather than 512 bytes.
We recommend turning this option off in memory-constrained environments;
however, when on, it will substantially improve the performance of services
that manipulate a large number of files.
Such services can include web caches, large mail systems, and news systems.
Turning on this option will generally not reduce performance even with the
wasted memory but you should experiment to find out.
.Pp
The
.Va vfs.write_behind
sysctl defaults to 1 (on).
This tells the file system to issue media
writes as full clusters are collected, which typically occurs when writing
large sequential files.
The idea is to avoid saturating the buffer
cache with dirty buffers when it would not benefit I/O performance.
However,
this may stall processes and under certain circumstances you may wish to turn
it off.
.Pp
The
.Va vfs.hirunningspace
sysctl determines how much outstanding write I/O may be queued to
disk controllers system-wide at any given instance.
The default is
usually sufficient but on machines with lots of disks you may want to bump
it up to four or five megabytes.
Note that setting too high a value
(exceeding the buffer cache's write threshold) can lead to extremely
bad clustering performance.
Do not set this value arbitrarily high!
Also,
higher write queueing values may add latency to reads occurring at the same
time.
.Pp
There are various other buffer-cache and VM page cache related sysctls.
We do not recommend modifying these values.
As of
.Fx 4.3 ,
the VM system does an extremely good job tuning itself.
.Pp
The
.Va net.inet.tcp.sendspace
and
.Va net.inet.tcp.recvspace
sysctls are of particular interest if you are running network intensive
applications.
They control the amount of send and receive buffer space
allowed for any given TCP connection.
The default sending buffer is 32K; the default receiving buffer
is 64K.
You can often
improve bandwidth utilization by increasing the default at the cost of
eating up more kernel memory for each connection.
We do not recommend
increasing the defaults if you are serving hundreds or thousands of
simultaneous connections because it is possible to quickly run the system
out of memory due to stalled connections building up.
But if you need
high bandwidth over a fewer number of connections, especially if you have
gigabit Ethernet, increasing these defaults can make a huge difference.
You can adjust the buffer size for incoming and outgoing data separately.
For example, if your machine is primarily doing web serving you may want
to decrease the recvspace in order to be able to increase the
sendspace without eating too much kernel memory.
Note that the routing table (see
.Xr route 8 )
can be used to introduce route-specific send and receive buffer size
defaults.
.Pp
As an additional management tool you can use pipes in your
firewall rules (see
.Xr ipfw 8 )
to limit the bandwidth going to or from particular IP blocks or ports.
For example, if you have a T1 you might want to limit your web traffic
to 70% of the T1's bandwidth in order to leave the remainder available
for mail and interactive use.
Normally a heavily loaded web server
will not introduce significant latencies into other services even if
the network link is maxed out, but enforcing a limit can smooth things
out and lead to longer term stability.
Many people also enforce artificial
bandwidth limitations in order to ensure that they are not charged for
using too much bandwidth.
.Pp
Setting the send or receive TCP buffer to values larger than 65535 will result
in a marginal performance improvement unless both hosts support the window
scaling extension of the TCP protocol, which is controlled by the
.Va net.inet.tcp.rfc1323
sysctl.
These extensions should be enabled and the TCP buffer size should be set
to a value larger than 65536 in order to obtain good performance from
certain types of network links; specifically, gigabit WAN links and
high-latency satellite links.
RFC1323 support is enabled by default.
.Pp
The
.Va net.inet.tcp.always_keepalive
sysctl determines whether or not the TCP implementation should attempt
to detect dead TCP connections by intermittently delivering
.Dq keepalives
on the connection.
By default, this is enabled for all applications; by setting this
sysctl to 0, only applications that specifically request keepalives
will use them.
In most environments, TCP keepalives will improve the management of
system state by expiring dead TCP connections, particularly for
systems serving dialup users who may not always terminate individual
TCP connections before disconnecting from the network.
However, in some environments, temporary network outages may be
incorrectly identified as dead sessions, resulting in unexpectedly
terminated TCP connections.
In such environments, setting the sysctl to 0 may reduce the occurrence of
TCP session disconnections.
.Pp
The
.Va net.inet.tcp.delayed_ack
TCP feature is largely misunderstood.
Historically speaking, this feature
was designed to allow the acknowledgement to transmitted data to be returned
along with the response.
For example, when you type over a remote shell,
the acknowledgement to the character you send can be returned along with the
data representing the echo of the character.
With delayed acks turned off,
the acknowledgement may be sent in its own packet, before the remote service
has a chance to echo the data it just received.
This same concept also
applies to any interactive protocol (e.g.\& SMTP, WWW, POP3), and can cut the
number of tiny packets flowing across the network in half.
The
.Fx
delayed ACK implementation also follows the TCP protocol rule that
at least every other packet be acknowledged even if the standard 100ms
timeout has not yet passed.
Normally the worst a delayed ACK can do is
slightly delay the teardown of a connection, or slightly delay the ramp-up
of a slow-start TCP connection.
While we are not sure we believe that
the several FAQs related to packages such as SAMBA and SQUID which advise
turning off delayed acks may be referring to the slow-start issue.
In
.Fx ,
it would be more beneficial to increase the slow-start flightsize via
the
.Va net.inet.tcp.slowstart_flightsize
sysctl rather than disable delayed acks.
.Pp
The
.Va net.inet.tcp.inflight.enable
sysctl turns on bandwidth delay product limiting for all TCP connections.
The system will attempt to calculate the bandwidth delay product for each
connection and limit the amount of data queued to the network to just the
amount required to maintain optimum throughput.
This feature is useful
if you are serving data over modems, GigE, or high speed WAN links (or
any other link with a high bandwidth*delay product), especially if you are
also using window scaling or have configured a large send window.
If you enable this option, you should also be sure to set
.Va net.inet.tcp.inflight.debug
to 0 (disable debugging), and for production use setting
.Va net.inet.tcp.inflight.min
to at least 6144 may be beneficial.
Note however, that setting high
minimums may effectively disable bandwidth limiting depending on the link.
The limiting feature reduces the amount of data built up in intermediate
router and switch packet queues as well as reduces the amount of data built
up in the local host's interface queue.
With fewer packets queued up,
interactive connections, especially over slow modems, will also be able
to operate with lower round trip times.
However, note that this feature
only affects data transmission (uploading / server-side).
It does not
affect data reception (downloading).
.Pp
Adjusting
.Va net.inet.tcp.inflight.stab
is not recommended.
This parameter defaults to 20, representing 2 maximal packets added
to the bandwidth delay product window calculation.
The additional
window is required to stabilize the algorithm and improve responsiveness
to changing conditions, but it can also result in higher ping times
over slow links (though still much lower than you would get without
the inflight algorithm).
In such cases you may
wish to try reducing this parameter to 15, 10, or 5, and you may also
have to reduce
.Va net.inet.tcp.inflight.min
(for example, to 3500) to get the desired effect.
Reducing these parameters
should be done as a last resort only.
.Pp
The
.Va net.inet.ip.portrange.*
sysctls control the port number ranges automatically bound to TCP and UDP
sockets.
There are three ranges: a low range, a default range, and a
high range, selectable via the
.Dv IP_PORTRANGE
.Xr setsockopt 2
call.
Most
network programs use the default range which is controlled by
.Va net.inet.ip.portrange.first
and
.Va net.inet.ip.portrange.last ,
which default to 1024 and 5000, respectively.
{'fraction_non_alphanumeric': 0.03297200612029395, 'fraction_numerical': 0.007715018425532832, 'mean_word_length': 4.234517766497462, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '736346', 'n_tokens_mistral': 12434, 'n_tokens_neox': 11602, 'n_words': 7779}