<commit_before>Use of index and Racy git problem
=================================

Background
----------

The index is one of the most important data structures in git.
It represents a virtual working tree state by recording list of
paths and their object names and serves as a staging area to
write out the next tree object to be committed.  The state is
"virtual" in the sense that it does not necessarily have to, and
often does not, match the files in the working tree.

There are cases git needs to examine the differences between the
virtual working tree state in the index and the files in the
working tree.  The most obvious case is when the user asks `git
diff` (or its low level implementation, `git diff-files`) or
`git-ls-files --modified`.  In addition, git internally checks
if the files in the working tree are different from what are
recorded in the index to avoid stomping on local changes in them
during patch application, switching branches, and merging.

In order to speed up this comparison between the files in the
working tree and the index entries, the index entries record the
information obtained from the filesystem via `lstat(2)` system
call when they were last updated.  When checking if they differ,
git first runs `lstat(2)` on the files and compares the result
with this information (this is what was originally done by the
`ce_match_stat()` function, but the current code does it in
`ce_match_stat_basic()` function).  If some of these "cached
stat information" fields do not match, git can tell that the
files are modified without even looking at their contents.

Note: not all members in `struct stat` obtained via `lstat(2)`
are used for this comparison.  For example, `st_atime` obviously
is not useful.  Currently, git compares the file type (regular
files vs symbolic links) and executable bits (only for regular
files) from `st_mode` member, `st_mtime` and `st_ctime`
timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members.
With a `USE_STDEV` compile-time option, `st_dev` is also
compared, but this is not enabled by default because this member
is not stable on network filesystems.  With `USE_NSEC`
compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec`
members are also compared, but this is not enabled by default
because the value of this member becomes meaningless once the
inode is evicted from the inode cache on filesystems that do not
store it on disk.


Racy git
--------

There is one slight problem with the optimization based on the
cached stat information.  Consider this sequence:

  : modify 'foo'
  $ git update-index 'foo'
  : modify 'foo' again, in-place, without changing its size

The first `update-index` computes the object name of the
contents of file `foo` and updates the index entry for `foo`
along with the `struct stat` information.  If the modification
that follows it happens very fast so that the file's `st_mtime`
timestamp does not change, after this sequence, the cached stat
information the index entry records still exactly match what you
would see in the filesystem, even though the file `foo` is now
different.
This way, git can incorrectly think files in the working tree
are unmodified even though they actually are.  This is called
the "racy git" problem (discovered by Pasky), and the entries
that appear clean when they may not be because of this problem<commit_msg>racy-git.txt: explain nsec problem in more detail<commit_after>Use of index and Racy git problem
=================================

Background
----------

The index is one of the most important data structures in git.
It represents a virtual working tree state by recording list of
paths and their object names and serves as a staging area to
write out the next tree object to be committed.  The state is
"virtual" in the sense that it does not necessarily have to, and
often does not, match the files in the working tree.

There are cases git needs to examine the differences between the
virtual working tree state in the index and the files in the
working tree.  The most obvious case is when the user asks `git
diff` (or its low level implementation, `git diff-files`) or
`git-ls-files --modified`.  In addition, git internally checks
if the files in the working tree are different from what are
recorded in the index to avoid stomping on local changes in them
during patch application, switching branches, and merging.

In order to speed up this comparison between the files in the
working tree and the index entries, the index entries record the
information obtained from the filesystem via `lstat(2)` system
call when they were last updated.  When checking if they differ,
git first runs `lstat(2)` on the files and compares the result
with this information (this is what was originally done by the
`ce_match_stat()` function, but the current code does it in
`ce_match_stat_basic()` function).  If some of these "cached
stat information" fields do not match, git can tell that the
files are modified without even looking at their contents.

Note: not all members in `struct stat` obtained via `lstat(2)`
are used for this comparison.  For example, `st_atime` obviously
is not useful.  Currently, git compares the file type (regular
files vs symbolic links) and executable bits (only for regular
files) from `st_mode` member, `st_mtime` and `st_ctime`
timestamps, `st_uid`, `st_gid`, `st_ino`, and `st_size` members.
With a `USE_STDEV` compile-time option, `st_dev` is also
compared, but this is not enabled by default because this member
is not stable on network filesystems.  With `USE_NSEC`
compile-time option, `st_mtim.tv_nsec` and `st_ctim.tv_nsec`
members are also compared, but this is not enabled by default
because in-core timestamps can have finer granularity than
on-disk timestamps, resulting in meaningless changes when an
inode is evicted from the inode cache.  See commit 8ce13b0
of git://git.kernel.org/pub/scm/linux/kernel/git/tglx/history.git
([PATCH] Sync in core time granuality with filesystems,
2005-01-04).

Racy git
--------

There is one slight problem with the optimization based on the
cached stat information.  Consider this sequence:

  : modify 'foo'
  $ git update-index 'foo'
  : modify 'foo' again, in-place, without changing its size

The first `update-index` computes the object name of the
contents of file `foo` and updates the index entry for `foo`
along with the `struct stat` information.  If the modification
that follows it happens very fast so that the file's `st_mtime`
timestamp does not change, after this sequence, the cached stat
information the index entry records still exactly match what you
would see in the filesystem, even though the file `foo` is now
different.
This way, git can incorrectly think files in the working tree
are unmodified even though they actually are.  This is called
the "racy git" problem (discovered by Pasky), and the entries
that appear clean when they may not be because of this problem
{'fraction_non_alphanumeric': 0.06327471893917555, 'fraction_numerical': 0.0025944076102623233, 'mean_word_length': 4.5290836653386455, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6522479', 'n_tokens_mistral': 1832, 'n_tokens_neox': 1766, 'n_words': 1074}