<commit_before># Incremental Build Improvements

This spec defines some improvements to improve incremental build and task up-to-date checks

# General Speed-ups

These are general speed ups that improve all builds.

## Story: Speed up File metadata lookup in task input/output snapshotting

File metadata operations .isFile(), .isDirectory(), .length() and .lastModified are
hotspots in task input/output snapshotting.

The Java nio2 directory walking method java.nio.file.Files.walkFileTree can pass the file
metadata used for directory scanning to "visiting" the file tree so that metadata
(BasicFileAttributes) doesn't have to be re-read.

### Implementation

- For JDK7+ with UTF-8 file encoding, use a nio2 file walker implemention. ✔︎
    - Cache isDirectory()/getSize()/getLastModified() in FileVisitDetails from BasicFileAttributes gathered from walking ✔︎
- Otherwise, use default file walker implementation (current behavior). ✔︎
    - Use a caching FileVisitDetails for getSize()/getLastModified() to cache on first use. ✔︎
    - Maybe reuse isFile/isDirectory result from the walker implementation ✔︎
- Replace calls to getFiles() in DefaultFileCollectionSnapshotter with a visitor ✔︎

### Test coverage

- Test that correct implementation is chosen for JDK platform and file encoding ✔︎
- Test that a file walker sees a snapshot of tree even if the tree is modified after walking has started. ✔︎
- Generate file tree and walk with JDK7+ file walker and non-nio2 file walker. Attributes and files should be the same for both. ✔︎
- Performance gains will be measured from existing performance tests. ✔︎
- Expect existing test coverage will cover behavior of input/output snapshotting and file collection operations. ✔︎

## Story: Changes to reduce byte/char array allocations and array copying

- Just do it

## Story: Changes for adjusting collection sizes when final size is known

- Just do it

## Story: Reduce the in-memory size of the task history cache by interning file paths

### Implementation

- Use Guava's [`Interners.newWeakInterner()`](http://google.github.io/guava/releases/18.0/api/docs/com/google/common/collect/Interners.html#newWeakInterner%28%29) to create a cache `StringInterner` for sharing the file path Strings. Place this cache in `GlobalScopeServices` so that the instance lives across multiple builds in the daemon.
- use the `StringInterner` to intern all duplicate path names contained in `fileSnapshots`, `taskArtifacts`, `outputFileStates` and `fileHashes` caches.
- Implementation can be based on the solution developed in the spike. The commit is https://github.com/gradle/gradle/commit/d26d4ce1098e0eee9896279cbeabefb1ca3e871c .

### Test coverage

- Add basic unit test coverage for StringInterner
  - interning different string instances with similar content return the first instance that was interned
  - allows calling method with null, returns null in that case

## Story: Add caching to Specs returned from PatternSet.getAsSpecs()

Evaluating patterns is a hotspot in directory scanning. The default excludes patterns
contains 28 entries. Checking all rules for each file sums up in a lot of operations.
Adding caching will improve performance of subsequent incremental builds.

### Implementation

Assumption: PatternSet class is part of the Gradle Public API and we cannot change it's interface.

#### 1. phase - target release Gradle 2.9

Spike commit: https://github.com/lhotari/gradle/commit/f235117fd0b8b125a8220c45dca8ee9dc2331559

- Mainly based on the spike commit
- Move Spec<FileTreeElement> creation logic to separate factory class from PatternSet class (currently in getAsSpec, getAsIncludeSpec, getAsExcludeSpec methods)
- Add caching for Spec<FileTreeElement> instance creation and evaluation results
- Only add caching to Spec<FileTreeElement> instances that are created from the include and exclude patterns.
- A PatternSet can contain a list of includeSpecs and excludeSpecs. Don't add caching to these.

#### Test coverage for 1. phase

- Test that Spec<FileTreeElement> includes (added with PatternSet.include(Spec<FileTreeElement> spec)) and excludes (added with PatternSet.exclude(Spec<FileTreeElement> spec)) are not cached.
- Existing PatternSet tests cover rest of the changes since there are no planned behavioural or API changes for 1. phase.

#### 2. phase - target release Gradle 2.10

Goal: separate caching logic and manage the cache instance in Gradle infrastructure instead of a singleton instance
- Separate caching logic from PatternSetSpecFactory class to a class called CachingPatternSetSpecFactory.
- Remove caching from the default PatternSetSpecFactory class. This removes the caching from PatternSet class which was added in 1. phase.
- Make the Gradle infrastructure manage the CachingPatternSetSpecFactory instance.
- Create a new PatternSet subclass that takes the PatternSetSpecFactory instance in the constructor.
- Replace usage of PatternSet class with the new subclass in Gradle core code. Wire the CachingPatternSetSpecFactory instance to the instances of the new PatternSet subclass.

Goal: support suppressing default excludes
- Add support for suppressing the default excludes that are part of Ant integration legacy in Gradle. Default excludes aren't needed when there are specific include patterns. The default excludes support can be added to the same new PatternSet subclass that supports the separate (Caching)PatternSetSpecFactory.
- make SourceSets in Gradle use the new PatternSet implementation that supports suppressing the default excludes and uses the CachingPatternSetSpecFactory managed by the Gradle infrastructure (`GlobalScopeServices`)


## Story: High number of UnknownDomainObjectExceptions when resolving libraries in native projects.

TBD

## Story: Add "discovered" inputs to incremental tasks

This story adds a way for an incremental task to register additional inputs once execution has started.  At the end of execution, the discovered inputs are recorded in the task's execution history.

    void perform(IncrementalTaskInputs inputs) {
      getInputs().getFiles.each { source ->
        File discoveredInput = complicatedAnalysis(source)
        inputs.newInput(discoveredInput)
      }
      if (inputs.incremental) {
        inputs.outOfDate {
          // do stuff
        }
      }
    }

### Implementation

- Add `void newInput(File)` to IncrementalTaskInputs
- Add `Set<File> getDiscoveredInputs()` to IncrementalTaskInputsInternal
- StatefulIncrementalTaskInputs will implement `newInput` and `getDiscoveredInputs` to capture discovered inputs.
- Add new getDiscoveredInputFilesSnapshot/setDiscoveredInputFilesSnapshot methods to TaskExecution and LazyTaskExecution to record a separate discovered FileCollectionSnapshot. This means a task has 3 snapshots (inputs, outputs, discovered inputs).
- Update LazyTaskExecution and CacheBackedTaskHistoryRepository to handle the new discovered inputs snapshot. Maybe there's a higher level extraction that could be done here? The code for inputs/outputs/discovered are all similar.
- Add new DiscoveredInputFilesStateChangeRule that behaves identical to InputFilesStateChangeRule, except there is no input snapshot. Instead, the input snapshot is calculated by getting the hash of all files in the previousExecution's DiscoveredInputFilesSnapshot.
- DiscoveredInputFilesStateChangeRule's snapshotAfterTask action is to take the snapshot of all discovered files and add it to the current execution.<commit_msg>update spec for PatternSet Spec caching<commit_after># Incremental Build Improvements

This spec defines some improvements to improve incremental build and task up-to-date checks

# General Speed-ups

These are general speed ups that improve all builds.

## Story: Speed up File metadata lookup in task input/output snapshotting

File metadata operations .isFile(), .isDirectory(), .length() and .lastModified are
hotspots in task input/output snapshotting.

The Java nio2 directory walking method java.nio.file.Files.walkFileTree can pass the file
metadata used for directory scanning to "visiting" the file tree so that metadata
(BasicFileAttributes) doesn't have to be re-read.

### Implementation

- For JDK7+ with UTF-8 file encoding, use a nio2 file walker implemention. ✔︎
    - Cache isDirectory()/getSize()/getLastModified() in FileVisitDetails from BasicFileAttributes gathered from walking ✔︎
- Otherwise, use default file walker implementation (current behavior). ✔︎
    - Use a caching FileVisitDetails for getSize()/getLastModified() to cache on first use. ✔︎
    - Maybe reuse isFile/isDirectory result from the walker implementation ✔︎
- Replace calls to getFiles() in DefaultFileCollectionSnapshotter with a visitor ✔︎

### Test coverage

- Test that correct implementation is chosen for JDK platform and file encoding ✔︎
- Test that a file walker sees a snapshot of tree even if the tree is modified after walking has started. ✔︎
- Generate file tree and walk with JDK7+ file walker and non-nio2 file walker. Attributes and files should be the same for both. ✔︎
- Performance gains will be measured from existing performance tests. ✔︎
- Expect existing test coverage will cover behavior of input/output snapshotting and file collection operations. ✔︎

## Story: Changes to reduce byte/char array allocations and array copying

- Just do it

## Story: Changes for adjusting collection sizes when final size is known

- Just do it

## Story: Reduce the in-memory size of the task history cache by interning file paths

### Implementation

- Use Guava's [`Interners.newWeakInterner()`](http://google.github.io/guava/releases/18.0/api/docs/com/google/common/collect/Interners.html#newWeakInterner%28%29) to create a cache `StringInterner` for sharing the file path Strings. Place this cache in `GlobalScopeServices` so that the instance lives across multiple builds in the daemon.
- use the `StringInterner` to intern all duplicate path names contained in `fileSnapshots`, `taskArtifacts`, `outputFileStates` and `fileHashes` caches.
- Implementation can be based on the solution developed in the spike. The commit is https://github.com/gradle/gradle/commit/d26d4ce1098e0eee9896279cbeabefb1ca3e871c .

### Test coverage

- Add basic unit test coverage for StringInterner
  - interning different string instances with similar content return the first instance that was interned
  - allows calling method with null, returns null in that case

## Story: Add caching to Specs returned from PatternSet.getAsSpecs()

Evaluating patterns is a hotspot in directory scanning. The default excludes patterns
contains 28 entries. Checking all rules for each file sums up in a lot of operations.
Adding caching will improve performance of subsequent incremental builds.

### Implementation

Assumption: PatternSet class is part of the Gradle Public API and we cannot change it's interface.

#### 1. phase - target release Gradle 2.9

Spike commit: https://github.com/lhotari/gradle/commit/f235117fd0b8b125a8220c45dca8ee9dc2331559

- Mainly based on the spike commit
- Move Spec<FileTreeElement> creation logic to separate factory class from PatternSet class (currently in getAsSpec, getAsIncludeSpec, getAsExcludeSpec methods)
- Add caching for Spec<FileTreeElement> instance creation and evaluation results
- Only add caching to Spec<FileTreeElement> instances that are created from the include and exclude patterns.
- A PatternSet can contain a list of includeSpecs and excludeSpecs. Don't add caching to these.

#### Test coverage for 1. phase

- Test that Spec<FileTreeElement> includes (added with PatternSet.include(Spec<FileTreeElement> spec)) and excludes (added with PatternSet.exclude(Spec<FileTreeElement> spec)) are not cached.
- Existing PatternSet tests cover rest of the changes since there are no planned behavioural or API changes for 1. phase.

#### 2. phase - target release Gradle 2.10

Goal: manage the cache instance in Gradle infrastructure instead of a singleton instance
- Use default non-caching PatternSpecFactory in PatternSet class, replace use of CachingPatternSpecFactory with plain PatternSpecFactory
- Make the Gradle infrastructure manage the CachingPatternSpecFactory instance.
- Create a new PatternSet subclass that takes the PatternSpecFactory instance in the constructor.
- Replace usage of PatternSet class with the new subclass in Gradle core code. Wire the CachingPatternSpecFactory instance to the instances of the new PatternSet subclass.

Goal: support suppressing default excludes
- Add support for suppressing the default excludes that are part of Ant integration legacy in Gradle. Default excludes aren't needed when there are specific include patterns. The default excludes support can be added to the same new PatternSet subclass that supports the separate (Caching)PatternSpecFactory.
- make SourceSets in Gradle use the new PatternSet implementation that supports suppressing the default excludes and uses the CachingPatternSpecFactory managed by the Gradle infrastructure (`GlobalScopeServices`)


## Story: High number of UnknownDomainObjectExceptions when resolving libraries in native projects.

TBD

## Story: Add "discovered" inputs to incremental tasks

This story adds a way for an incremental task to register additional inputs once execution has started.  At the end of execution, the discovered inputs are recorded in the task's execution history.

    void perform(IncrementalTaskInputs inputs) {
      getInputs().getFiles.each { source ->
        File discoveredInput = complicatedAnalysis(source)
        inputs.newInput(discoveredInput)
      }
      if (inputs.incremental) {
        inputs.outOfDate {
          // do stuff
        }
      }
    }

### Implementation

- Add `void newInput(File)` to IncrementalTaskInputs
- Add `Set<File> getDiscoveredInputs()` to IncrementalTaskInputsInternal
- StatefulIncrementalTaskInputs will implement `newInput` and `getDiscoveredInputs` to capture discovered inputs.
- Add new getDiscoveredInputFilesSnapshot/setDiscoveredInputFilesSnapshot methods to TaskExecution and LazyTaskExecution to record a separate discovered FileCollectionSnapshot. This means a task has 3 snapshots (inputs, outputs, discovered inputs).
- Update LazyTaskExecution and CacheBackedTaskHistoryRepository to handle the new discovered inputs snapshot. Maybe there's a higher level extraction that could be done here? The code for inputs/outputs/discovered are all similar.
- Add new DiscoveredInputFilesStateChangeRule that behaves identical to InputFilesStateChangeRule, except there is no input snapshot. Instead, the input snapshot is calculated by getting the hash of all files in the previousExecution's DiscoveredInputFilesSnapshot.
- DiscoveredInputFilesStateChangeRule's snapshotAfterTask action is to take the snapshot of all discovered files and add it to the current execution.
{'fraction_non_alphanumeric': 0.05250773161220922, 'fraction_numerical': 0.009614091703643942, 'mean_word_length': 4.902777777777778, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 19, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1860600', 'n_tokens_mistral': 3774, 'n_tokens_neox': 3601, 'n_words': 1797}