<commit_before><br/>
<hr/>
<a name="io"></a> 
<h3>I/O</h3>
<h4>File Format</h4>
<ul><li><b>The default for streaming the content of STL containers was changed from object-wise
to member-wise.</b><p/>

We evaluated the impact of moving to MemberWise streaming using 5 different CMS data files:
<br/>
<ul><li>cms1.root an older split using level 99 Reco file 
</li><li>cms2.root a more recent non split raw data file 
</li><li>cms3.root a more recent non split Reco file 
</li><li>cms4.root is an example of the lepton plus jet analysis format known as a user PAT-tuple (split)
</li><li>cms5.root is an example AOD (analysis object dataset) format file.  It is not split because the objects here are a strict subset of the RECO objects. 
</li></ul>
We rewrote all the files using the v5.26/00 new basket clustering algorithm using both memberwise streaming and objectwise streaming. 
<p/>
In the table below the read time is the CPU time to completion including loading the libraries. 
When reading the file where always in the os cache (since we are focusing evaluating cpu time). 
<p/>
The number of event actually read varies from file set to file set but was calibrated to result 
in about 10s of cpu time.  The files are read using a library generated with TFile::MakeProject. 
<p/>
The object-wise files are between 2% and 10% larger compared to their member-wise counterpart.
<p/>
The CPU time for reading object-wise files is 12% higher for split files 
and 30% higher for non-split files. 
<p/>
So the improvement is significant enough to warrant switch the default from objectwise to memberwise.
<p/>

<table border="1" cellpadding="4">
<caption><em>Split files</em></caption>
<tr>
<th>Filename</th><th>Memberwise</th><th>Size</th><th>Cpu Time To read</th>
</tr>
<tr><td align="center">  cms1.root </td><td align="center">   N   </td><td align="center">  17.5 Gb  </td><td align="center">  10.55s +/- 0.15 (2200 entries) </td></tr>
<tr><td align="center">  cms1.root </td><td align="center">   Y   </td><td align="center">  16.8 Gb  </td><td align="center">  9.12s +/- 0.08 (2200 entries)   </td></tr>

<tr><td align="center">  cms4.root </td><td align="center">   N   </td><td align="center">  1.47 Gb  </td><td align="center">  10.18s +/- 0.19 (2500 entries)  </td></tr>
<tr><td align="center">  cms4.root </td><td align="center">   Y   </td><td align="center">  1.43 Gb  </td><td align="center">  9.24s +/- 0.06 (2500 entries)  </td></tr>
</table>

<p/>

<table border="1" cellpadding="4">
<caption><em>Non Split files</em></caption>
<tr>
<th>Filename</th><th>Memberwise</th><th>Size</th><th>Cpu Time To read</th>
</tr>
<tr><td align="center">cms2.root</td><td align="center">    N   </td><td align="center">  1.65 Gb  </td><td align="center">  10.95s +/- 0.05 (1000 entries)  </td></tr>
<tr><td align="center">cms2.root</td><td align="center">    Y   </td><td align="center">  1.53 Gb  </td><td align="center">   8.20s +/- 0.05 (1000 entries)  </td></tr>

<tr><td align="center">cms3.root</td><td align="center">    N   </td><td align="center">  0.780 Gb </td><td align="center"> 10.59s +/- 0.05 (700 entries)  </td></tr>
<tr><td align="center">cms3.root</td><td align="center">    Y   </td><td align="center">  0.717 Gb </td><td align="center">   8.29s +/- 0.08 (700 entries)  </td></tr>

<tr><td align="center">cms5.root</td><td align="center">    N   </td><td align="center">  1.55 Gb  </td><td align="center">  10.20s +/- 0.17 (700 entries)  </td></tr>
<tr><td align="center">cms5.root</td><td align="center">    Y   </td><td align="center">  1.40 Gb  </td><td align="center">   8.09s +/- 0.08 (700 entries) </td></tr>
</table>

</li><li>In the case of a data member which is a pointer to a STL container, eg:
<pre style="border:gray 1px solid;padding:0.5em 2em;background:#ffe">   std::container&lt;Data&gt; *fDataObjects;</pre>
and which is stored member-wise, add support for the schema evolution of the class 'Data'.
<br/>
This requires a change in the on file format used to store this type
of data members (i.e. by adding inline the version number of the class
'Data').
<br/>
To read file containing this construct and written with this revision
using an older version of ROOT you will need the following patches:
   <ul><li>For v5.22/00, you will need the <a href="http://root.cern.ch/viewvc?view=rev&amp;revision=33174">patch r33174</a>
     or v5.22/00k</li>
   <li>For v5.26/00, you will need <a href="http://root.cern.ch/viewvc?view=rev&amp;revision=33176">patch r33176</a>
     or v5.26/00c</li>
   </ul>
Additionally, we no longer allow the member wise streaming of a class which
has a custom streamer nor of any data members marked with //||
</li>
</ul>

<h4>Run time performance</h4>

We introduced an optimized infrastructure for reading objects using a StreamerInfo.  Rather than driving the streaming using a switch statement inside TStreamerInfo::ReadBuffer,
the streaming is now driven using a simple loop over a sequence of configured StreamerInfo actions.  This improves run-time performance by allowing a dramatic reduction in function calls and code
branches at the expense of some code duplication.  There are 3 versions of this loop implemented in TBufferFile and overloaded in TBufferXML and TBufferSQL:
<ol>
<li>virtual Int_t ReadSequence(const TStreamerInfoActions::TActionSequence &amp;sequence, void *object);</li>
<li>virtual Int_t ReadSequenceVecPtr(const TStreamerInfoActions::TActionSequence &amp;sequence, void *start_collection, void *end_collection);</li>
<li>virtual Int_t ReadSequence(const TStreamerInfoActions::TActionSequence &amp;sequence, void *start_collection, void *end_collection);</li>
</ol>
The 1st version is optimized to read a single object.  The 2nd version is optimized to read the content of TClonesArrays and vectors of pointers to objects.  The 3rd version is used to streamed any collections.
<br/><br/>
TBufferXML and TBufferSQL overload the loops to introduce extra code to help the buffer keep track of which streamer element is being streamed (this functionality is not used by TBufferFile.)
<br/><br/>
A TStreamerInfoActions::TActionSequence is an ordered sequence of configured actions.   
<br/><br/>
A configured action has both an action which is a free standing function and a configuration object deriving 
from TStreamerInfoActions::TConfiguration. The configuration contains information that is specific to the action
but varies from use to use, including the offset from the beginning of the object that needs to be updated.   
Other examples of configuration include the number of bits requested for storing a Double32_t or its factor and minimum.   
<br/><br/>
When the sequence is intended for a collection, the sequence has a configuration object deriving
from TStreamerInfoActions::TLoopConfiguration which contains for example the size of the element of
a vector or the pointers to the iterators functions (see below).
<br/><br/>
Each TStreamerInfo has 2 reading sequences, one for object-wise reading (GetReadObjectWiseActions)
and one for member-wise reading (GetReadMemberWiseActions) which is used when streaming a TClonesArray
of a vector of pointer to the type of objects described by the TClass.
<br/><br/>
Each collection proxy has at least one reading sequences, one for the reading each version of the
contained class layout.
<br/><br/>
Each case of the TStreamerInfo::ReadBuffer switch statement is replaced by 4 new action functions,
one for the object wise reading, one for the member wise reading for TClonesArray and vector of pointers,
one for the member wise reading for a vector of object and one for all other collections.
<br/><br/>
Each collection (proxy) needs to provide 5 new free standing functions:
<br/>
<pre style="border:gray 1px solid;padding:0.5em 2em;">
   // Set of functions to iterate easily throught the collection
   static const Int_t fgIteratorArenaSize = 16; // greater than sizeof(void*) + sizeof(UInt_t)

   typedef void (*CreateIterators_t)(void *collection, void **begin_arena, void **end_arena);
   virtual CreateIterators_t GetFunctionCreateIterators(Bool_t read = kTRUE) = 0; 
   // begin_arena and end_arena should contain the location of a memory arena of size fgIteratorSize. 
   // If the collection iterator are of that size or less, the iterators will be constructed in place in those location (new with placement)
   // Otherwise the iterators will be allocated via a regular new and their address returned by modifying the value of begin_arena and end_arena.
   
   typedef void* (*CopyIterator_t)(void *dest, const void *source);
   virtual CopyIterator_t GetFunctionCopyIterator(Bool_t read = kTRUE) = 0;
   // Copy the iterator source, into dest.   dest should contain the location of a memory arena of size fgIteratorSize.
   // If the collection iterator is of that size or less, the iterator will be constructed in place in this location (new with placement)
   // Otherwise the iterator will be allocated via a regular new and its address returned by modifying the value of dest.
   
   typedef void* (*Next_t)(void *iter, const void *end);
   virtual Next_t GetFunctionNext(Bool_t read = kTRUE) = 0;
   // iter and end should be pointers to respectively an iterator to be incremented and the result of collection.end()
   // If the iterator has not reached the end of the collection, 'Next' increment the iterator 'iter' and return 0 if 
   // the iterator reached the end.
   // If the end was not reached, 'Next' returns the address of the content pointed to by the iterator before the 
   // incrementation ; if the collection contains pointers, 'Next' will return the value of the pointer.
   
   typedef void (*DeleteIterator_t)(void *iter);
   typedef void (*DeleteTwoIterators_t)(void *begin, void *end);

   virtual DeleteIterator_t GetFunctionDeleteIterator(Bool_t read = kTRUE) = 0;
   virtual DeleteTwoIterators_t GetFunctionDeleteTwoIterators(Bool_t read = kTRUE) = 0;
   // If the size of the iterator is greater than fgIteratorArenaSize, call delete on the addresses,
   // Otherwise just call the iterator's destructor.
</pre>

<h4>TFile::MakeProject</h4>
Extend TFile::MakeProject to support genreflex, cases of user's data model where
the 2 distincts pointers point to a single object and more cases where we are 
missing the StreamerInfo and need to guess whether the symbol represent an enum,
a class or a namespace.

To use genreflex, call MakeProject with the "genreflex" option, for example:
<pre style="border:gray 1px solid;padding:0.5em 2em;background:#ffe">
   file->MakeProject(libdir,"*","NEW+genreflex");
</pre>
To make sure the library created by MakeProject does not double delete an object,
tell the StreamerElement representing one of the pointers pointing to the object
to never delete the object.  For example:

<pre style="border:gray 1px solid;padding:0.5em 2em;background:#ffe">
   TClass::AddRule("HepMC::GenVertex m_event attributes=NotOwner");
</pre>

MakeProject now implements a move constructor for each classes.   For the implementation, we 'use' the 'copy constructor' until the C++ compilers properly support the official move constructor notation.   Implementing a move constructor avoid having to delete and reconstruct resource during a std::vector resize and avoid the double delete induced by using the default copy constructor.

<h4>Misc.</h4>
<ul>
<li>Enable the use of the I/O customization rules on data members that are either a variable size array or a fixed size array.  For example:
<pre lang="cxx">
#pragma read sourceClass = "ACache" targetClass = "ACache" version     = "[8]" \
   source      = "Int_t *fArray; Int_t fN;" \
   target      = "fArray" \<commit_msg>Mention support for auto_ptr I/O<commit_after><br/>
<hr/>
<a name="io"></a> 
<h3>I/O</h3>
<h4>File Format</h4>
<ul><li><b>The default for streaming the content of STL containers was changed from object-wise
to member-wise.</b><p/>

We evaluated the impact of moving to MemberWise streaming using 5 different CMS data files:
<br/>
<ul><li>cms1.root an older split using level 99 Reco file 
</li><li>cms2.root a more recent non split raw data file 
</li><li>cms3.root a more recent non split Reco file 
</li><li>cms4.root is an example of the lepton plus jet analysis format known as a user PAT-tuple (split)
</li><li>cms5.root is an example AOD (analysis object dataset) format file.  It is not split because the objects here are a strict subset of the RECO objects. 
</li></ul>
We rewrote all the files using the v5.26/00 new basket clustering algorithm using both memberwise streaming and objectwise streaming. 
<p/>
In the table below the read time is the CPU time to completion including loading the libraries. 
When reading the file where always in the os cache (since we are focusing evaluating cpu time). 
<p/>
The number of event actually read varies from file set to file set but was calibrated to result 
in about 10s of cpu time.  The files are read using a library generated with TFile::MakeProject. 
<p/>
The object-wise files are between 2% and 10% larger compared to their member-wise counterpart.
<p/>
The CPU time for reading object-wise files is 12% higher for split files 
and 30% higher for non-split files. 
<p/>
So the improvement is significant enough to warrant switch the default from objectwise to memberwise.
<p/>

<table border="1" cellpadding="4">
<caption><em>Split files</em></caption>
<tr>
<th>Filename</th><th>Memberwise</th><th>Size</th><th>Cpu Time To read</th>
</tr>
<tr><td align="center">  cms1.root </td><td align="center">   N   </td><td align="center">  17.5 Gb  </td><td align="center">  10.55s +/- 0.15 (2200 entries) </td></tr>
<tr><td align="center">  cms1.root </td><td align="center">   Y   </td><td align="center">  16.8 Gb  </td><td align="center">  9.12s +/- 0.08 (2200 entries)   </td></tr>

<tr><td align="center">  cms4.root </td><td align="center">   N   </td><td align="center">  1.47 Gb  </td><td align="center">  10.18s +/- 0.19 (2500 entries)  </td></tr>
<tr><td align="center">  cms4.root </td><td align="center">   Y   </td><td align="center">  1.43 Gb  </td><td align="center">  9.24s +/- 0.06 (2500 entries)  </td></tr>
</table>

<p/>

<table border="1" cellpadding="4">
<caption><em>Non Split files</em></caption>
<tr>
<th>Filename</th><th>Memberwise</th><th>Size</th><th>Cpu Time To read</th>
</tr>
<tr><td align="center">cms2.root</td><td align="center">    N   </td><td align="center">  1.65 Gb  </td><td align="center">  10.95s +/- 0.05 (1000 entries)  </td></tr>
<tr><td align="center">cms2.root</td><td align="center">    Y   </td><td align="center">  1.53 Gb  </td><td align="center">   8.20s +/- 0.05 (1000 entries)  </td></tr>

<tr><td align="center">cms3.root</td><td align="center">    N   </td><td align="center">  0.780 Gb </td><td align="center"> 10.59s +/- 0.05 (700 entries)  </td></tr>
<tr><td align="center">cms3.root</td><td align="center">    Y   </td><td align="center">  0.717 Gb </td><td align="center">   8.29s +/- 0.08 (700 entries)  </td></tr>

<tr><td align="center">cms5.root</td><td align="center">    N   </td><td align="center">  1.55 Gb  </td><td align="center">  10.20s +/- 0.17 (700 entries)  </td></tr>
<tr><td align="center">cms5.root</td><td align="center">    Y   </td><td align="center">  1.40 Gb  </td><td align="center">   8.09s +/- 0.08 (700 entries) </td></tr>
</table>

</li><li>In the case of a data member which is a pointer to a STL container, eg:
<pre style="border:gray 1px solid;padding:0.5em 2em;background:#ffe">   std::container&lt;Data&gt; *fDataObjects;</pre>
and which is stored member-wise, add support for the schema evolution of the class 'Data'.
<br/>
This requires a change in the on file format used to store this type
of data members (i.e. by adding inline the version number of the class
'Data').
<br/>
To read file containing this construct and written with this revision
using an older version of ROOT you will need the following patches:
   <ul><li>For v5.22/00, you will need the <a href="http://root.cern.ch/viewvc?view=rev&amp;revision=33174">patch r33174</a>
     or v5.22/00k</li>
   <li>For v5.26/00, you will need <a href="http://root.cern.ch/viewvc?view=rev&amp;revision=33176">patch r33176</a>
     or v5.26/00c</li>
   </ul>
Additionally, we no longer allow the member wise streaming of a class which
has a custom streamer nor of any data members marked with //||
</li>
</ul>

<h4>Run time performance</h4>

We introduced an optimized infrastructure for reading objects using a StreamerInfo.  Rather than driving the streaming using a switch statement inside TStreamerInfo::ReadBuffer,
the streaming is now driven using a simple loop over a sequence of configured StreamerInfo actions.  This improves run-time performance by allowing a dramatic reduction in function calls and code
branches at the expense of some code duplication.  There are 3 versions of this loop implemented in TBufferFile and overloaded in TBufferXML and TBufferSQL:
<ol>
<li>virtual Int_t ReadSequence(const TStreamerInfoActions::TActionSequence &amp;sequence, void *object);</li>
<li>virtual Int_t ReadSequenceVecPtr(const TStreamerInfoActions::TActionSequence &amp;sequence, void *start_collection, void *end_collection);</li>
<li>virtual Int_t ReadSequence(const TStreamerInfoActions::TActionSequence &amp;sequence, void *start_collection, void *end_collection);</li>
</ol>
The 1st version is optimized to read a single object.  The 2nd version is optimized to read the content of TClonesArrays and vectors of pointers to objects.  The 3rd version is used to streamed any collections.
<br/><br/>
TBufferXML and TBufferSQL overload the loops to introduce extra code to help the buffer keep track of which streamer element is being streamed (this functionality is not used by TBufferFile.)
<br/><br/>
A TStreamerInfoActions::TActionSequence is an ordered sequence of configured actions.   
<br/><br/>
A configured action has both an action which is a free standing function and a configuration object deriving 
from TStreamerInfoActions::TConfiguration. The configuration contains information that is specific to the action
but varies from use to use, including the offset from the beginning of the object that needs to be updated.   
Other examples of configuration include the number of bits requested for storing a Double32_t or its factor and minimum.   
<br/><br/>
When the sequence is intended for a collection, the sequence has a configuration object deriving
from TStreamerInfoActions::TLoopConfiguration which contains for example the size of the element of
a vector or the pointers to the iterators functions (see below).
<br/><br/>
Each TStreamerInfo has 2 reading sequences, one for object-wise reading (GetReadObjectWiseActions)
and one for member-wise reading (GetReadMemberWiseActions) which is used when streaming a TClonesArray
of a vector of pointer to the type of objects described by the TClass.
<br/><br/>
Each collection proxy has at least one reading sequences, one for the reading each version of the
contained class layout.
<br/><br/>
Each case of the TStreamerInfo::ReadBuffer switch statement is replaced by 4 new action functions,
one for the object wise reading, one for the member wise reading for TClonesArray and vector of pointers,
one for the member wise reading for a vector of object and one for all other collections.
<br/><br/>
Each collection (proxy) needs to provide 5 new free standing functions:
<br/>
<pre style="border:gray 1px solid;padding:0.5em 2em;">
   // Set of functions to iterate easily throught the collection
   static const Int_t fgIteratorArenaSize = 16; // greater than sizeof(void*) + sizeof(UInt_t)

   typedef void (*CreateIterators_t)(void *collection, void **begin_arena, void **end_arena);
   virtual CreateIterators_t GetFunctionCreateIterators(Bool_t read = kTRUE) = 0; 
   // begin_arena and end_arena should contain the location of a memory arena of size fgIteratorSize. 
   // If the collection iterator are of that size or less, the iterators will be constructed in place in those location (new with placement)
   // Otherwise the iterators will be allocated via a regular new and their address returned by modifying the value of begin_arena and end_arena.
   
   typedef void* (*CopyIterator_t)(void *dest, const void *source);
   virtual CopyIterator_t GetFunctionCopyIterator(Bool_t read = kTRUE) = 0;
   // Copy the iterator source, into dest.   dest should contain the location of a memory arena of size fgIteratorSize.
   // If the collection iterator is of that size or less, the iterator will be constructed in place in this location (new with placement)
   // Otherwise the iterator will be allocated via a regular new and its address returned by modifying the value of dest.
   
   typedef void* (*Next_t)(void *iter, const void *end);
   virtual Next_t GetFunctionNext(Bool_t read = kTRUE) = 0;
   // iter and end should be pointers to respectively an iterator to be incremented and the result of collection.end()
   // If the iterator has not reached the end of the collection, 'Next' increment the iterator 'iter' and return 0 if 
   // the iterator reached the end.
   // If the end was not reached, 'Next' returns the address of the content pointed to by the iterator before the 
   // incrementation ; if the collection contains pointers, 'Next' will return the value of the pointer.
   
   typedef void (*DeleteIterator_t)(void *iter);
   typedef void (*DeleteTwoIterators_t)(void *begin, void *end);

   virtual DeleteIterator_t GetFunctionDeleteIterator(Bool_t read = kTRUE) = 0;
   virtual DeleteTwoIterators_t GetFunctionDeleteTwoIterators(Bool_t read = kTRUE) = 0;
   // If the size of the iterator is greater than fgIteratorArenaSize, call delete on the addresses,
   // Otherwise just call the iterator's destructor.
</pre>

<h4>TFile::MakeProject</h4>
Extend TFile::MakeProject to support genreflex, cases of user's data model where
the 2 distincts pointers point to a single object and more cases where we are 
missing the StreamerInfo and need to guess whether the symbol represent an enum,
a class or a namespace.

To use genreflex, call MakeProject with the "genreflex" option, for example:
<pre style="border:gray 1px solid;padding:0.5em 2em;background:#ffe">
   file->MakeProject(libdir,"*","NEW+genreflex");
</pre>
To make sure the library created by MakeProject does not double delete an object,
tell the StreamerElement representing one of the pointers pointing to the object
to never delete the object.  For example:

<pre style="border:gray 1px solid;padding:0.5em 2em;background:#ffe">
   TClass::AddRule("HepMC::GenVertex m_event attributes=NotOwner");
</pre>

MakeProject now implements a move constructor for each classes.   For the implementation, we 'use' the 'copy constructor' until the C++ compilers properly support the official move constructor notation.   Implementing a move constructor avoid having to delete and reconstruct resource during a std::vector resize and avoid the double delete induced by using the default copy constructor.

<h4>Misc.</h4>
<ul>
<li>Add support for streaming auto_ptr when generating their dictionary via rootcint</li>
<li>Enable the use of the I/O customization rules on data members that are either a variable size array or a fixed size array.  For example:
<pre lang="cxx">
#pragma read sourceClass = "ACache" targetClass = "ACache" version     = "[8]" \
   source      = "Int_t *fArray; Int_t fN;" \
   target      = "fArray" \
{'fraction_non_alphanumeric': 0.10657238999872215, 'fraction_numerical': 0.021638199088469565, 'mean_word_length': 4.482951891639421, 'pattern_counts': {'":': 0, '<': 451, '<?xml version=': 0, '>': 453, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '4701721', 'n_tokens_mistral': 7204, 'n_tokens_neox': 6774, 'n_words': 3005}