user82632: Background:
For a research project, I am trying to periodically clear the content of my Bitcoin node's mempool. For this purpose I have created a new thread in init.cpp using the boost threadgroup (I have experimented with this successfully with a different project in the past and I am sure this is not the cause of the problem) in the Bitcoin Core software. In the new thread, I periodically call a function func (defined below), the sole purpose of which is to clear the content of the node's mempool.
void func()
{
    LOCK2(cs_main, mempool.cs);
    AssertLockHeld(cs_main);
    AssertLockHeld(mempool.cs);
    mempool.clear();
}

I check for and lock critical sections that I think I should before I modify the mempool (the code above mimics what happens when a new transaction is added to the mempool; I basically looked at what happens in the NetMsgType::TX branch of ProcessMessage in net_processing.cpp and then AcceptToMemoryPoolWorker in validation.cpp and mimic that).
Issue:
When I run Bitcoin after successfully compiling my code, I sometimes immediately (i.e., as soon as the GUI window pops up) and sometimes much later in the execution face a segfault that causes the program to crash.
ubuntu@root:~$ bitcoin-qt -mpctimeout=1
Segmentation fault (core dumped)

In the code above, the function func is executed every 1 minute (represented by mpctimeout=1).
Question:
What could be causing the segfault? Could it possibly be race conditions when trying to get a lock on the mempool or execution of the main thread? It is my understanding that LOCK2 should keep trying to get a lock on the critical sections when they become available in case they are already locked by another thread/process. Since these locks only exist within the scope of the function func, they should ideally be released when returning from the function.
Edit # 1:
Some debugging shows that the segfault doesn't occur while func is being executed but while the thread is put in sleep for mpctimeout amount of time. This leaves me wondering whether clearing the mempool has some long-term effects? I have made sure that the lock is released by verifying that transactions are indeed accepted to the mempool while the new thread is asleep. The segfault still occurs even if the mpctimeout value is large (e.g., 1 hour).
FWIW, I use boost::this_thread::sleep_for to put the thread to sleep. I assume this works without fault and that the segfault is instead caused in the long run due to clearing the mempool. Any tips?
Edit # 2:
Below is the output of debugging the binary through gdb:
(gdb) run
Starting program: /usr/local/bin/bitcoin-qt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffe9900700 (LWP 25643)]
[New Thread 0x7fffdf37f700 (LWP 25644)]
[New Thread 0x7fffdeb7e700 (LWP 25645)]
[New Thread 0x7fffdd0e7700 (LWP 25646)]
[New Thread 0x7fffc7cd2700 (LWP 25648)]
[New Thread 0x7fffc74d1700 (LWP 25649)]
[New Thread 0x7fffc6cd0700 (LWP 25650)]
[New Thread 0x7fffc64cf700 (LWP 25651)]
[New Thread 0x7fffc5cce700 (LWP 25652)]
[New Thread 0x7fffc54cd700 (LWP 25653)]
[New Thread 0x7fffc4ccc700 (LWP 25654)]
[New Thread 0x7fffa7fff700 (LWP 25655)]
[New Thread 0x7fffa77fe700 (LWP 25656)]
[New Thread 0x7fffa6ffd700 (LWP 25657)]
[New Thread 0x7fffa67fc700 (LWP 25658)]
[New Thread 0x7fffa5ffb700 (LWP 25659)]
[New Thread 0x7fffa57fa700 (LWP 25660)]
[New Thread 0x7fffa4f63700 (LWP 25661)]
[New Thread 0x7fff8dffd700 (LWP 25662)]
[New Thread 0x7fff8d7fc700 (LWP 25663)]
[New Thread 0x7fff8cffb700 (LWP 25664)]
[New Thread 0x7fff7bfff700 (LWP 25665)]
[New Thread 0x7fff7b7fe700 (LWP 25666)]
[New Thread 0x7fff7affd700 (LWP 25667)]
[New Thread 0x7fff7a7fc700 (LWP 25668)]
[New Thread 0x7fff79ffb700 (LWP 25669)]
[New Thread 0x7fff797fa700 (LWP 25670)]
[New Thread 0x7fff78ff9700 (LWP 25671)]
[New Thread 0x7fff53fff700 (LWP 25672)]
[New Thread 0x7fff537fe700 (LWP 25673)]
[New Thread 0x7fff4a174700 (LWP 25677)]
[New Thread 0x7ffed1a7b700 (LWP 25678)]
[Thread 0x7ffed1a7b700 (LWP 25678) exited]
[New Thread 0x7ffed127a700 (LWP 25679)]
[New Thread 0x7ffed0a79700 (LWP 25680)]
[New Thread 0x7ffecbfff700 (LWP 25681)]
[New Thread 0x7ffecb7fe700 (LWP 25682)]
[New Thread 0x7ffecaffd700 (LWP 25683)]
[New Thread 0x7ffeca7fc700 (LWP 25684)]
[New Thread 0x7ffec9ffb700 (LWP 25685)]
[New Thread 0x7ffec97fa700 (LWP 25686)]
[New Thread 0x7ffec8ff9700 (LWP 25687)]
[Thread 0x7ffed127a700 (LWP 25679) exited]
[Thread 0x7ffecb7fe700 (LWP 25682) exited]

Thread 40 "bitcoin-msghand" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec9ffb700 (LWP 25685)]
__gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x1a45fe36190ee438)
    at /usr/include/c++/7/ext/atomicity.h:96
96        __atomic_add(__mem, __val);
(gdb)

I suppose that the segfault is caused when trying to update an atomic variable.
Edit # 3:
gdb backtrace
#0  0x0000555555854178 in __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0xf) at /usr/include/c++/7/ext/atomicity.h:96
#1  0x0000555555854178 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_add_ref_copy() (this=0x7) at /usr/include/c++/7/bits/shared_ptr_base.h:138
#2  0x0000555555854178 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count(std::__shared_count<(__gnu_cxx::_Lock_policy)2> const&) (__r=..., this=0x7ffef5ff71b8)
    at /usr/include/c++/7/bits/shared_ptr_base.h:691
#3  0x0000555555854178 in std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2>::__shared_ptr(std::__shared_ptr<CTransaction const, (__gnu_cxx::_Lock_policy)2> const&)
    (this=0x7ffef5ff71b0) at /usr/include/c++/7/bits/shared_ptr_base.h:1121
#4  0x0000555555854178 in std::shared_ptr<CTransaction const>::shared_ptr(std::shared_ptr<CTransaction const> const&) (this=0x7ffef5ff71b0)
    at /usr/include/c++/7/bits/shared_ptr.h:119
#5  0x0000555555854178 in CTxMemPoolEntry::GetSharedTx() const (this=0x7ffee12106a0) at ./txmempool.h:101
#6  0x0000555555854178 in PartiallyDownloadedBlock::InitData(CBlockHeaderAndShortTxIDs const&, std::vector<std::pair<uint256, std::shared_ptr<CTransaction const> >, std::allocator<std::pair<uint256, std::shared_ptr<CTransaction const> > > > const&) (this=this@entry=0x7ffef5ff9f30, cmpctblock=..., extra_txn=std::vector of length 100, capacity 100 = {...})
    at blockencodings.cpp:114
#7  0x00005555556a2d4a in ProcessMessage(CNode*, std::__cxx11::string const&, CDataStream&, int64_t, CChainParams const&, CConnman*, std::atomic<bool> const&, bool)
    (pfrom=pfrom@entry=0x7ffedc0010d0, strCommand="cmpctblock", vRecv=..., nTimeReceived=1593549860206352, chainparams=..., connman=0x7fffbc04ba20, interruptMsgProc=..., enable_bip61=false) at net_processing.cpp:3051
#8  0x00005555556a6f01 in PeerLogicValidation::ProcessMessages(CNode*, std::atomic<bool>&) (this=<optimized out>, pfrom=0x7ffedc0010d0, interruptMsgProc=...) at net_processing.cpp:3629
#9  0x000055555566b19e in CConnman::ThreadMessageHandler() (this=0x7fffbc04ba20) at net.cpp:1966


user52021: The only place clear is used on the mempool is after all peers are down. Presumably there are references to it that you can't just leave dangling.
See CTxMemPool::Expire in txmempool.cpp for how expiration works.  It wouldn't be hard to modify that existing function to just remove everything at a particular point.
I assume you're the author of http://people.bu.edu/staro/Characterizing_churn.pdf ?
I saw that paper back on May 22nd and I found the claims of the paper to be false to the best of my ability to determine.  If the paper was correct nodes would see large numbers of getblocktxn messages-- patching in missing transactions that showed up in blocks, yet they see fairly few.
To give some concrete figures, yesterday my node received 210 getblocktxn requests from ~66 peers. There were 162 blocks. So that is about 0.0196 request per block*peer. I sent 3097 compact blocks during the same interval. My own rate of having missing transactions and needing a getblocktxn was 6.7%, which is not wildly inconsistent from what I see with other peers: it's higher because I'll only see 1/N-th my peers getblocktxn, since I'm not always the first to send them a block. But if I assume that the rate that I sent compact blocks (28.9%) reflects the rate that I'm first, I estimate my peers getblocktxn rate to be almost exactly the same as mine.
I did more analysis with multiple nodes back when I first saw the paper, but I'm not sure where I put the figures. :)
I believe the paper makes an error taking stats from "bitnodes":   There are enormous numbers of fake nodes on the networks which don't behave much all like real nodes-- e.g. for the purpose of spying on transactions to deanonymize bitcoin users or academic research, most of "nodes" displayed on bitnodes are of this sort. It's irrelevant to the network if fake 'monitoring' nodes churn a lot.
You might also be interested in the FIBRE protocol, which renders block propagation times close to to independent of the presence of a few missing transactions (Can someone please explain FIBRE to me like I'm 5 and why is it useful?) and Erlay (https://arxiv.org/abs/1905.10518) which uses reconciliation for transaction relay  to reduce relay overheads.

82632: <p><strong>Background:</strong></p>
<p>For a research project, I am trying to periodically clear the content of my Bitcoin node's mempool. For this purpose I have created a new thread in <code>init.cpp</code> using the <code>boost</code> <code>threadgroup</code> (I have experimented with this successfully with a different project in the past and I am sure this is not the cause of the problem) in the Bitcoin Core software. In the new thread, I periodically call a function <code>func</code> (defined below), the sole purpose of which is to clear the content of the node's mempool.</p>
<pre><code>void func()
{
    LOCK2(cs_main, mempool.cs);
    AssertLockHeld(cs_main);
    AssertLockHeld(mempool.cs);
    mempool.clear();
}
</code></pre>
<p>I check for and lock critical sections that I think I should before I modify the mempool (the code above mimics what happens when a new transaction is added to the mempool; I basically looked at what happens in the <code>NetMsgType::TX</code> branch of <code>ProcessMessage</code> in <code>net_processing.cpp</code> and then <code>AcceptToMemoryPoolWorker</code> in <code>validation.cpp</code> and mimic that).</p>
<p><strong>Issue:</strong></p>
<p>When I run Bitcoin after successfully compiling my code, I sometimes <em>immediately</em> (i.e., as soon as the GUI window pops up) and sometimes much later in the execution face a segfault that causes the program to crash.</p>
<pre><code>ubuntu@root:~$ bitcoin-qt -mpctimeout=1
Segmentation fault (core dumped)
</code></pre>
<p>In the code above, the function <code>func</code> is executed every 1 minute (represented by <code>mpctimeout=1</code>).</p>
<p><strong>Question:</strong></p>
<p>What could be causing the segfault? Could it possibly be race conditions when trying to get a lock on the mempool or execution of the main thread? It is my understanding that <code>LOCK2</code> should keep trying to get a lock on the critical sections when they become available in case they are already locked by another thread/process. Since these locks only exist within the scope of the function <code>func</code>, they <em>should</em> ideally be released when returning from the function.</p>
<p><strong>Edit # 1</strong>:</p>
<p>Some debugging shows that the segfault doesn't occur while <code>func</code> is being executed but while the thread is put in sleep for <code>mpctimeout</code> amount of time. <strong>This leaves me wondering whether clearing the mempool has some long-term effects?</strong> I have made sure that the lock is released by verifying that transactions are indeed accepted to the mempool while the new thread is asleep. The segfault still occurs even if the <code>mpctimeout</code> value is large (e.g., 1 hour).</p>
<p>FWIW, I use <code>boost::this_thread::sleep_for</code> to put the thread to sleep. I assume this works without fault and that the segfault is instead caused in the long run due to clearing the mempool. Any tips?</p>
<p><strong>Edit # 2:</strong></p>
<p>Below is the output of debugging the binary through <code>gdb</code>:</p>
<pre><code>(gdb) run
Starting program: /usr/local/bin/bitcoin-qt
[Thread debugging using libthread_db enabled]
Using host libthread_db library &quot;/lib/x86_64-linux-gnu/libthread_db.so.1&quot;.
[New Thread 0x7fffe9900700 (LWP 25643)]
[New Thread 0x7fffdf37f700 (LWP 25644)]
[New Thread 0x7fffdeb7e700 (LWP 25645)]
[New Thread 0x7fffdd0e7700 (LWP 25646)]
[New Thread 0x7fffc7cd2700 (LWP 25648)]
[New Thread 0x7fffc74d1700 (LWP 25649)]
[New Thread 0x7fffc6cd0700 (LWP 25650)]
[New Thread 0x7fffc64cf700 (LWP 25651)]
[New Thread 0x7fffc5cce700 (LWP 25652)]
[New Thread 0x7fffc54cd700 (LWP 25653)]
[New Thread 0x7fffc4ccc700 (LWP 25654)]
[New Thread 0x7fffa7fff700 (LWP 25655)]
[New Thread 0x7fffa77fe700 (LWP 25656)]
[New Thread 0x7fffa6ffd700 (LWP 25657)]
[New Thread 0x7fffa67fc700 (LWP 25658)]
[New Thread 0x7fffa5ffb700 (LWP 25659)]
[New Thread 0x7fffa57fa700 (LWP 25660)]
[New Thread 0x7fffa4f63700 (LWP 25661)]
[New Thread 0x7fff8dffd700 (LWP 25662)]
[New Thread 0x7fff8d7fc700 (LWP 25663)]
[New Thread 0x7fff8cffb700 (LWP 25664)]
[New Thread 0x7fff7bfff700 (LWP 25665)]
[New Thread 0x7fff7b7fe700 (LWP 25666)]
[New Thread 0x7fff7affd700 (LWP 25667)]
[New Thread 0x7fff7a7fc700 (LWP 25668)]
[New Thread 0x7fff79ffb700 (LWP 25669)]
[New Thread 0x7fff797fa700 (LWP 25670)]
[New Thread 0x7fff78ff9700 (LWP 25671)]
[New Thread 0x7fff53fff700 (LWP 25672)]
[New Thread 0x7fff537fe700 (LWP 25673)]
[New Thread 0x7fff4a174700 (LWP 25677)]
[New Thread 0x7ffed1a7b700 (LWP 25678)]
[Thread 0x7ffed1a7b700 (LWP 25678) exited]
[New Thread 0x7ffed127a700 (LWP 25679)]
[New Thread 0x7ffed0a79700 (LWP 25680)]
[New Thread 0x7ffecbfff700 (LWP 25681)]
[New Thread 0x7ffecb7fe700 (LWP 25682)]
[New Thread 0x7ffecaffd700 (LWP 25683)]
[New Thread 0x7ffeca7fc700 (LWP 25684)]
[New Thread 0x7ffec9ffb700 (LWP 25685)]
[New Thread 0x7ffec97fa700 (LWP 25686)]
[New Thread 0x7ffec8ff9700 (LWP 25687)]
[Thread 0x7ffed127a700 (LWP 25679) exited]
[Thread 0x7ffecb7fe700 (LWP 25682) exited]

Thread 40 &quot;bitcoin-msghand&quot; received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffec9ffb700 (LWP 25685)]
__gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0x1a45fe36190ee438)
    at /usr/include/c++/7/ext/atomicity.h:96
96        __atomic_add(__mem, __val);
(gdb)
</code></pre>
<p>I suppose that the segfault is caused when trying to update an <code>atomic</code> variable.</p>
<p><strong>Edit # 3:</strong></p>
<p><code>gdb</code> backtrace</p>
<pre><code>#0  0x0000555555854178 in __gnu_cxx::__atomic_add_dispatch (__val=1, __mem=0xf) at /usr/include/c++/7/ext/atomicity.h:96
#1  0x0000555555854178 in std::_Sp_counted_base&lt;(__gnu_cxx::_Lock_policy)2&gt;::_M_add_ref_copy() (this=0x7) at /usr/include/c++/7/bits/shared_ptr_base.h:138
#2  0x0000555555854178 in std::__shared_count&lt;(__gnu_cxx::_Lock_policy)2&gt;::__shared_count(std::__shared_count&lt;(__gnu_cxx::_Lock_policy)2&gt; const&amp;) (__r=..., this=0x7ffef5ff71b8)
    at /usr/include/c++/7/bits/shared_ptr_base.h:691
#3  0x0000555555854178 in std::__shared_ptr&lt;CTransaction const, (__gnu_cxx::_Lock_policy)2&gt;::__shared_ptr(std::__shared_ptr&lt;CTransaction const, (__gnu_cxx::_Lock_policy)2&gt; const&amp;)
    (this=0x7ffef5ff71b0) at /usr/include/c++/7/bits/shared_ptr_base.h:1121
#4  0x0000555555854178 in std::shared_ptr&lt;CTransaction const&gt;::shared_ptr(std::shared_ptr&lt;CTransaction const&gt; const&amp;) (this=0x7ffef5ff71b0)
    at /usr/include/c++/7/bits/shared_ptr.h:119
#5  0x0000555555854178 in CTxMemPoolEntry::GetSharedTx() const (this=0x7ffee12106a0) at ./txmempool.h:101
#6  0x0000555555854178 in PartiallyDownloadedBlock::InitData(CBlockHeaderAndShortTxIDs const&amp;, std::vector&lt;std::pair&lt;uint256, std::shared_ptr&lt;CTransaction const&gt; &gt;, std::allocator&lt;std::pair&lt;uint256, std::shared_ptr&lt;CTransaction const&gt; &gt; &gt; &gt; const&amp;) (this=this@entry=0x7ffef5ff9f30, cmpctblock=..., extra_txn=std::vector of length 100, capacity 100 = {...})
    at blockencodings.cpp:114
#7  0x00005555556a2d4a in ProcessMessage(CNode*, std::__cxx11::string const&amp;, CDataStream&amp;, int64_t, CChainParams const&amp;, CConnman*, std::atomic&lt;bool&gt; const&amp;, bool)
    (pfrom=pfrom@entry=0x7ffedc0010d0, strCommand=&quot;cmpctblock&quot;, vRecv=..., nTimeReceived=1593549860206352, chainparams=..., connman=0x7fffbc04ba20, interruptMsgProc=..., enable_bip61=false) at net_processing.cpp:3051
#8  0x00005555556a6f01 in PeerLogicValidation::ProcessMessages(CNode*, std::atomic&lt;bool&gt;&amp;) (this=&lt;optimized out&gt;, pfrom=0x7ffedc0010d0, interruptMsgProc=...) at net_processing.cpp:3629
#9  0x000055555566b19e in CConnman::ThreadMessageHandler() (this=0x7fffbc04ba20) at net.cpp:1966
</code></pre>
 52021: <p>The only place clear is used on the mempool is after all peers are down. Presumably there are references to it that you can't just leave dangling.</p>
<p>See CTxMemPool::Expire in txmempool.cpp for how expiration works.  It wouldn't be hard to modify that existing function to just remove everything at a particular point.</p>
<p>I assume you're the author of <a href="http://people.bu.edu/staro/Characterizing_churn.pdf" rel="noreferrer">http://people.bu.edu/staro/Characterizing_churn.pdf</a> ?</p>
<p>I saw that paper back on May 22nd and I found the claims of the paper to be false to the best of my ability to determine.  If the paper was correct nodes would see large numbers of getblocktxn messages-- patching in missing transactions that showed up in blocks, yet they see fairly few.</p>
<p>To give some concrete figures, yesterday my node received 210 getblocktxn requests from ~66 peers. There were 162 blocks. So that is about 0.0196 request per block*peer. I sent 3097 compact blocks during the same interval. My own rate of having missing transactions and needing a getblocktxn was 6.7%, which is not wildly inconsistent from what I see with other peers: it's higher because I'll only see 1/N-th my peers getblocktxn, since I'm not always the first to send them a block. But if I assume that the rate that I sent compact blocks (28.9%) reflects the rate that I'm first, I estimate my peers getblocktxn rate to be almost exactly the same as mine.</p>
<p>I did more analysis with multiple nodes back when I first saw the paper, but I'm not sure where I put the figures. :)</p>
<p>I believe the paper makes an error taking stats from &quot;bitnodes&quot;:   There are enormous numbers of fake nodes on the networks which don't behave much all like real nodes-- e.g. for the purpose of spying on transactions to deanonymize bitcoin users or academic research, most of &quot;nodes&quot; displayed on bitnodes are of this sort. It's irrelevant to the network if fake 'monitoring' nodes churn a lot.</p>
<p>You might also be interested in the FIBRE protocol, which renders block propagation times close to to independent of the presence of a few missing transactions (<a href="https://bitcoin.stackexchange.com/questions/56485/can-someone-please-explain-fibre-to-me-like-im-5-and-why-is-it-useful">Can someone please explain FIBRE to me like I&#39;m 5 and why is it useful?</a>) and Erlay (<a href="https://arxiv.org/abs/1905.10518" rel="noreferrer">https://arxiv.org/abs/1905.10518</a>) which uses reconciliation for transaction relay  to reduce relay overheads.</p>
