user138017: I am trying to go over all transactions data from every block on the bitcoin blockchain from the previous 4 years. With almost 2k transaction per block, it will take a lot of queries per block. I have a full node running locally and I tried two ways:
Python with RPC: This is very slow and keeps losing connection after some time (httpx.ReadTimeout)
Python with os.popen commands: Doesn't have the connection problem, but still very slow.
Would there be any other way? Any recommendation on how to analyze bulk data from the blockchain? The methods listed above are unfeasible given the time it would take.
EDIT: The problem isn't memory, but the time the bitcoin node takes to answer the queries.

user129640: 
With almost 2k transaction per block, it will take a lot of queries per block.

The getblock RPC supports a verbosity=2 argument which returns all the transactions in the block as JSON objects, so you can make do with a single query per block.
With RPC batching (i.e. sending multiple commands in a single request), you can do even better! You can query the transactions of n blocks with just 2 RPC requests: one to get all the block hashes, and one to get the blocks (with transactions). The below code snippet shows how you can implement both approaches, and includes a simple performance benchmark. On my machine, the batch approach is ~13x faster when querying the first 2000 blocks.
An example Python implementation, which should work out of the box if:

requests is installed (pip3 install requests)
bitcoind -signet -rpcuser=user -rpcpassword=pass

import json
import time
from typing import List

import requests


def get_n_blockhashes(n: int, start_height: int = 0):
    data = [{
        "method": "getblockhash",
        "params": [height]
    } for height in range(start_height, start_height + n)]
    hashes = [item["result"] for item in make_request(data)]
    return hashes


def get_block_transactions_single(block_hashes: List[str]):
    transactions = []
    for block_hash in block_hashes:
        data = {
            "method": "getblock",
            "params": [block_hash, 2]
        }
        block_data = make_request(data)["result"]
        transactions.append(block_data["tx"])

    return transactions


def get_block_transactions_batch(block_hashes: List[str]):
    data = [
        {
            "method": "getblock",
            "params": [block_hash, 2]
        } for block_hash in block_hashes
    ]
    transactions = [item["result"]["tx"] for item in make_request(data)]

    return transactions


def make_request(data):
    url = "http://user:pass@127.0.0.1:38332/"
    r = requests.post(url, data=json.dumps(data))
    assert r.status_code == 200
    return r.json()


def time_function(fn, *args: str, **kwargs) -> float:
    """Return average fn time execution and check that the last obtained blockheader hash matches last_hash_check """
    iters = 5
    start = time.perf_counter()
    for i in range(iters):
        fn(*args, **kwargs)
    avg_duration = (time.perf_counter() - start) / iters
    return avg_duration


if __name__ == '__main__':
    block_hashes = get_n_blockhashes(2000)
    print(f"single: {time_function(get_block_transactions_single, block_hashes):.4f}s")
    print(f"batch: {time_function(get_block_transactions_batch, block_hashes):.4f}s")


138017: <p>I am trying to go over all transactions data from every block on the bitcoin blockchain from the previous 4 years. With almost 2k transaction per block, it will take a lot of queries per block. I have a full node running locally and I tried two ways:</p>
<p>Python with RPC: This is very slow and keeps losing connection after some time (httpx.ReadTimeout)</p>
<p>Python with os.popen commands: Doesn't have the connection problem, but still very slow.</p>
<p>Would there be any other way? Any recommendation on how to analyze bulk data from the blockchain? The methods listed above are unfeasible given the time it would take.</p>
<p>EDIT: The problem isn't memory, but the time the bitcoin node takes to answer the queries.</p>
 129640: <blockquote>
<p>With almost 2k transaction per block, it will take a lot of queries per block.</p>
</blockquote>
<p>The <a href="https://developer.bitcoin.org/reference/rpc/getblock.html" rel="nofollow noreferrer"><code>getblock</code> RPC</a> supports a <code>verbosity=2</code> argument which returns all the transactions in the block as JSON objects, so you can make do with a single query per block.</p>
<p>With <a href="https://www.jsonrpc.org/specification#batch" rel="nofollow noreferrer">RPC batching</a> (i.e. sending multiple commands in a single request), you can do even better! You can query the transactions of <code>n</code> blocks with just 2 RPC requests: one to get all the block hashes, and one to get the blocks (with transactions). The below code snippet shows how you can implement both approaches, and includes a simple performance benchmark. On my machine, the batch approach is ~13x faster when querying the first 2000 blocks.</p>
<p>An example Python implementation, which should work out of the box if:</p>
<ul>
<li><code>requests</code> is installed (<code>pip3 install requests</code>)</li>
<li><code>bitcoind -signet -rpcuser=user -rpcpassword=pass</code></li>
</ul>
<pre class="lang-python prettyprint-override"><code>import json
import time
from typing import List

import requests


def get_n_blockhashes(n: int, start_height: int = 0):
    data = [{
        &quot;method&quot;: &quot;getblockhash&quot;,
        &quot;params&quot;: [height]
    } for height in range(start_height, start_height + n)]
    hashes = [item[&quot;result&quot;] for item in make_request(data)]
    return hashes


def get_block_transactions_single(block_hashes: List[str]):
    transactions = []
    for block_hash in block_hashes:
        data = {
            &quot;method&quot;: &quot;getblock&quot;,
            &quot;params&quot;: [block_hash, 2]
        }
        block_data = make_request(data)[&quot;result&quot;]
        transactions.append(block_data[&quot;tx&quot;])

    return transactions


def get_block_transactions_batch(block_hashes: List[str]):
    data = [
        {
            &quot;method&quot;: &quot;getblock&quot;,
            &quot;params&quot;: [block_hash, 2]
        } for block_hash in block_hashes
    ]
    transactions = [item[&quot;result&quot;][&quot;tx&quot;] for item in make_request(data)]

    return transactions


def make_request(data):
    url = &quot;http://user:pass@127.0.0.1:38332/&quot;
    r = requests.post(url, data=json.dumps(data))
    assert r.status_code == 200
    return r.json()


def time_function(fn, *args: str, **kwargs) -&gt; float:
    &quot;&quot;&quot;Return average fn time execution and check that the last obtained blockheader hash matches last_hash_check &quot;&quot;&quot;
    iters = 5
    start = time.perf_counter()
    for i in range(iters):
        fn(*args, **kwargs)
    avg_duration = (time.perf_counter() - start) / iters
    return avg_duration


if __name__ == '__main__':
    block_hashes = get_n_blockhashes(2000)
    print(f&quot;single: {time_function(get_block_transactions_single, block_hashes):.4f}s&quot;)
    print(f&quot;batch: {time_function(get_block_transactions_batch, block_hashes):.4f}s&quot;)
</code></pre>
