Loading an entire large file into memory will slow you down as your OS will need to start swapping pages of virtual memory. In such cases, it is best to deal with only the section of the file that you need.

In your case, you seem to be processing lines that have the same value in the first field together, so you could do something like:

my @lines = ();
my $current_key = '';

while (&lt;FILE&gt;) {
    my ($key) = split /,/;     # get first column
    if ($key ne $current_key) {
        # new key. Process all the lines from the previous key.
        if (@lines &gt; 0) {
            process(@lines);
        }
        @lines = ();
        $current_key = $key;
    }
    push @lines, $_
}
# don't forget the lines from the last key
if (@lines &gt; 0) {
    process(@lines);
}


This way, you are only storing in memory enough lines to make up one group.

(I am assuming that the input data is sorted or organized by the key. If that's not the case, you could make multiple passes through the file: a first pass to see what keys you will need to process, and subsequent passes to collect the lines associated with each key.)

Slowdown in large perl array
Hi,

I have a question that I've been thinking about for a long time.

How can you assure yourself that you've fully comprehended a concept or the true meaning of a theorem in mathematics?

I mean how can you realize that you totally get the concept and it's time to move on to the next pages of the book you're reading?

Thanks in advance for your responses.
