Q:

Why can't node handle this regex but python can?

I have a large text file that I am extracting URLs from. If I run:
import re

with open ('file.in', 'r') as fh:
   for match in re.findall(r'http://matchthis\.com', fh.read()):
      print match

it runs in a second or so user time and gets the URLs I was wanting, but if I run either of these:
var regex = /http:\/\/matchthis\.com/g;
fs.readFile('file.in', 'ascii', function(err, data) {
   while(match = regex.exec(data))
      console.log(match);
});

OR
fs.readFile('file.in', 'ascii', function(err, data) {
   var matches = data.match(/http:\/\/matchthis\.com/g);
   for (var i = 0; i < matches.length; ++i) {
      console.log(matches[i]);
   }
});

I get:
FATAL ERROR: CALL_AND_RETRY_0 Allocation failed - process out of memory

What is happening with the node.js regex engine? Is there any way I can modify things such that they work in node?
EDIT: The error appears to be fs centric as this also produces the error:
fs.readFile('file.in', 'ascii', function(err, data) {

});

file.in is around 800MB.

A:

You should process the file line by line using the streaming file interface. Something like this:
var fs = require('fs');

var byline = require('byline');

var input = fs.createReadStream('tmp.txt');
var lines = input.pipe(byline.createStream());

lines.on('readable', function(){
    var line = lines.read().toString('ascii');

    var matches = line.match(/http:\/\/matchthis\.com/g);
    for (var i = 0; i < matches.length; ++i) {
        console.log(matches[i]);
    }
});

In this example, I'm using the byline module to split the stream into lines so that you won't miss matches by getting partial chunks of lines per .read() call.
To elaborate more, what you were doing is allocating ~800MB of RAM as a Buffer (outside of V8's heap) and then converting that to an ASCII string (and thus transferring it into V8's heap), which will take at least 800MB and likely more depending on V8's internal optimizations. I believe V8 stores strings as UCS2 or UTF16, which means each character will be 2 bytes (given ASCII input) so your string would really be about 1600MB.
Node's max allocated heap space is 1.4GB, so by trying to create such a large string, you cause V8 to throw an exception.
Python does not have this problem because it does not have a maximum heap size and will chew through all of your RAM. As others have pointed out, you should also avoid fh.read() in Python since that will copy all the file data into RAM as a string instead of streaming it line by line with an iterator.

