I have a text file composed of several subsequent tables. I need to get certain values from certain tables and save them in an output file. Every table has a header which contains a string that can be used to find specific tables. The size of these text files can vary from tenths of MB to some GB. I have written the following script to do the job:string = 'str'index = 20n = 2in_file = open('file.txt')out_file = open(out.txt, 'w')current_line = 0for i in range(-index,index+1):    for j in range(-index,index+1):        for line in in_file:            if string in line:                En = line.split().pop(4)                for line in in_file:                    current_line += 1                    if current_line == 2*(n+1)+2:                        x = line.split().pop(10)                    elif current_line == 3*(n+1)+2:                        y = line.split().pop(10)                    elif current_line == 4*(n+1)+2:                        z = line.split().pop(10)                        current_line = 0                        break                print i, j, En, x, y, z                data = %d %d %s %s %s %s\n % (i,j,En,x,y,z)                out_file.write(data)                breakin_file.close()out_file.close()The script reads the file line by line searching for the specified string ('str' in this example). When found, it then extracts a value from the line containing the string and continue reading the lines that form the data table itself. Since all the tables in the file have the same number of lines and columns, I've used the variable current_line to keep track of which line is read and to specify which line contains the data I need. The first two for-loops are just there to generate a pair of indexes that I need to be printed in the output file (in this case they are between -20 and 20). The script works fine. But since I've been learning python by myself for about one month, and the files I have to handle can be very big, I'm asking for advices on how to make the script more efficient, and overall, better.Also, since the tables are regular, I can know beforehand which are the lines that contain the values I need. So I was wondering, instead of reading all the lines in the file, is it possible to specify which lines have to be read and then jump directly between them?Sample input fileHere's a sample input file. I've included just some tables so you can have an idea how it's organized. This file is composed by two blocks with three tables each. In this sample file, the string table   # is what is used to find the data to be extracted.Sample output fileAnd here's a sample output file. Keep in mind that these two files are not equivalent! This output was created by my script using an input file containing 1681 blocks of 16 tables. Each table had 13 lines just as in the sample input file.
Gathering data from huge text files
python
All this nesting obfuscates your code pretty badly, so let's start by reducing it so it's easier to read. If you need to test something in your for loop, you should flip the test and use continue, like this:    for line in in_file:        if not string in line:            continue        En = line.split().pop(4)This saves you one block of nesting at least. You could also combine your two range calls with itertools.product. product performs a nested loop based on the two iterables it's passed. It essentially creates the exact nested loop you have with just one line:for i, j in itertools.product(range(-index,index+1), range(-index,index+1)):    for line in in_file:        if not string in line:            continue        En = line.split().pop(4)        for line in in_file:            current_line += 1            if current_line == 2*(n+1)+2:                x = line.split().pop(10)            elif current_line == 3*(n+1)+2:                y = line.split().pop(10)            elif current_line == 4*(n+1)+2:                z = line.split().pop(10)                current_line = 0                break        print i, j, En, x, y, z        data = %d %d %s %s %s %s\n % (i,j,En,x,y,z)        out_file.write(data)        breakIt's certainly more readable now. Though still confusing, why are you looping over in_file within the in_file nested group? It seems redundant but also I think you have misunderstood how file reading works. When you have read the first ten lines of a file object and then start a new loop over that same object you wont re-read the first ten lines. This means there's no reason to nest your inner for loop like this. This will iterate over the same values:for i, j in itertools.product(range(-index,index+1), range(-index,index+1)):    for line in in_file:        if not string in line:            continue        En = line.split().pop(4)    for line in in_file:        current_line += 1Now about that current_line, you actually don't need to use that, you could use enumerate instead to automatically count the iteration you're on. enumerate takes an iterable and returns both the value, ie. line but also a number which you can use to replace current_line. enumerate usually starts counting from 0, but you can pass an optional starting parameter:    for current_line, line in enumerate(in_file, 1):I think current_line is a confusing name especially now that it's next to line. index or i are what I'd use, personally.Also you're creating the same string twice, once to print and once to write. Just create data upfront so you can print the same value, this will avoid confusion with typo discrepancies between the two. You can manually add the newline in the write call instead. You should also use the new way of formatting, str.format, as it's type agnostic:        data = {} {} {} {} {}.format(i, j, En, x, y, z)        print (data)        out_file.write(data + '\n')Now let's see how this looks with these changes:for i, j in itertools.product(range(-index,index+1), range(-index,index+1)):    for line in in_file:        if not string in line:            continue        En = line.split().pop(4)        break    for idx, line in enumerate(in_file, 1):        if idx == 2*(n+1)+2:            x = line.split().pop(10)        elif idx == 3*(n+1)+2:            y = line.split().pop(10)        elif idx == 4*(n+1)+2:            z = line.split().pop(10)            break    data = {} {} {} {} {}.format(i, j, En, x, y, z)    print (data)    out_file.write(data + '\n')Now, a note about using open for your files. It's actually better to use with because it will always ensure your files are closed no matter what, even if error's occur during your script. It will unfortunately reintroduce nesting, but it it is worth it.with open(file.txt) as in_file, open(out.txt, w) as out_file:    for i, j in itertools.product(range(-index,index+1), range(-index,index+1)):Your names could definitely use work too. x, y and z don't mean anything to me. What is n? Is it a constant? Does it change? string is likewise vague, something like key would make more sense. index is confusing too, as it's only for specificying the range you loop over. Would boundary work perhaps? The problem is, your names are so vague I can't even suggest better ones because I don't know what these things do.Also you should match Python naming conventions. You use snake_case for current_line which is good, but En looks like a class due to the capital. It should just be en, or better yet a descriptive name. Even i and j would benefit from descriptive names if it was possible.