<jupyter_start><jupyter_text>MARCS Common BlocksIdentifying Fortran common blocks used throughout the MARCS model atmosphere package. The goal is to have a list of common blocks with an index of each routine they appear in.<jupyter_code>import fileinput as fi<jupyter_output><empty_output><jupyter_text>I have already run `grep` from the command line using```bashgrep -i -n "common" *.f > marcs_common_blocks.txt```The option `-i` indicates that the search should be case insensitive and `-n` returns the line number on which the search phrase is used. I've also piped the output to a file called `marcs_common_blocks.txt` for easy manipulation. All MARCS files have the `.f` Fortran extension, so all instances should be returned using this search.Now, let's look at the file structure.<jupyter_code>!head -n 5 marcs_common_blocks.txt<jupyter_output>algebn.f:50:      COMMON/SPACE1/A1,A2,A3,D1,D2,B1,B2,B3,C,Q1,O1,
archiv.f:23:C        COMMONS SHARED BY MAIN PROGRAM
archiv.f:25:      COMMON /TAUC/TAU(NDP),DTAULN(NDP),JTAU
archiv.f:26:      COMMON/STATEC/PRAD(NDP),PTURB(NDP),P(NDP),GG(NDP),ZZ(NDP),DD(NDP),
archiv.f:29:      COMMON /ROSSC/XKAPR(NDP),CROSS(NDP)<jupyter_text>The basic structure is `filename.f::` followed by the contents on the line. Since older Fortran required users to start in the 7th column, there is ample whitespace between the file information and the line content. The only exception is when a line is commented out. We can read the data in and separate it using the colon, `:`, as a delimeter.<jupyter_code>marcs_common_blocks = [line.split(':') for line in fi.input('marcs_common_blocks.txt')]<jupyter_output><empty_output><jupyter_text>Check to make sure we've acheived what we set out to do.<jupyter_code>marcs_common_blocks[0]<jupyter_output><empty_output><jupyter_text>Now we need to figure out whether we can easily access common block names. They are always surrounded by `/ /`, but we need to be careful to avoid irregular spacings. It is therefore advantageous to trim all whitespace in the third column before populating the list. We also want to strip new line characters and convert everything to lower case. However, let us also avoid commented lines and focus only on active common blocks. Comments are indicated by either `c`, `!`, or `* `.<jupyter_code>common_block_names = [entry[2].rstrip('\n').lower().replace(' ', '') 
                      for entry in marcs_common_blocks if entry[2][0].lower() not in ['c', '!', '*']]<jupyter_output><empty_output><jupyter_text>With commented entries removed, all common blocks can be identified by their initial `c` character. This will ensure that all unwanted entries that spuriously ended up in the list are removed. Then, we extract common block names by looking what is between the `/ /`.<jupyter_code>common_block_names = [entry[entry.find('/') + 1:entry.rfind('/')] for entry in common_block_names 
                      if entry[0].lower() == 'c']<jupyter_output><empty_output><jupyter_text>Check whether we've isolated common block names.<jupyter_code>common_block_names[0], common_block_names[50], common_block_names[-1]<jupyter_output><empty_output><jupyter_text>Remove duplicates from the list.<jupyter_code>common_block_names = list(set(common_block_names))<jupyter_output><empty_output><jupyter_text>Here's a full listing.<jupyter_code>common_block_names.sort()
common_block_names<jupyter_output><empty_output><jupyter_text>Thre are clearly some issues related to programming styles. Most repeated occurrences are the result of the user "closing" the common block or by including multiple common blocks on a single line. Let's remove those with some brute force tactics.<jupyter_code>second_round_names = [entry[entry.rfind('/') + 1:] for entry in common_block_names if entry.rfind('/') != -1]
second_round_names<jupyter_output><empty_output><jupyter_text>Only one entry has three common block names, but luckily the third name is already indexed, so we can move on. Get only the first common block name from the original list.<jupyter_code>first_round_names = [entry for entry in common_block_names if entry.rfind('/') == -1]<jupyter_output><empty_output><jupyter_text>Combine the two lists and remove duplicate entries.<jupyter_code>common_block_names = list(set(first_round_names + second_round_names))
common_block_names.sort()<jupyter_output><empty_output><jupyter_text>Now we are in a position to create a table of contents for our common blocks. It may be best for visualization if we write it in both plain text and markdown. First, a test to get a proper formatting.<jupyter_code>key = common_block_names[0]
print key.upper()
for entry in marcs_common_blocks:
    if entry[2].lower().find(key) != -1:
        print "\t {:16s} on line: {:4s}".format(entry[0], entry[1])<jupyter_output>AUXABUND
	 archiv.f         on line: 71  
	 eqmol_pe.f       on line: 102 
	 eqmol_pe_lu.f    on line: 102 
	 injon.f          on line: 108<jupyter_text>That seems to be quite reasonable. Now for all keys,<jupyter_code>for key in common_block_names:
    print key.upper()
    for entry in marcs_common_blocks:
        if entry[2].lower().find(key) != -1:
            print "\t {:16s} on line: {:4s}".format(entry[0], entry[1])
        else:
            pass<jupyter_output>AUXABUND
	 archiv.f         on line: 71  
	 eqmol_pe.f       on line: 102 
	 eqmol_pe_lu.f    on line: 102 
	 injon.f          on line: 108 
BABSMA
	 oldsta.f         on line: 24  
	 oldsta_test.f    on line: 20  
BPLC
	 bpl.f            on line: 4   
C030617
	 archiv.f         on line: 97  
	 detabs.f         on line: 119 
	 osabsko.f        on line: 65  
CA1
	 inabs.f          on line: 96  
	 osinit.f         on line: 21  
	 tabs.f           on line: 20  
	 tabs.f           on line: 36  
CA2
	 detabs.f         on line: 117 
	 inabs.f          on line: 99  
	 osabsko.f        on line: 31  
	 osabsko.f        on line: 49  
	 tabs.f           on line: 39  
CA3
	 inabs.f          on line: 100 
	 osabsko.f        on line: 31  
	 osabsko.f        on line: 50  
CA4
	 osabsko.f        on line: 32  
	 osabsko.f        on line: 51  
	 tabs.f           on line: 40  
CA5
	 detabs.f         on line: 56  
	 detabs.f         on line: 90  
	 osabsko.f        on line: 52  
	 tryck.f          on line:[...]<jupyter_text>That clearly works, so let's output that information to a plain text file.<jupyter_code>plaint = open('common_block_index.txt', 'w')
for key in common_block_names:
    plaint.write(key.upper() + '\n')
    for entry in marcs_common_blocks:
        if entry[2].lower().find(key) != -1:
            plaint.write("\t {:30s} on line: {:4s} \n".format(entry[0], entry[1]))
        else:
            pass
    plaint.write('\n')
plaint.close()<jupyter_output><empty_output><jupyter_text>And in markdown for easy reading online.<jupyter_code>markd = open('common_block_index.md', 'w')
for key in common_block_names:
    markd.write('## ' + key.upper() + '\n')
    for entry in marcs_common_blocks:
        if entry[2].lower().find(key) != -1:
            markd.write("\t {:30s} on line: {:4s} \n".format(entry[0], entry[1]))
        else:
            pass
    markd.write('\n')
markd.close()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10123593945285377, 'fraction_numerical': 0.02055270101374809, 'mean_word_length': 3.5931122448979593, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 63, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '315888', 'license': 'MIT', 'n_tokens_mistral': 2512, 'n_tokens_neox': 2339, 'n_words': 755, 'path': 'Projects/senap/common_blocks.ipynb', 'repo_name': 'gfeiden/Notebook', 'size': '7201'}