I have Python code that splits a given large csv into smaller csvs. This large CSV has an ID column (column 1), which consecutive entries in the csv can share. The large csv might look something like this:sfsddf8sdf8, 123, -234, dfsdfe, fsefsddfesfsddf8sdf8, 754,  464, sdfgdg, QFdgdfgdrsfsddf8sdf8, 485,  469, mgyhjd, brgfgrdfgsfsddf8sdf8, 274, -234, dnthfh, jyfhghfthsfsddf8sdf8, 954, -145, lihgyb, fthgfhthjpowedfnsk93, 257, -139, sdfsfs, sdfsdfsdfpowedfnsk93, 284, -126, sdgdgr, sdagssdffpowedfnsk93, 257, -139, srfgfr, sdffffssserfsfeeeeef, 978,  677, dfgdrg, ssdttnmmmetc...The IDs are not sorted alphabetically in the input file, but consecutive identical IDs are grouped together.My code does not split the IDs into different csvs, ensuring that each id appears in only one output csv.My code is:import pandas as pdimport osdef iterateIDs(file): #create chunks based on tripID    csv_reader = pd.read_csv(file, iterator=True, chunksize=1, header=None)    first_chunk = csv_reader.get_chunk()    id = first_chunk.iloc[0,0]    chunk = pd.DataFrame(first_chunk)    for l in csv_reader:        if id == l.iloc[0,0] or len(chunk)<1000000: #Keep adding to chunk if less than 1,000,000, or in middle of trip            id = l.iloc[0,0]            chunk = chunk.append(l)            continue        id = l.iloc[0,0]        yield chunk        chunk = pd.DataFrame(l)    yield chunkwaypoint_filesize = os.stat('TripRecordsReportWaypoints.csv').st_size #checks filesizeif waypoint_filesize > 100000000: #if file too big, split into seperate chunks    chunk_count = 1    chunk_Iterate = iterateIDs(TripRecordsReportWaypoints.csv)    for chunk in chunk_Iterate:        chunk.to_csv('SmallWaypoints_{}.csv'.format(chunk_count),header=None,index=None)        chunk_count = chunk_count+1However, this code runs very slowly. I tested it on a small file, 284 MB and 3.5 million rows, however it took over an hour to run. Is there any way I can achieve this result quicker? I don't mind if it's outside of python.
Python code to split csv into smaller csvs, not splitting IDs
python;time limit exceeded;csv;pandas
If I understand correctly, you want to split a file into smaller files, based on size (no more than 1000000 lines) and ID (no ID should be split among files).If that's the case, I think you're over-complicating things. You don't need pandas and you definitely don't need to keep all data in memory.You just need two counters, one for the amount of lines you've written and one for the index of the file to write.Sample code (of course, replace file names with what you need, or move the check after the write to start from 0 instead of 1):current_id = ''index = 0written_lines = 0max_lines = 1000000with open('data.csv', 'r') as input_file:    for line in input_file:        values = line.split(',')        if (current_id != values[0]) or (written_lines > max_lines):            index += 1            current_id = values[0]        with open('output_{:08d}.csv'.format(index), 'a') as output_file:            output_file.write(line)            written_lines += 1EDIT: This works assuming the file is sorted or that at least the IDs are grouped together as you said in the comment.