I need to split a file into unique file names.I can do it with sed command eg,  sed -n '/scaffold135_/w 135-scaf.txt' input file.txt but it's time consuming so I need a smart way to do it faster. Below is an input sample (the original file has one million lines):scaffold1_115,T,N,N,N,N,A,N,N,N,N,N,N,T,N,T,T,N,A,A,N,N,Ascaffold1_123,A,N,N,N,N,G,N,N,N,N,N,N,A,N,A,A,N,G,G,N,N,Gscaffold1_140,C,N,N,N,N,C,N,N,N,N,N,N,C,N,C,C,N,T,C,N,N,Cscaffold2_161,G,N,N,N,N,G,N,C,N,N,C,N,G,N,G,G,N,G,G,C,N,Gscaffold2_162,C,N,N,N,N,C,N,T,N,N,T,N,C,N,C,C,N,C,C,T,N,Cscaffold2_180,C,N,N,N,N,C,N,T,N,N,C,C,C,T,C,C,T,C,C,C,N,Cscaffold2_194,C,N,N,C,N,C,C,C,C,C,C,C,C,C,T,C,C,C,C,C,N,Cscaffold3_195,G,N,N,G,G,C,G,G,G,G,G,G,C,G,C,G,G,C,C,G,N,Cscaffold3_234,T,N,A,T,A,A,T,T,T,A,T,A,A,T,A,A,T,A,A,T,N,Ascaffold101_282,C,T,T,T,C,C,T,C,T,C,C,C,C,T,C,C,T,C,C,C,N,Cscaffold101_371,T,T,T,T,T,C,T,T,T,T,T,T,T,T,T,T,T,T,T,T,N,Cscaffold101_372,T,T,T,T,C,C,T,T,T,T,T,T,T,T,T,T,T,T,T,T,N,CThe lines are unique. I want lines specific to each scafold into a separate file, say all lines that start with scaffold1_ into a file named scaffold1.txt and so on until scaffold10156.txt which contains the lines starting with scaffold10156_
split a large file to new file with unique file names
text processing;sed;awk;grep;split
You should be able to use redirection with awkawk -F'_' '{print > $1.txt}' fileIf lines sharing the scaffoldn_ prefix are contiguous, you could do the following to avoid breaching open file handles limitawk -F'_' 'NR == 1 || $1 != prev{if (f) close(f);f=$1.txt; prev=$1}; {print > f};END{if (f) close(f)}' file