I have a vertical file where each word (token) is on the separate line in 4 columns. There are also metastructures <doc>, <s>, ...The documents looks as follows:<doc name=sth url=http><p><s>Here   here   k1gInSc1   hereis   be   k1gMnPc2   bea  a   k2eAgMnPc1d1   asentence   sentence   k1gMnPc1   sentence<g/>.       .       kIx.</s></p></doc>the problem is that sometimes there is wrong encoding with characters as  or  in the first column, e.g.<doc name=sth url=http><p><s>Here   here   k1gInSc1   hereis   be   k1gMnPc2   be     k?   sentence   sentence   k1gMnPc1   sentence<g/>.       .       kIx.</s></p></doc>I would need to find these characters and delete the whole document structure. So, if I find  on a line, I need to delete the whole content between <doc...> all lines </doc>.My file has a billion lines and ca a few thousand lines contain wrong encoded characters.I used grep to finding bad characters:xzcat file.vert.xz | grep -i \|\|\|\| > file_bad_characters.txtHow can I detect these characters and delete not only the line but the whole text between <doc>structures.
find a specific string and delete the whole structure
text processing;xml