petr cerny dptntnsusecz greg freemyer wrote feb 19 2008 801 petr cerny wrote greg freemyer wrote see im bit confused worked much unicode linux unicode file looks like normal ascii null every byte think thats normal limited exposure unicode status im office recreated 17 million line file last night pure unicode good cant test couple hours claim use grep search unicode file ascii text pattern likely covert grep see yesterday looked grep switch would tell file searched unicode didnt see one guess could automatic actually attempt feasible tool convert unicode file ascii file would great text interested convert ascii obvious way need mappings basically need get rid nulls iconv friend basically would need something like iconv f some_unicode_encoding ascii unicode_file ascii_file also choke nonascii characters unless make translate unknown characters translit appended output encoding probably using ucs2le file nulls even positions file ucs2be null odd positions file basic overview unicode standard encoding national characters multiple bytes several encodings unicode exist 1 utf8 encodes characters 14 bytes eg takes 1 byte chinese characters may need lets say 3 bytes 2 utf16 uses 2 4 bytes 3 ucs2le ucs2be use 2 bytes per character ascii characters one bytes always null depending whether big endian little endian variation 4 ucs4utf32 differ slighly use 4 bytes per character see eg httpenwikipediaorgwikiunicode best regards petr unsubscribe email assipqffnmekprykassopensuseorg additional commands email qybuovjskcmhopensuseorg thu 07 aug 2008 082533 0100 opensuse unicode v ascii