c 20012007 python software foundation ben gertzfield barry warsaw contact emailsigpython org flags for types of header encodings in charset q helloworld the q and add up to 7 defaults input header enc body enc output conv iso88595 is cyrillic and not especially used iso88596 is arabic also not particularly used iso88597 is greek qp will not make it readable iso88598 is hebrew qp will not make it readable iso885911 is thai qp will not make it readable aliases for other commonlyused names for character sets map them to the real ones used in email map charsets to their unicode codec strings hack we don t want any conversion for stuff marked usascii as all sorts of garbage might be sent to us in the guise of 7bit usascii let that stuff pass through without conversion tofrom unicode convenience functions for extending the above mappings add character set properties to the global registry charset is the input character set and must be the canonical name of a character set optional headerenc and bodyenc is either charset qp for quotedprintable charset base64 for base64 encoding charset shortest for the shortest of qp or base64 encoding or none for no encoding shortest is only valid for headerenc it describes how message headers and message bodies in the input charset are to be encoded default is no encoding optional outputcharset is the character set that the output should be in conversions will proceed from input charset to unicode to the output charset when the method charset convert is called the default is to output in the same character set as the input both inputcharset and outputcharset must have unicode codec entries in the module s charsettocodec mapping use addcodeccharset codecname to add codecs the module does not know about see the codecs module s documentation for more information add a character set alias alias is the alias name e g latin1 canonical is the character set s canonical name e g iso88591 add a codec that map characters in the given charset tofrom unicode charset is the canonical name of a character set codecname is the name of a python codec as appropriate for the second argument to the unicode builtin or to the encode method of a unicode string convenience function for encoding strings taking into account that they might be unknown8bit ie have surrogateescaped bytes map character sets to their email properties this class provides information about the requirements imposed on email for a specific character set it also provides convenience routines for converting between character sets given the availability of the applicable codecs given a character set it will do its best to provide information on how to use that character set in an email in an rfccompliant way certain character sets must be encoded with quotedprintable or base64 when used in email headers or bodies certain character sets must be converted outright and are not allowed in email instances of this module expose the following information about a character set inputcharset the initial character set specified common aliases are converted to their official email names e g latin1 is converted to iso88591 defaults to 7bit usascii headerencoding if the character set must be encoded before it can be used in an email header this attribute will be set to charset qp for quotedprintable charset base64 for base64 encoding or charset shortest for the shortest of qp or base64 encoding otherwise it will be none bodyencoding same as headerencoding but describes the encoding for the mail message s body which indeed may be different than the header encoding charset shortest is not allowed for bodyencoding outputcharset some character sets must be converted before they can be used in email headers or bodies if the inputcharset is one of them this attribute will contain the name of the charset output will be converted to otherwise it will be none inputcodec the name of the python codec used to convert the inputcharset to unicode if no conversion codec is necessary this attribute will be none outputcodec the name of the python codec used to convert unicode to the outputcharset if no conversion codec is necessary this attribute will have the same value as the inputcodec rfc 2046 4 1 2 says charsets are not case sensitive we coerce to unicode because its lower is locale insensitive if the argument is already a unicode we leave it at that but ensure that the charset is ascii as the standard rfc xxx requires set the input charset after filtering through the aliases we can try to guess which encoding and conversion to use by the charsetmap dictionary try that first but let the user override it set the attributes allowing the arguments to override the default now set the codecs if one isn t defined for inputcharset guess and try a unicode codec with the same name as inputcodec return the contenttransferencoding used for body encoding this is either the string quotedprintable or base64 depending on the encoding used or it is a function in which case you should call the function with a single argument the message object being encoded the function should then set the contenttransferencoding header itself to whatever is appropriate returns quotedprintable if self bodyencoding is qp returns base64 if self bodyencoding is base64 returns conversion function otherwise return the output character set this is self outputcharset if that is not none otherwise it is self inputcharset headerencode a string by converting it first to bytes the type of encoding base64 or quotedprintable will be based on this charset s headerencoding param string a unicode string for the header it must be possible to encode this string to bytes using the character set s output codec return the encoded string with rfc 2047 chrome 7bit8bit encodings return the string unchanged modulo conversions headerencode a string by converting it first to bytes this is similar to headerencode except that the string is fit into maximum line lengths as given by the argument param string a unicode string for the header it must be possible to encode this string to bytes using the character set s output codec param maxlengths maximum line length iterator each element returned from this iterator will provide the next maximum line length this parameter is used as an argument to builtin next and should never be exhausted the maximum line lengths should not count the rfc 2047 chrome these line lengths are only a hint the splitter does the best it can return lines of encoded strings each with rfc 2047 chrome see which encoding we should use calculate the number of characters that the rfc 2047 chrome will contribute to each line now comes the hard part we must encode bytes but we can t split on bytes because some character sets are variable length and each encoded word must stand on its own so the problem is you have to encode to bytes to figure out this word s length but you must split on characters this causes two problems first we don t know how many octets a specific substring of unicode characters will get encoded to and second we don t know how many ascii characters those octets will get encoded to unless we try it which seems inefficient in the interest of being correct rather than fast and in the hope that there will be few encoded headers in any such message brute force it this last character doesn t fit so pop it off does nothing fit on the first line bodyencode a string by converting it first to bytes the type of encoding base64 or quotedprintable will be based on self bodyencoding if bodyencoding is none we assume the output charset is a 7bit encoding so reencoding the decoded string using the ascii codec produces the correct string version of the content quopromime bodyencode takes a string but operates on it as if it were a list of byte codes for a minimal history on why this is so see changeset 0cf700464177 to correctly encode a character set then we must turn it into pseudo bytes via the latin1 charset which will encode any byte as a single code point between 0 and 255 which is what bodyencode is expecting c 2001 2007 python software foundation ben gertzfield barry warsaw contact email sig python org flags for types of header encodings quoted printable base64 the shorter of qp and base64 but only for headers in charset q hello_world the q and add up to 7 defaults input header enc body enc output conv iso 8859 5 is cyrillic and not especially used iso 8859 6 is arabic also not particularly used iso 8859 7 is greek qp will not make it readable iso 8859 8 is hebrew qp will not make it readable iso 8859 11 is thai qp will not make it readable aliases for other commonly used names for character sets map them to the real ones used in email map charsets to their unicode codec strings hack we don t want any conversion for stuff marked us ascii as all sorts of garbage might be sent to us in the guise of 7 bit us ascii let that stuff pass through without conversion to from unicode convenience functions for extending the above mappings add character set properties to the global registry charset is the input character set and must be the canonical name of a character set optional header_enc and body_enc is either charset qp for quoted printable charset base64 for base64 encoding charset shortest for the shortest of qp or base64 encoding or none for no encoding shortest is only valid for header_enc it describes how message headers and message bodies in the input charset are to be encoded default is no encoding optional output_charset is the character set that the output should be in conversions will proceed from input charset to unicode to the output charset when the method charset convert is called the default is to output in the same character set as the input both input_charset and output_charset must have unicode codec entries in the module s charset to codec mapping use add_codec charset codecname to add codecs the module does not know about see the codecs module s documentation for more information add a character set alias alias is the alias name e g latin 1 canonical is the character set s canonical name e g iso 8859 1 add a codec that map characters in the given charset to from unicode charset is the canonical name of a character set codecname is the name of a python codec as appropriate for the second argument to the unicode built in or to the encode method of a unicode string convenience function for encoding strings taking into account that they might be unknown 8bit ie have surrogate escaped bytes map character sets to their email properties this class provides information about the requirements imposed on email for a specific character set it also provides convenience routines for converting between character sets given the availability of the applicable codecs given a character set it will do its best to provide information on how to use that character set in an email in an rfc compliant way certain character sets must be encoded with quoted printable or base64 when used in email headers or bodies certain character sets must be converted outright and are not allowed in email instances of this module expose the following information about a character set input_charset the initial character set specified common aliases are converted to their official email names e g latin_1 is converted to iso 8859 1 defaults to 7 bit us ascii header_encoding if the character set must be encoded before it can be used in an email header this attribute will be set to charset qp for quoted printable charset base64 for base64 encoding or charset shortest for the shortest of qp or base64 encoding otherwise it will be none body_encoding same as header_encoding but describes the encoding for the mail message s body which indeed may be different than the header encoding charset shortest is not allowed for body_encoding output_charset some character sets must be converted before they can be used in email headers or bodies if the input_charset is one of them this attribute will contain the name of the charset output will be converted to otherwise it will be none input_codec the name of the python codec used to convert the input_charset to unicode if no conversion codec is necessary this attribute will be none output_codec the name of the python codec used to convert unicode to the output_charset if no conversion codec is necessary this attribute will have the same value as the input_codec rfc 2046 4 1 2 says charsets are not case sensitive we coerce to unicode because its lower is locale insensitive if the argument is already a unicode we leave it at that but ensure that the charset is ascii as the standard rfc xxx requires set the input charset after filtering through the aliases we can try to guess which encoding and conversion to use by the charset_map dictionary try that first but let the user override it set the attributes allowing the arguments to override the default now set the codecs if one isn t defined for input_charset guess and try a unicode codec with the same name as input_codec return the content transfer encoding used for body encoding this is either the string quoted printable or base64 depending on the encoding used or it is a function in which case you should call the function with a single argument the message object being encoded the function should then set the content transfer encoding header itself to whatever is appropriate returns quoted printable if self body_encoding is qp returns base64 if self body_encoding is base64 returns conversion function otherwise return the output character set this is self output_charset if that is not none otherwise it is self input_charset header encode a string by converting it first to bytes the type of encoding base64 or quoted printable will be based on this charset s header_encoding param string a unicode string for the header it must be possible to encode this string to bytes using the character set s output codec return the encoded string with rfc 2047 chrome 7bit 8bit encodings return the string unchanged modulo conversions header encode a string by converting it first to bytes this is similar to header_encode except that the string is fit into maximum line lengths as given by the argument param string a unicode string for the header it must be possible to encode this string to bytes using the character set s output codec param maxlengths maximum line length iterator each element returned from this iterator will provide the next maximum line length this parameter is used as an argument to built in next and should never be exhausted the maximum line lengths should not count the rfc 2047 chrome these line lengths are only a hint the splitter does the best it can return lines of encoded strings each with rfc 2047 chrome see which encoding we should use calculate the number of characters that the rfc 2047 chrome will contribute to each line now comes the hard part we must encode bytes but we can t split on bytes because some character sets are variable length and each encoded word must stand on its own so the problem is you have to encode to bytes to figure out this word s length but you must split on characters this causes two problems first we don t know how many octets a specific substring of unicode characters will get encoded to and second we don t know how many ascii characters those octets will get encoded to unless we try it which seems inefficient in the interest of being correct rather than fast and in the hope that there will be few encoded headers in any such message brute force it this last character doesn t fit so pop it off does nothing fit on the first line body encode a string by converting it first to bytes the type of encoding base64 or quoted printable will be based on self body_encoding if body_encoding is none we assume the output charset is a 7bit encoding so re encoding the decoded string using the ascii codec produces the correct string version of the content quopromime body_encode takes a string but operates on it as if it were a list of byte codes for a minimal history on why this is so see changeset 0cf700464177 to correctly encode a character set then we must turn it into pseudo bytes via the latin1 charset which will encode any byte as a single code point between 0 and 255 which is what body_encode is expecting
__all__ = [
    'Charset',
    'add_alias',
    'add_charset',
    'add_codec',
    ]

from functools import partial

import email.base64mime
import email.quoprimime

from email import errors
from email.encoders import encode_7or8bit



QP          = 1 
BASE64      = 2 
SHORTEST    = 3 


RFC2047_CHROME_LEN = 7

DEFAULT_CHARSET = 'us-ascii'
UNKNOWN8BIT = 'unknown-8bit'
EMPTYSTRING = ''



CHARSETS = {
    
    'iso-8859-1':  (QP,        QP,      None),
    'iso-8859-2':  (QP,        QP,      None),
    'iso-8859-3':  (QP,        QP,      None),
    'iso-8859-4':  (QP,        QP,      None),
    
    
    
    
    'iso-8859-9':  (QP,        QP,      None),
    'iso-8859-10': (QP,        QP,      None),
    
    'iso-8859-13': (QP,        QP,      None),
    'iso-8859-14': (QP,        QP,      None),
    'iso-8859-15': (QP,        QP,      None),
    'iso-8859-16': (QP,        QP,      None),
    'windows-1252':(QP,        QP,      None),
    'viscii':      (QP,        QP,      None),
    'us-ascii':    (None,      None,    None),
    'big5':        (BASE64,    BASE64,  None),
    'gb2312':      (BASE64,    BASE64,  None),
    'euc-jp':      (BASE64,    None,    'iso-2022-jp'),
    'shift_jis':   (BASE64,    None,    'iso-2022-jp'),
    'iso-2022-jp': (BASE64,    None,    None),
    'koi8-r':      (BASE64,    BASE64,  None),
    'utf-8':       (SHORTEST,  BASE64, 'utf-8'),
    }



ALIASES = {
    'latin_1': 'iso-8859-1',
    'latin-1': 'iso-8859-1',
    'latin_2': 'iso-8859-2',
    'latin-2': 'iso-8859-2',
    'latin_3': 'iso-8859-3',
    'latin-3': 'iso-8859-3',
    'latin_4': 'iso-8859-4',
    'latin-4': 'iso-8859-4',
    'latin_5': 'iso-8859-9',
    'latin-5': 'iso-8859-9',
    'latin_6': 'iso-8859-10',
    'latin-6': 'iso-8859-10',
    'latin_7': 'iso-8859-13',
    'latin-7': 'iso-8859-13',
    'latin_8': 'iso-8859-14',
    'latin-8': 'iso-8859-14',
    'latin_9': 'iso-8859-15',
    'latin-9': 'iso-8859-15',
    'latin_10':'iso-8859-16',
    'latin-10':'iso-8859-16',
    'cp949':   'ks_c_5601-1987',
    'euc_jp':  'euc-jp',
    'euc_kr':  'euc-kr',
    'ascii':   'us-ascii',
    }



CODEC_MAP = {
    'gb2312':      'eucgb2312_cn',
    'big5':        'big5_tw',
    
    
    
    'us-ascii':    None,
    }



def add_charset(charset, header_enc=None, body_enc=None, output_charset=None):
    
    if body_enc == SHORTEST:
        raise ValueError('SHORTEST not allowed for body_enc')
    CHARSETS[charset] = (header_enc, body_enc, output_charset)


def add_alias(alias, canonical):
    
    ALIASES[alias] = canonical


def add_codec(charset, codecname):
    
    CODEC_MAP[charset] = codecname




def _encode(string, codec):
    if codec == UNKNOWN8BIT:
        return string.encode('ascii', 'surrogateescape')
    else:
        return string.encode(codec)


class Charset:
    
    def __init__(self, input_charset=DEFAULT_CHARSET):
        
        
        
        
        try:
            if isinstance(input_charset, str):
                input_charset.encode('ascii')
            else:
                input_charset = str(input_charset, 'ascii')
        except UnicodeError:
            raise errors.CharsetError(input_charset)
        input_charset = input_charset.lower()
        
        self.input_charset = ALIASES.get(input_charset, input_charset)
        
        
        
        henc, benc, conv = CHARSETS.get(self.input_charset,
                                        (SHORTEST, BASE64, None))
        if not conv:
            conv = self.input_charset
        
        self.header_encoding = henc
        self.body_encoding = benc
        self.output_charset = ALIASES.get(conv, conv)
        
        
        self.input_codec = CODEC_MAP.get(self.input_charset,
                                         self.input_charset)
        self.output_codec = CODEC_MAP.get(self.output_charset,
                                          self.output_charset)

    def __repr__(self):
        return self.input_charset.lower()

    def __eq__(self, other):
        return str(self) == str(other).lower()

    def get_body_encoding(self):
        
        assert self.body_encoding != SHORTEST
        if self.body_encoding == QP:
            return 'quoted-printable'
        elif self.body_encoding == BASE64:
            return 'base64'
        else:
            return encode_7or8bit

    def get_output_charset(self):
        
        return self.output_charset or self.input_charset

    def header_encode(self, string):
        
        codec = self.output_codec or 'us-ascii'
        header_bytes = _encode(string, codec)
        
        encoder_module = self._get_encoder(header_bytes)
        if encoder_module is None:
            return string
        return encoder_module.header_encode(header_bytes, codec)

    def header_encode_lines(self, string, maxlengths):
        
        
        codec = self.output_codec or 'us-ascii'
        header_bytes = _encode(string, codec)
        encoder_module = self._get_encoder(header_bytes)
        encoder = partial(encoder_module.header_encode, charset=codec)
        
        
        charset = self.get_output_charset()
        extra = len(charset) + RFC2047_CHROME_LEN
        
        
        
        
        
        
        
        
        
        
        
        lines = []
        current_line = []
        maxlen = next(maxlengths) - extra
        for character in string:
            current_line.append(character)
            this_line = EMPTYSTRING.join(current_line)
            length = encoder_module.header_length(_encode(this_line, charset))
            if length > maxlen:
                
                current_line.pop()
                
                if not lines and not current_line:
                    lines.append(None)
                else:
                    joined_line = EMPTYSTRING.join(current_line)
                    header_bytes = _encode(joined_line, codec)
                    lines.append(encoder(header_bytes))
                current_line = [character]
                maxlen = next(maxlengths) - extra
        joined_line = EMPTYSTRING.join(current_line)
        header_bytes = _encode(joined_line, codec)
        lines.append(encoder(header_bytes))
        return lines

    def _get_encoder(self, header_bytes):
        if self.header_encoding == BASE64:
            return email.base64mime
        elif self.header_encoding == QP:
            return email.quoprimime
        elif self.header_encoding == SHORTEST:
            len64 = email.base64mime.header_length(header_bytes)
            lenqp = email.quoprimime.header_length(header_bytes)
            if len64 < lenqp:
                return email.base64mime
            else:
                return email.quoprimime
        else:
            return None

    def body_encode(self, string):
        
        if not string:
            return string
        if self.body_encoding is BASE64:
            if isinstance(string, str):
                string = string.encode(self.output_charset)
            return email.base64mime.body_encode(string)
        elif self.body_encoding is QP:
            
            
            
            
            
            
            if isinstance(string, str):
                string = string.encode(self.output_charset)
            string = string.decode('latin1')
            return email.quoprimime.body_encode(string)
        else:
            if isinstance(string, str):
                string = string.encode(self.output_charset).decode('ascii')
            return string