routines for manipulating rfc2047 encoded words this is currently a packageprivate api but will be considered for promotion to a public api if there is demand an ecoded word looks like this charsetlang cte encodedstring for more information about charset see the charset module here it is one of the preferred mime charset names hopefully you never know when parsing cte content transfer encoding is either q or b ignoring case in theory other letters could be used for other encodings but in practice this almost never happens there could be a public api for adding entries to the cte tables but yagni for now q is quoted printable b is base64 the meaning of encodedstring should be obvious lang is optional as indicated by the brackets they are not part of the syntax but is almost never encountered in practice the general interface for a cte decoder is that it takes the encodedstring as its argument and returns a tuple ctedecodedstring defects the ctedecodedstring is the original binary that was encoded using the specified cte defects is a list of messagedefect instances indicating any problems encountered during conversion charset and lang are the corresponding strings extracted from the ew case preserved the general interface for a cte encoder is that it takes a binary sequence as input and returns the cteencodedstring which is an asciionly string each decoder must also supply a length function that takes the binary sequence as its argument and returns the length of the resulting encoded string the main api functions for the module are decode which calls the decoder referenced by the cte specifier and encode which adds the appropriate rfc 2047 chrome to the encoded string and can optionally automatically select the shortest possible encoding see their docstrings below for details quoted printable regex based decoder dict mapping bytes to their encoded form in headers spaces are mapped to base64 first try encoding with validatetrue fixing the padding if needed this will succeed only if encoded includes no invalid characters since we had correct padding this is likely an invalid char error the nonalphabet characters are ignored as far as padding goes but we don t know how many there are so try without adding padding to see if it works add as much padding as could possibly be necessary extra padding is ignored this only happens when the encoded string s length is 1 more than a multiple of 4 which is invalid bpo27397 just return the encoded string since there s no way to decode 4 bytes out for each 3 bytes or nonzero fraction thereof in decode encoded word and return string charset lang defects tuple an rfc 20472243 encoded word has the form charsetlang cte encodedstring where lang may be omitted but the other parts may not be this function expects exactly such a string that is it does not check the syntax and may raise errors if the string is not well formed and returns the encodedstring decoded first from its content transfer encoding and then from the resulting bytes into unicode using the specified charset if the ctedecoded string does not successfully decode using the specified character set a defect is added to the defects list and the unknown octets are replaced by the unicode unknown character ufdff the specified charset and language are returned the default for language which is rarely if ever encountered is the empty string recover the original bytes and do cte decoding turn the cte decoded bytes into unicode encode string using the cte encoding that produces the shorter result produces an rfc 20472243 encoded word of the form charsetlang cte encodedstring where lang is omitted unless the lang parameter is given a value optional argument charset defaults to utf8 specifies the charset to use to encode the string to binary before cte encoding it optional argument encoding is the cte specifier for the encoding that should be used q or b if it is none the default the encoding which produces the shortest encoded sequence is used except that q is preferred if it is up to five characters longer optional argument lang default gives the rfc 2243 language string to specify in the encoded word bias toward q 5 is arbitrary an ecoded word looks like this charset lang cte encoded_string for more information about charset see the charset module here it is one of the preferred mime charset names hopefully you never know when parsing cte content transfer encoding is either q or b ignoring case in theory other letters could be used for other encodings but in practice this almost never happens there could be a public api for adding entries to the cte tables but yagni for now q is quoted printable b is base64 the meaning of encoded_string should be obvious lang is optional as indicated by the brackets they are not part of the syntax but is almost never encountered in practice the general interface for a cte decoder is that it takes the encoded_string as its argument and returns a tuple cte_decoded_string defects the cte_decoded_string is the original binary that was encoded using the specified cte defects is a list of messagedefect instances indicating any problems encountered during conversion charset and lang are the corresponding strings extracted from the ew case preserved the general interface for a cte encoder is that it takes a binary sequence as input and returns the cte_encoded_string which is an ascii only string each decoder must also supply a length function that takes the binary sequence as its argument and returns the length of the resulting encoded string the main api functions for the module are decode which calls the decoder referenced by the cte specifier and encode which adds the appropriate rfc 2047 chrome to the encoded string and can optionally automatically select the shortest possible encoding see their docstrings below for details quoted printable regex based decoder dict mapping bytes to their encoded form in headers spaces are mapped to _ base64 first try encoding with validate true fixing the padding if needed this will succeed only if encoded includes no invalid characters since we had correct padding this is likely an invalid char error the non alphabet characters are ignored as far as padding goes but we don t know how many there are so try without adding padding to see if it works add as much padding as could possibly be necessary extra padding is ignored this only happens when the encoded string s length is 1 more than a multiple of 4 which is invalid bpo 27397 just return the encoded string since there s no way to decode 4 bytes out for each 3 bytes or nonzero fraction thereof in decode encoded word and return string charset lang defects tuple an rfc 2047 2243 encoded word has the form charset lang cte encoded_string where lang may be omitted but the other parts may not be this function expects exactly such a string that is it does not check the syntax and may raise errors if the string is not well formed and returns the encoded_string decoded first from its content transfer encoding and then from the resulting bytes into unicode using the specified charset if the cte decoded string does not successfully decode using the specified character set a defect is added to the defects list and the unknown octets are replaced by the unicode unknown character ufdff the specified charset and language are returned the default for language which is rarely if ever encountered is the empty string recover the original bytes and do cte decoding turn the cte decoded bytes into unicode encode string using the cte encoding that produces the shorter result produces an rfc 2047 2243 encoded word of the form charset lang cte encoded_string where lang is omitted unless the lang parameter is given a value optional argument charset defaults to utf 8 specifies the charset to use to encode the string to binary before cte encoding it optional argument encoding is the cte specifier for the encoding that should be used q or b if it is none the default the encoding which produces the shortest encoded sequence is used except that q is preferred if it is up to five characters longer optional argument lang default gives the rfc 2243 language string to specify in the encoded word bias toward q 5 is arbitrary
import re
import base64
import binascii
import functools
from string import ascii_letters, digits
from email import errors

__all__ = ['decode_q',
           'encode_q',
           'decode_b',
           'encode_b',
           'len_q',
           'len_b',
           'decode',
           'encode',
           ]






_q_byte_subber = functools.partial(re.compile(br'=([a-fA-F0-9]{2})').sub,
        lambda m: bytes.fromhex(m.group(1).decode()))

def decode_q(encoded):
    encoded = encoded.replace(b'_', b' ')
    return _q_byte_subber(encoded), []



class _QByteMap(dict):

    safe = b'-!*+/' + ascii_letters.encode('ascii') + digits.encode('ascii')

    def __missing__(self, key):
        if key in self.safe:
            self[key] = chr(key)
        else:
            self[key] = "={:02X}".format(key)
        return self[key]

_q_byte_map = _QByteMap()


_q_byte_map[ord(' ')] = '_'

def encode_q(bstring):
    return ''.join(_q_byte_map[x] for x in bstring)

def len_q(bstring):
    return sum(len(_q_byte_map[x]) for x in bstring)






def decode_b(encoded):
    
    
    pad_err = len(encoded) % 4
    missing_padding = b'==='[:4-pad_err] if pad_err else b''
    try:
        return (
            base64.b64decode(encoded + missing_padding, validate=True),
            [errors.InvalidBase64PaddingDefect()] if pad_err else [],
        )
    except binascii.Error:
        
        
        
        
        
        try:
            return (
                base64.b64decode(encoded, validate=False),
                [errors.InvalidBase64CharactersDefect()],
            )
        except binascii.Error:
            
            
            try:
                return (
                    base64.b64decode(encoded + b'==', validate=False),
                    [errors.InvalidBase64CharactersDefect(),
                     errors.InvalidBase64PaddingDefect()],
                )
            except binascii.Error:
                
                
                
                
                
                return encoded, [errors.InvalidBase64LengthDefect()]

def encode_b(bstring):
    return base64.b64encode(bstring).decode('ascii')

def len_b(bstring):
    groups_of_3, leftover = divmod(len(bstring), 3)
    
    return groups_of_3 * 4 + (4 if leftover else 0)


_cte_decoders = {
    'q': decode_q,
    'b': decode_b,
    }

def decode(ew):
    
    _, charset, cte, cte_string, _ = ew.split('?')
    charset, _, lang = charset.partition('*')
    cte = cte.lower()
    
    bstring = cte_string.encode('ascii', 'surrogateescape')
    bstring, defects = _cte_decoders[cte](bstring)
    
    try:
        string = bstring.decode(charset)
    except UnicodeDecodeError:
        defects.append(errors.UndecodableBytesDefect("Encoded word "
            f"contains bytes not decodable using {charset!r} charset"))
        string = bstring.decode(charset, 'surrogateescape')
    except (LookupError, UnicodeEncodeError):
        string = bstring.decode('ascii', 'surrogateescape')
        if charset.lower() != 'unknown-8bit':
            defects.append(errors.CharsetError(f"Unknown charset {charset!r} "
                f"in encoded word; decoded as unknown bytes"))
    return string, charset, lang, defects


_cte_encoders = {
    'q': encode_q,
    'b': encode_b,
    }

_cte_encode_length = {
    'q': len_q,
    'b': len_b,
    }

def encode(string, charset='utf-8', encoding=None, lang=''):
    
    if charset == 'unknown-8bit':
        bstring = string.encode('ascii', 'surrogateescape')
    else:
        bstring = string.encode(charset)
    if encoding is None:
        qlen = _cte_encode_length['q'](bstring)
        blen = _cte_encode_length['b'](bstring)
        
        encoding = 'q' if qlen - blen < 5 else 'b'
    encoded = _cte_encoders[encoding](bstring)
    if lang:
        lang = '*' + lang
    return "=?{}{}?{}?{}?=".format(charset, lang, encoding, encoded)