c 20012010 python software foundation barry warsaw contact emailsigpython org classes to generate plain text from a message object tree all generator decodedgenerator bytesgenerator import re import sys import time import random from copy import deepcopy from io import stringio bytesio from email utils import hassurrogates underscore nl n xxx no longer used by the code below nlcre re compiler rnrn fcre re compiler from re multiline class generator public interface def initself outfp manglefromnone maxheaderlennone policynone if manglefrom is none manglefrom true if policy is none else policy manglefrom self fp outfp self manglefrom manglefrom self maxheaderlen maxheaderlen self policy policy def writeself s just delegate to the file object self fp writes def flattenself msg unixfromfalse linesepnone rprint the message object tree rooted at msg to the output file specified when the generator instance was created unixfrom is a flag that forces the printing of a unix from delimiter before the first object in the message tree if the original message has no from delimiter a standard one is crafted by default this is false to inhibit the printing of any from delimiter note that for subobjects no from line is printed linesep specifies the characters used to indicate a new line in the output the default value is determined by the policy specified when the generator instance was created or if none was specified from the policy associated with the msg we use the xxx constants for operating on data that comes directly from the msg and encodedxxx constants for operating on data that has already been converted to bytes in the bytesgenerator and inserted into a temporary buffer because we use clone below when we recursively process message subparts and because clone uses the computed policy not none submessages will automatically get set to the computed policy when they are processed by this code clone this generator with the exact same options return self classfp self manglefrom none use policy setting which we ve adjusted policyself policy protected interface undocumented note that we use self write when what we are writing is coming from the source and self fp write when what we are writing is coming from a buffer because the bytes subclass has already had a chance to transform the data in its write method in that case this is an entirely pragmatic split determined by experiment we could be more general by always using write and having the bytes subclass write method detect when it has already transformed the input but since this whole thing is a hack anyway this seems good enough def newbufferself bytesgenerator overrides this to return bytesio return stringio def encodeself s bytesgenerator overrides this to encode strings to bytes return s def writelinesself lines we have to transform the line endings if not lines return lines nlcre splitlines for line in lines 1 self writeline self writeself nl if lines1 self writelines1 xxx logic tells me this else should be needed but the tests fail with it and pass without it nlcre split ends with a blank element if and only if there was a trailing newline else self writeself nl def writeself msg we can t write the headers yet because of the following scenario say a multipart message includes the boundary string somewhere in its body we d have to calculate the new boundary before we write the headers so that we can write the correct contenttype parameter the way we do this so as to make the handle methods simpler is to cache any subpart writes into a buffer then we write the headers and the buffer contents that way subpart handlers can do the right thing and can still modify the contenttype header if necessary oldfp self fp try self mungecte none self fp sfp self newbuffer self dispatchmsg finally self fp oldfp mungecte self mungecte del self mungecte if we munged the cte copy the message again and refix the cte if mungecte msg deepcopymsg preserve the header order if the cte header already exists if msg get contenttransferencoding is none msg contenttransferencoding mungecte0 else msg replaceheader contenttransferencoding mungecte0 msg replaceheader contenttype mungecte1 write the headers first we see if the message object wants to handle that itself if not we ll do it generically meth getattrmsg writeheaders none if meth is none self writeheadersmsg else methself self fp writesfp getvalue def dispatchself msg get the contenttype for the message then try to dispatch to self handlemaintypesubtype if there s no handler for the full mime type then dispatch to self handlemaintype if that s missing too then dispatch to self writebody main msg getcontentmaintype sub msg getcontentsubtype specific underscore joinmain sub replace meth getattrself handle specific none if meth is none generic main replace meth getattrself handle generic none if meth is none meth self writebody methmsg default handlers def writeheadersself msg for h v in msg rawitems self writeself policy foldh v a blank line always separates headers from body self writeself nl handlers for writing types and subtypes def handletextself msg payload msg getpayload if payload is none return if not isinstancepayload str raise typeerror string payload expected s typepayload if hassurrogatesmsg payload charset msg getparam charset if charset is not none xxx this copy stuff is an ugly hack to avoid modifying the existing message msg deepcopymsg del msg contenttransferencoding msg setpayloadpayload charset payload msg getpayload self mungecte msg contenttransferencoding msg contenttype if self manglefrom payload fcre sub from payload self writelinespayload default body handler writebody handletext def handlemultipartself msg the trick here is to write out each part separately merge them all together and then make sure that the boundary we ve chosen isn t present in the payload msgtexts subparts msg getpayload if subparts is none subparts elif isinstancesubparts str e g a nonstrict parse of a message with no starting boundary self writesubparts return elif not isinstancesubparts list scalar payload subparts subparts for part in subparts s self newbuffer g self clones g flattenpart unixfromfalse linesepself nl msgtexts appends getvalue baw what about boundaries that are wrapped in doublequotes boundary msg getboundary if not boundary create a boundary that doesn t appear in any of the message texts alltext self encodednl joinmsgtexts boundary self makeboundaryalltext msg setboundaryboundary if there s a preamble write it out with a trailing crlf if msg preamble is not none if self manglefrom preamble fcre sub from msg preamble else preamble msg preamble self writelinespreamble self writeself nl dashboundary transportpadding crlf self write boundary self nl bodypart if msgtexts self fp writemsgtexts pop0 encapsulation delimiter transportpadding crlf bodypart for bodypart in msgtexts delimiter transportpadding crlf self writeself nl boundary self nl bodypart self fp writebodypart closedelimiter transportpadding self writeself nl boundary self nl if msg epilogue is not none if self manglefrom epilogue fcre sub from msg epilogue else epilogue msg epilogue self writelinesepilogue def handlemultipartsignedself msg the contents of signed parts has to stay unmodified in order to keep the signature intact per rfc1847 2 1 so we disable header wrapping rdm this isn t enough to completely preserve the part but it helps p self policy self policy p clonemaxlinelength0 try self handlemultipartmsg finally self policy p def handlemessagedeliverystatusself msg we can t just write the headers directly to self s file object because this will leave an extra newline between the last header block and the boundary sigh blocks for part in msg getpayload s self newbuffer g self clones g flattenpart unixfromfalse linesepself nl text s getvalue lines text splitself encodednl strip off the unnecessary trailing empty line if lines and lines1 self encodedempty blocks appendself encodednl joinlines 1 else blocks appendtext now join all the blocks with an empty line this has the lovely effect of separating each block with an empty line but not adding an extra one after the last one self fp writeself encodednl joinblocks def handlemessageself msg s self newbuffer g self clones the payload of a messagerfc822 part should be a multipart sequence of length 1 the zeroth element of the list should be the message object for the subpart extract that object stringify it and write it out except it turns out when it s a string instead which happens when and only when headerparser is used on a message of mime type messagerfc822 such messages are generated by for example groupwise when forwarding unadorned messages issue 7970 so in that case we just emit the string body payload msg payload if isinstancepayload list g flattenmsg getpayload0 unixfromfalse linesepself nl payload s getvalue else payload self encodepayload self fp writepayload this used to be a module level function we use a classmethod for this and compilere so we can continue to provide the module level function for backward compatibility by doing makeboundary generator makeboundary at the end of the module it is internal so we could drop that classmethod def makeboundarycls textnone craft a random boundary if text is given ensure that the chosen boundary doesn t appear in the text token random randrangesys maxsize boundary 15 fmt token if text is none return boundary b boundary counter 0 while true cre cls compilere re escapeb re multiline if not cre searchtext break b boundary strcounter counter 1 return b classmethod def compilerecls s flags return re compiles flags class bytesgeneratorgenerator def writeself s self fp writes encode ascii surrogateescape def newbufferself return bytesio def encodeself s return s encode ascii def writeheadersself msg this is almost the same as the string version except for handling strings with 8bit bytes for h v in msg rawitems self fp writeself policy foldbinaryh v a blank line always separates headers from body self writeself nl def handletextself msg if the string has surrogates the original source was bytes so just write it back out if msg payload is none return if hassurrogatesmsg payload and not self policy ctetype 7bit if self manglefrom msg payload fcre subfrom msg payload self writelinesmsg payload else superbytesgenerator self handletextmsg default body handler writebody handletext classmethod def compilerecls s flags return re compiles encode ascii flags fmt nontext types part of message omitted filename filenames class decodedgeneratorgenerator def initself outfp manglefromnone maxheaderlennone fmtnone policynone generator initself outfp manglefrom maxheaderlen policypolicy if fmt is none self fmt fmt else self fmt fmt def dispatchself msg for part in msg walk maintype part getcontentmaintype if maintype text printpart getpayloaddecodefalse fileself elif maintype multipart just skip this pass else printself fmt type part getcontenttype maintype part getcontentmaintype subtype part getcontentsubtype filename part getfilename no filename description part get contentdescription no description encoding part get contenttransferencoding no encoding fileself helper used by generator makeboundary width lenreprsys maxsize1 fmt 0dd width backward compatibility makeboundary generator makeboundary c 2001 2010 python software foundation barry warsaw contact email sig python org classes to generate plain text from a message object tree xxx no longer used by the code below generates output from a message object tree this basic generator writes the message to the given file object as plain text public interface create the generator for message flattening outfp is the output file like object for writing the message to it must have a write method optional mangle_from_ is a flag that when true the default if policy is not set escapes from_ lines in the body of the message by putting a in front of them optional maxheaderlen specifies the longest length for a non continued header when a header line is longer in characters with tabs expanded to 8 spaces than maxheaderlen the header will split as defined in the header class set maxheaderlen to zero to disable header wrapping the default is 78 as recommended but not required by rfc 2822 the policy keyword specifies a policy object that controls a number of aspects of the generator s operation if no policy is specified the policy associated with the message object passed to the flatten method is used just delegate to the file object print the message object tree rooted at msg to the output file specified when the generator instance was created unixfrom is a flag that forces the printing of a unix from_ delimiter before the first object in the message tree if the original message has no from_ delimiter a standard one is crafted by default this is false to inhibit the printing of any from_ delimiter note that for subobjects no from_ line is printed linesep specifies the characters used to indicate a new line in the output the default value is determined by the policy specified when the generator instance was created or if none was specified from the policy associated with the msg we use the _xxx constants for operating on data that comes directly from the msg and _encoded_xxx constants for operating on data that has already been converted to bytes in the bytesgenerator and inserted into a temporary buffer because we use clone below when we recursively process message subparts and because clone uses the computed policy not none submessages will automatically get set to the computed policy when they are processed by this code clone this generator with the exact same options use policy setting which we ve adjusted protected interface undocumented note that we use self write when what we are writing is coming from the source and self _fp write when what we are writing is coming from a buffer because the bytes subclass has already had a chance to transform the data in its write method in that case this is an entirely pragmatic split determined by experiment we could be more general by always using write and having the bytes subclass write method detect when it has already transformed the input but since this whole thing is a hack anyway this seems good enough bytesgenerator overrides this to return bytesio bytesgenerator overrides this to encode strings to bytes we have to transform the line endings xxx logic tells me this else should be needed but the tests fail with it and pass without it nlcre split ends with a blank element if and only if there was a trailing newline else self write self _nl we can t write the headers yet because of the following scenario say a multipart message includes the boundary string somewhere in its body we d have to calculate the new boundary before we write the headers so that we can write the correct content type parameter the way we do this so as to make the _handle_ methods simpler is to cache any subpart writes into a buffer then we write the headers and the buffer contents that way subpart handlers can do the right thing and can still modify the content type header if necessary if we munged the cte copy the message again and re fix the cte preserve the header order if the cte header already exists write the headers first we see if the message object wants to handle that itself if not we ll do it generically get the content type for the message then try to dispatch to self _handle_ maintype _ subtype if there s no handler for the full mime type then dispatch to self _handle_ maintype if that s missing too then dispatch to self _writebody default handlers a blank line always separates headers from body handlers for writing types and subtypes xxx this copy stuff is an ugly hack to avoid modifying the existing message default body handler the trick here is to write out each part separately merge them all together and then make sure that the boundary we ve chosen isn t present in the payload e g a non strict parse of a message with no starting boundary scalar payload baw what about boundaries that are wrapped in double quotes create a boundary that doesn t appear in any of the message texts if there s a preamble write it out with a trailing crlf dash boundary transport padding crlf body part encapsulation delimiter transport padding crlf body part delimiter transport padding crlf body part close delimiter transport padding the contents of signed parts has to stay unmodified in order to keep the signature intact per rfc1847 2 1 so we disable header wrapping rdm this isn t enough to completely preserve the part but it helps we can t just write the headers directly to self s file object because this will leave an extra newline between the last header block and the boundary sigh strip off the unnecessary trailing empty line now join all the blocks with an empty line this has the lovely effect of separating each block with an empty line but not adding an extra one after the last one the payload of a message rfc822 part should be a multipart sequence of length 1 the zeroth element of the list should be the message object for the subpart extract that object stringify it and write it out except it turns out when it s a string instead which happens when and only when headerparser is used on a message of mime type message rfc822 such messages are generated by for example groupwise when forwarding unadorned messages issue 7970 so in that case we just emit the string body this used to be a module level function we use a classmethod for this and _compile_re so we can continue to provide the module level function for backward compatibility by doing _make_boundary generator _make_boundary at the end of the module it is internal so we could drop that craft a random boundary if text is given ensure that the chosen boundary doesn t appear in the text generates a bytes version of a message object tree functionally identical to the base generator except that the output is bytes and not string when surrogates were used in the input to encode bytes these are decoded back to bytes for output if the policy has cte_type set to 7bit then the message is transformed such that the non ascii bytes are properly content transfer encoded using the charset unknown 8bit the outfp object must accept bytes in its write method this is almost the same as the string version except for handling strings with 8bit bytes a blank line always separates headers from body if the string has surrogates the original source was bytes so just write it back out default body handler generates a text representation of a message like the generator base class except that non text parts are substituted with a format string representing the part like generator __init__ except that an additional optional argument is allowed walks through all subparts of a message if the subpart is of main type text then it prints the decoded payload of the subpart otherwise fmt is a format string that is used instead of the message payload fmt is expanded with the following keywords in keyword s format type full mime type of the non text part maintype main mime type of the non text part subtype sub mime type of the non text part filename filename of the non text part description description associated with the non text part encoding content transfer encoding of the non text part the default value for fmt is none meaning non text type s part of message omitted filename filename s just skip this helper used by generator _make_boundary backward compatibility
__all__ = ['Generator', 'DecodedGenerator', 'BytesGenerator']

import re
import sys
import time
import random

from copy import deepcopy
from io import StringIO, BytesIO
from email.utils import _has_surrogates

UNDERSCORE = '_'
NL = '\n'  

NLCRE = re.compile(r'\r\n|\r|\n')
fcre = re.compile(r'^From ', re.MULTILINE)


class Generator:
    
    
    
    

    def __init__(self, outfp, mangle_from_=None, maxheaderlen=None, *,
                 policy=None):
        

        if mangle_from_ is None:
            mangle_from_ = True if policy is None else policy.mangle_from_
        self._fp = outfp
        self._mangle_from_ = mangle_from_
        self.maxheaderlen = maxheaderlen
        self.policy = policy

    def write(self, s):
        
        self._fp.write(s)

    def flatten(self, msg, unixfrom=False, linesep=None):
        r
        
        
        
        
        policy = msg.policy if self.policy is None else self.policy
        if linesep is not None:
            policy = policy.clone(linesep=linesep)
        if self.maxheaderlen is not None:
            policy = policy.clone(max_line_length=self.maxheaderlen)
        self._NL = policy.linesep
        self._encoded_NL = self._encode(self._NL)
        self._EMPTY = ''
        self._encoded_EMPTY = self._encode(self._EMPTY)
        
        
        
        
        old_gen_policy = self.policy
        old_msg_policy = msg.policy
        try:
            self.policy = policy
            msg.policy = policy
            if unixfrom:
                ufrom = msg.get_unixfrom()
                if not ufrom:
                    ufrom = 'From nobody ' + time.ctime(time.time())
                self.write(ufrom + self._NL)
            self._write(msg)
        finally:
            self.policy = old_gen_policy
            msg.policy = old_msg_policy

    def clone(self, fp):
        
        return self.__class__(fp,
                              self._mangle_from_,
                              None, 
                              policy=self.policy)

    
    
    

    
    
    
    
    
    
    
    

    def _new_buffer(self):
        
        return StringIO()

    def _encode(self, s):
        
        return s

    def _write_lines(self, lines):
        
        if not lines:
            return
        lines = NLCRE.split(lines)
        for line in lines[:-1]:
            self.write(line)
            self.write(self._NL)
        if lines[-1]:
            self.write(lines[-1])
        
        
        
        
        

    def _write(self, msg):
        
        
        
        
        
        
        
        
        
        
        
        oldfp = self._fp
        try:
            self._munge_cte = None
            self._fp = sfp = self._new_buffer()
            self._dispatch(msg)
        finally:
            self._fp = oldfp
            munge_cte = self._munge_cte
            del self._munge_cte
        
        if munge_cte:
            msg = deepcopy(msg)
            
            if msg.get('content-transfer-encoding') is None:
                msg['Content-Transfer-Encoding'] = munge_cte[0]
            else:
                msg.replace_header('content-transfer-encoding', munge_cte[0])
            msg.replace_header('content-type', munge_cte[1])
        
        
        meth = getattr(msg, '_write_headers', None)
        if meth is None:
            self._write_headers(msg)
        else:
            meth(self)
        self._fp.write(sfp.getvalue())

    def _dispatch(self, msg):
        
        
        
        
        main = msg.get_content_maintype()
        sub = msg.get_content_subtype()
        specific = UNDERSCORE.join((main, sub)).replace('-', '_')
        meth = getattr(self, '_handle_' + specific, None)
        if meth is None:
            generic = main.replace('-', '_')
            meth = getattr(self, '_handle_' + generic, None)
            if meth is None:
                meth = self._writeBody
        meth(msg)

    
    
    

    def _write_headers(self, msg):
        for h, v in msg.raw_items():
            self.write(self.policy.fold(h, v))
        
        self.write(self._NL)

    
    
    

    def _handle_text(self, msg):
        payload = msg.get_payload()
        if payload is None:
            return
        if not isinstance(payload, str):
            raise TypeError('string payload expected: %s' % type(payload))
        if _has_surrogates(msg._payload):
            charset = msg.get_param('charset')
            if charset is not None:
                
                
                msg = deepcopy(msg)
                del msg['content-transfer-encoding']
                msg.set_payload(payload, charset)
                payload = msg.get_payload()
                self._munge_cte = (msg['content-transfer-encoding'],
                                   msg['content-type'])
        if self._mangle_from_:
            payload = fcre.sub('>From ', payload)
        self._write_lines(payload)

    
    _writeBody = _handle_text

    def _handle_multipart(self, msg):
        
        
        
        msgtexts = []
        subparts = msg.get_payload()
        if subparts is None:
            subparts = []
        elif isinstance(subparts, str):
            
            self.write(subparts)
            return
        elif not isinstance(subparts, list):
            
            subparts = [subparts]
        for part in subparts:
            s = self._new_buffer()
            g = self.clone(s)
            g.flatten(part, unixfrom=False, linesep=self._NL)
            msgtexts.append(s.getvalue())
        
        boundary = msg.get_boundary()
        if not boundary:
            
            
            alltext = self._encoded_NL.join(msgtexts)
            boundary = self._make_boundary(alltext)
            msg.set_boundary(boundary)
        
        if msg.preamble is not None:
            if self._mangle_from_:
                preamble = fcre.sub('>From ', msg.preamble)
            else:
                preamble = msg.preamble
            self._write_lines(preamble)
            self.write(self._NL)
        
        self.write('--' + boundary + self._NL)
        
        if msgtexts:
            self._fp.write(msgtexts.pop(0))
        
        
        
        for body_part in msgtexts:
            
            self.write(self._NL + '--' + boundary + self._NL)
            
            self._fp.write(body_part)
        
        self.write(self._NL + '--' + boundary + '--' + self._NL)
        if msg.epilogue is not None:
            if self._mangle_from_:
                epilogue = fcre.sub('>From ', msg.epilogue)
            else:
                epilogue = msg.epilogue
            self._write_lines(epilogue)

    def _handle_multipart_signed(self, msg):
        
        
        
        p = self.policy
        self.policy = p.clone(max_line_length=0)
        try:
            self._handle_multipart(msg)
        finally:
            self.policy = p

    def _handle_message_delivery_status(self, msg):
        
        
        
        blocks = []
        for part in msg.get_payload():
            s = self._new_buffer()
            g = self.clone(s)
            g.flatten(part, unixfrom=False, linesep=self._NL)
            text = s.getvalue()
            lines = text.split(self._encoded_NL)
            
            if lines and lines[-1] == self._encoded_EMPTY:
                blocks.append(self._encoded_NL.join(lines[:-1]))
            else:
                blocks.append(text)
        
        
        
        self._fp.write(self._encoded_NL.join(blocks))

    def _handle_message(self, msg):
        s = self._new_buffer()
        g = self.clone(s)
        
        
        
        
        
        
        
        
        
        payload = msg._payload
        if isinstance(payload, list):
            g.flatten(msg.get_payload(0), unixfrom=False, linesep=self._NL)
            payload = s.getvalue()
        else:
            payload = self._encode(payload)
        self._fp.write(payload)

    
    
    
    
    
    @classmethod
    def _make_boundary(cls, text=None):
        
        
        token = random.randrange(sys.maxsize)
        boundary = ('=' * 15) + (_fmt % token) + '=='
        if text is None:
            return boundary
        b = boundary
        counter = 0
        while True:
            cre = cls._compile_re('^--' + re.escape(b) + '(--)?$', re.MULTILINE)
            if not cre.search(text):
                break
            b = boundary + '.' + str(counter)
            counter += 1
        return b

    @classmethod
    def _compile_re(cls, s, flags):
        return re.compile(s, flags)


class BytesGenerator(Generator):
    

    def write(self, s):
        self._fp.write(s.encode('ascii', 'surrogateescape'))

    def _new_buffer(self):
        return BytesIO()

    def _encode(self, s):
        return s.encode('ascii')

    def _write_headers(self, msg):
        
        
        for h, v in msg.raw_items():
            self._fp.write(self.policy.fold_binary(h, v))
        
        self.write(self._NL)

    def _handle_text(self, msg):
        
        
        if msg._payload is None:
            return
        if _has_surrogates(msg._payload) and not self.policy.cte_type=='7bit':
            if self._mangle_from_:
                msg._payload = fcre.sub(">From ", msg._payload)
            self._write_lines(msg._payload)
        else:
            super(BytesGenerator,self)._handle_text(msg)

    
    _writeBody = _handle_text

    @classmethod
    def _compile_re(cls, s, flags):
        return re.compile(s.encode('ascii'), flags)


_FMT = '[Non-text (%(type)s) part of message omitted, filename %(filename)s]'

class DecodedGenerator(Generator):
    
    def __init__(self, outfp, mangle_from_=None, maxheaderlen=None, fmt=None, *,
                 policy=None):
        
        Generator.__init__(self, outfp, mangle_from_, maxheaderlen,
                           policy=policy)
        if fmt is None:
            self._fmt = _FMT
        else:
            self._fmt = fmt

    def _dispatch(self, msg):
        for part in msg.walk():
            maintype = part.get_content_maintype()
            if maintype == 'text':
                print(part.get_payload(decode=False), file=self)
            elif maintype == 'multipart':
                
                pass
            else:
                print(self._fmt % {
                    'type'       : part.get_content_type(),
                    'maintype'   : part.get_content_maintype(),
                    'subtype'    : part.get_content_subtype(),
                    'filename'   : part.get_filename('[no filename]'),
                    'description': part.get('Content-Description',
                                            '[no description]'),
                    'encoding'   : part.get('Content-Transfer-Encoding',
                                            '[no encoding]'),
                    }, file=self)



_width = len(repr(sys.maxsize-1))
_fmt = '%%0%dd' % _width


_make_boundary = Generator._make_boundary