c 20042006 python software foundation s baxter wouters and warsaw contact emailsigpython org feedparser an email feed parser the feed parser implements an interface for incrementally parsing an email message line by line this has advantages for certain applications such as those reading email messages off a socket feedparser feed is the primary interface for pushing new data into the parser it returns when there s nothing more it can do with the available data when you have no more data to push into the parser call close this completes the parsing and returns the root message object the other advantage of this parser is that it will never raise a parsing exception instead when it finds something unexpected it adds a defect to the current message defects are just instances that live on the message object s defects attribute rfc 2822 3 6 8 optional fields ftext is d3357 d59126 any character except controls sp and a fileish object that can have new data loaded into it you can also push and pop linematching predicates onto a stack when the current predicate matches the current line a false eof response i e empty string is returned instead this lets the parser adhere to a simple abstraction it parses until eof closes the current message text stream of the last partial line pushed into this object see issue 22233 for why this is a text stream and not a list a deque of full pushed lines the stack of falseeof checking predicates a flag indicating whether the file has been closed or not don t forget any trailing partial line pop the line off the stack and see if it matches the current falseeof predicate rfc 2046 section 5 1 2 requires us to recognize outer level boundaries at any level of inner nesting do this but be sure it s in the order of most to least nested we re at the false eof but push the last line back first let the consumer push a line back into the buffer push some new data into this object self partial writedata if n not in data and r not in data no new complete lines wait for more return crack into lines preserving the linesep characters self partial seek0 parts self partial readlines self partial seek0 self partial truncate if the last element of the list does not end in a newline then treat it as a partial line we only check for n here because a line ending with r might be a line that was split in the middle of a rn sequence see bugs 1555570 and 1721862 if not parts1 endswith n self partial writeparts pop self pushlinesparts def pushlinesself lines self lines extendlines def iterself return self def nextself line self readline if line raise stopiteration return line class feedparser factory is called with no arguments to create a new message obj the policy keyword specifies a policy object that controls a number of aspects of the parser s operation the default policy maintains backward compatibility assume this is an oldstyle factory nonpublic interface for supporting parser s headersonly flag push more data into the parser self input pushdata self callparse def callparseself try self parse except stopiteration pass def closeself look for final set of defects create a new message and start by parsing headers collect the headers searching for a line that doesn t match the rfc 2822 header or continuation pattern including an empty line if we saw the rfc defined headerbody separator i e newline just throw it away otherwise the line is part of the body so push it back done with the headers so parse them and figure out what we re supposed to see in the body of the message headersonly parsing is a backwards compatibility hack which was necessary in the older parser which could raise errors all remaining lines in the input are thrown into the message body messagedeliverystatus contains blocks of headers separated by a blank line we ll represent each header block as a separate nested message object but the processing is a bit different than standard message types because there is no body for the nested messages a blank line separates the subparts we need to pop the eof matcher in order to tell if we re at the end of the current file not the end of the last block of message headers the input stream must be sitting at the newline or at the eof we want to see if we re at the end of this subpart so first consume the blank line then test the next line to see if we re at this subpart s eof not at eof so this is a line we re going to need the message claims to be a message type then what follows is another rfc 2822 message the message claims to be a multipart but it has not defined a boundary that s a problem which we ll handle by reading everything until the eof and marking the message as defective make sure a valid content type was specified per rfc 2045 6 4 create a line match predicate which matches the interpart boundary as well as the endofmultipart boundary don t push this onto the input stream until we ve scanned past the preamble if we re looking at the end boundary we re done with this multipart if there was a newline at the end of the closing boundary then we need to initialize the epilogue with the empty string see below we saw an interpart boundary were we in the preamble according to rfc 2046 the last newline belongs to the boundary we saw a boundary separating two parts consume any multiple boundary lines that may be following our interpretation of rfc 2046 bnf grammar does not produce body parts within such double boundaries recurse to parse this subpart the input stream points at the subpart s first line because of rfc 2046 the newline preceding the boundary separator actually belongs to the boundary not the previous subpart s payload or epilogue if the previous part is a multipart set the multipart up for newline cleansing which will happen if we re in a nested multipart i think we must be in the preamble we ve seen either the eof or the end boundary if we re still capturing the preamble we never saw the start boundary note that as a defect and store the captured text as the payload if we re not processing the preamble then we might have seen eof without seeing that end boundary that is also a defect everything from here to the eof is epilogue if the end boundary ended in a newline we ll need to make sure the epilogue isn t none any crlf at the front of the epilogue is not technically part of the epilogue also watch out for an empty string epilogue which means a single newline otherwise it s some nonmultipart type so the entire rest of the file contents becomes the payload passed a list of lines that make up the headers for the current msg check for continuation the first line of the headers was a continuation this is illegal so let s note the defect store the illegal line and ignore it for purposes of headers check for envelope header i e unixfrom strip off the trailing newline something looking like a unixfrom at the end it s probably the first line of the body so push back the line and stop weirdly placed unixfrom line note this as a defect and ignore it split the line on the colon separating field name from value there will always be a colon because if there wasn t the part of the parser that calls us would have started parsing the body if the colon is on the start of the line the header is clearly malformed but we might be able to salvage the rest of the message track the error but keep going done with all the lines so handle the last header like feedparser but feed accepts bytes def feedself data super feeddata decode ascii surrogateescape c 2004 2006 python software foundation s baxter wouters and warsaw contact email sig python org feedparser an email feed parser the feed parser implements an interface for incrementally parsing an email message line by line this has advantages for certain applications such as those reading email messages off a socket feedparser feed is the primary interface for pushing new data into the parser it returns when there s nothing more it can do with the available data when you have no more data to push into the parser call close this completes the parsing and returns the root message object the other advantage of this parser is that it will never raise a parsing exception instead when it finds something unexpected it adds a defect to the current message defects are just instances that live on the message object s defects attribute rfc 2822 3 6 8 optional fields ftext is d33 57 d59 126 any character except controls sp and a file ish object that can have new data loaded into it you can also push and pop line matching predicates onto a stack when the current predicate matches the current line a false eof response i e empty string is returned instead this lets the parser adhere to a simple abstraction it parses until eof closes the current message text stream of the last partial line pushed into this object see issue 22233 for why this is a text stream and not a list a deque of full pushed lines the stack of false eof checking predicates a flag indicating whether the file has been closed or not don t forget any trailing partial line pop the line off the stack and see if it matches the current false eof predicate rfc 2046 section 5 1 2 requires us to recognize outer level boundaries at any level of inner nesting do this but be sure it s in the order of most to least nested we re at the false eof but push the last line back first let the consumer push a line back into the buffer push some new data into this object no new complete lines wait for more crack into lines preserving the linesep characters if the last element of the list does not end in a newline then treat it as a partial line we only check for n here because a line ending with r might be a line that was split in the middle of a r n sequence see bugs 1555570 and 1721862 a feed style parser of email _factory is called with no arguments to create a new message obj the policy keyword specifies a policy object that controls a number of aspects of the parser s operation the default policy maintains backward compatibility assume this is an old style factory non public interface for supporting parser s headersonly flag push more data into the parser parse all remaining data and return the root message object look for final set of defects create a new message and start by parsing headers collect the headers searching for a line that doesn t match the rfc 2822 header or continuation pattern including an empty line if we saw the rfc defined header body separator i e newline just throw it away otherwise the line is part of the body so push it back done with the headers so parse them and figure out what we re supposed to see in the body of the message headers only parsing is a backwards compatibility hack which was necessary in the older parser which could raise errors all remaining lines in the input are thrown into the message body message delivery status contains blocks of headers separated by a blank line we ll represent each header block as a separate nested message object but the processing is a bit different than standard message types because there is no body for the nested messages a blank line separates the subparts we need to pop the eof matcher in order to tell if we re at the end of the current file not the end of the last block of message headers the input stream must be sitting at the newline or at the eof we want to see if we re at the end of this subpart so first consume the blank line then test the next line to see if we re at this subpart s eof not at eof so this is a line we re going to need the message claims to be a message type then what follows is another rfc 2822 message the message claims to be a multipart but it has not defined a boundary that s a problem which we ll handle by reading everything until the eof and marking the message as defective make sure a valid content type was specified per rfc 2045 6 4 create a line match predicate which matches the inter part boundary as well as the end of multipart boundary don t push this onto the input stream until we ve scanned past the preamble if we re looking at the end boundary we re done with this multipart if there was a newline at the end of the closing boundary then we need to initialize the epilogue with the empty string see below we saw an inter part boundary were we in the preamble according to rfc 2046 the last newline belongs to the boundary we saw a boundary separating two parts consume any multiple boundary lines that may be following our interpretation of rfc 2046 bnf grammar does not produce body parts within such double boundaries recurse to parse this subpart the input stream points at the subpart s first line because of rfc 2046 the newline preceding the boundary separator actually belongs to the boundary not the previous subpart s payload or epilogue if the previous part is a multipart set the multipart up for newline cleansing which will happen if we re in a nested multipart i think we must be in the preamble we ve seen either the eof or the end boundary if we re still capturing the preamble we never saw the start boundary note that as a defect and store the captured text as the payload if we re not processing the preamble then we might have seen eof without seeing that end boundary that is also a defect everything from here to the eof is epilogue if the end boundary ended in a newline we ll need to make sure the epilogue isn t none any crlf at the front of the epilogue is not technically part of the epilogue also watch out for an empty string epilogue which means a single newline otherwise it s some non multipart type so the entire rest of the file contents becomes the payload passed a list of lines that make up the headers for the current msg check for continuation the first line of the headers was a continuation this is illegal so let s note the defect store the illegal line and ignore it for purposes of headers check for envelope header i e unix from strip off the trailing newline something looking like a unix from at the end it s probably the first line of the body so push back the line and stop weirdly placed unix from line note this as a defect and ignore it split the line on the colon separating field name from value there will always be a colon because if there wasn t the part of the parser that calls us would have started parsing the body if the colon is on the start of the line the header is clearly malformed but we might be able to salvage the rest of the message track the error but keep going done with all the lines so handle the last header like feedparser but feed accepts bytes
__all__ = ['FeedParser', 'BytesFeedParser']

import re

from email import errors
from email._policybase import compat32
from collections import deque
from io import StringIO

NLCRE = re.compile(r'\r\n|\r|\n')
NLCRE_bol = re.compile(r'(\r\n|\r|\n)')
NLCRE_eol = re.compile(r'(\r\n|\r|\n)\Z')
NLCRE_crack = re.compile(r'(\r\n|\r|\n)')


headerRE = re.compile(r'^(From |[\041-\071\073-\176]*:|[\t ])')
EMPTYSTRING = ''
NL = '\n'
boundaryendRE = re.compile(
    r'(?P<end>--)?(?P<ws>[ \t]*)(?P<linesep>\r\n|\r|\n)?$')

NeedMoreData = object()


class BufferedSubFile(object):
    
    def __init__(self):
        
        
        self._partial = StringIO(newline='')
        
        self._lines = deque()
        
        self._eofstack = []
        
        self._closed = False

    def push_eof_matcher(self, pred):
        self._eofstack.append(pred)

    def pop_eof_matcher(self):
        return self._eofstack.pop()

    def close(self):
        
        self._partial.seek(0)
        self.pushlines(self._partial.readlines())
        self._partial.seek(0)
        self._partial.truncate()
        self._closed = True

    def readline(self):
        if not self._lines:
            if self._closed:
                return ''
            return NeedMoreData
        
        
        line = self._lines.popleft()
        
        
        
        for ateof in reversed(self._eofstack):
            if ateof(line):
                
                self._lines.appendleft(line)
                return ''
        return line

    def unreadline(self, line):
        
        assert line is not NeedMoreData
        self._lines.appendleft(line)

    def push(self, data):
        
        self._partial.write(data)
        if '\n' not in data and '\r' not in data:
            
            return

        
        self._partial.seek(0)
        parts = self._partial.readlines()
        self._partial.seek(0)
        self._partial.truncate()

        
        
        
        
        if not parts[-1].endswith('\n'):
            self._partial.write(parts.pop())
        self.pushlines(parts)

    def pushlines(self, lines):
        self._lines.extend(lines)

    def __iter__(self):
        return self

    def __next__(self):
        line = self.readline()
        if line == '':
            raise StopIteration
        return line


class FeedParser:
    

    def __init__(self, _factory=None, *, policy=compat32):
        
        self.policy = policy
        self._old_style_factory = False
        if _factory is None:
            if policy.message_factory is None:
                from email.message import Message
                self._factory = Message
            else:
                self._factory = policy.message_factory
        else:
            self._factory = _factory
            try:
                _factory(policy=self.policy)
            except TypeError:
                
                self._old_style_factory = True
        self._input = BufferedSubFile()
        self._msgstack = []
        self._parse = self._parsegen().__next__
        self._cur = None
        self._last = None
        self._headersonly = False

    
    def _set_headersonly(self):
        self._headersonly = True

    def feed(self, data):
        
        self._input.push(data)
        self._call_parse()

    def _call_parse(self):
        try:
            self._parse()
        except StopIteration:
            pass

    def close(self):
        
        self._input.close()
        self._call_parse()
        root = self._pop_message()
        assert not self._msgstack
        
        if root.get_content_maintype() == 'multipart' \
               and not root.is_multipart() and not self._headersonly:
            defect = errors.MultipartInvariantViolationDefect()
            self.policy.handle_defect(root, defect)
        return root

    def _new_message(self):
        if self._old_style_factory:
            msg = self._factory()
        else:
            msg = self._factory(policy=self.policy)
        if self._cur and self._cur.get_content_type() == 'multipart/digest':
            msg.set_default_type('message/rfc822')
        if self._msgstack:
            self._msgstack[-1].attach(msg)
        self._msgstack.append(msg)
        self._cur = msg
        self._last = msg

    def _pop_message(self):
        retval = self._msgstack.pop()
        if self._msgstack:
            self._cur = self._msgstack[-1]
        else:
            self._cur = None
        return retval

    def _parsegen(self):
        
        self._new_message()
        headers = []
        
        
        for line in self._input:
            if line is NeedMoreData:
                yield NeedMoreData
                continue
            if not headerRE.match(line):
                
                
                
                if not NLCRE.match(line):
                    defect = errors.MissingHeaderBodySeparatorDefect()
                    self.policy.handle_defect(self._cur, defect)
                    self._input.unreadline(line)
                break
            headers.append(line)
        
        
        self._parse_headers(headers)
        
        
        
        if self._headersonly:
            lines = []
            while True:
                line = self._input.readline()
                if line is NeedMoreData:
                    yield NeedMoreData
                    continue
                if line == '':
                    break
                lines.append(line)
            self._cur.set_payload(EMPTYSTRING.join(lines))
            return
        if self._cur.get_content_type() == 'message/delivery-status':
            
            
            
            
            
            while True:
                self._input.push_eof_matcher(NLCRE.match)
                for retval in self._parsegen():
                    if retval is NeedMoreData:
                        yield NeedMoreData
                        continue
                    break
                self._pop_message()
                
                
                
                self._input.pop_eof_matcher()
                
                
                
                
                while True:
                    line = self._input.readline()
                    if line is NeedMoreData:
                        yield NeedMoreData
                        continue
                    break
                while True:
                    line = self._input.readline()
                    if line is NeedMoreData:
                        yield NeedMoreData
                        continue
                    break
                if line == '':
                    break
                
                self._input.unreadline(line)
            return
        if self._cur.get_content_maintype() == 'message':
            
            
            for retval in self._parsegen():
                if retval is NeedMoreData:
                    yield NeedMoreData
                    continue
                break
            self._pop_message()
            return
        if self._cur.get_content_maintype() == 'multipart':
            boundary = self._cur.get_boundary()
            if boundary is None:
                
                
                
                
                defect = errors.NoBoundaryInMultipartDefect()
                self.policy.handle_defect(self._cur, defect)
                lines = []
                for line in self._input:
                    if line is NeedMoreData:
                        yield NeedMoreData
                        continue
                    lines.append(line)
                self._cur.set_payload(EMPTYSTRING.join(lines))
                return
            
            if (str(self._cur.get('content-transfer-encoding', '8bit')).lower()
                    not in ('7bit', '8bit', 'binary')):
                defect = errors.InvalidMultipartContentTransferEncodingDefect()
                self.policy.handle_defect(self._cur, defect)
            
            
            
            
            separator = '--' + boundary
            def boundarymatch(line):
                if not line.startswith(separator):
                    return None
                return boundaryendRE.match(line, len(separator))
            capturing_preamble = True
            preamble = []
            linesep = False
            close_boundary_seen = False
            while True:
                line = self._input.readline()
                if line is NeedMoreData:
                    yield NeedMoreData
                    continue
                if line == '':
                    break
                mo = boundarymatch(line)
                if mo:
                    
                    
                    
                    
                    if mo.group('end'):
                        close_boundary_seen = True
                        linesep = mo.group('linesep')
                        break
                    
                    if capturing_preamble:
                        if preamble:
                            
                            
                            lastline = preamble[-1]
                            eolmo = NLCRE_eol.search(lastline)
                            if eolmo:
                                preamble[-1] = lastline[:-len(eolmo.group(0))]
                            self._cur.preamble = EMPTYSTRING.join(preamble)
                        capturing_preamble = False
                        self._input.unreadline(line)
                        continue
                    
                    
                    
                    
                    while True:
                        line = self._input.readline()
                        if line is NeedMoreData:
                            yield NeedMoreData
                            continue
                        mo = boundarymatch(line)
                        if not mo:
                            self._input.unreadline(line)
                            break
                    
                    
                    self._input.push_eof_matcher(boundarymatch)
                    for retval in self._parsegen():
                        if retval is NeedMoreData:
                            yield NeedMoreData
                            continue
                        break
                    
                    
                    
                    
                    if self._last.get_content_maintype() == 'multipart':
                        epilogue = self._last.epilogue
                        if epilogue == '':
                            self._last.epilogue = None
                        elif epilogue is not None:
                            mo = NLCRE_eol.search(epilogue)
                            if mo:
                                end = len(mo.group(0))
                                self._last.epilogue = epilogue[:-end]
                    else:
                        payload = self._last._payload
                        if isinstance(payload, str):
                            mo = NLCRE_eol.search(payload)
                            if mo:
                                payload = payload[:-len(mo.group(0))]
                                self._last._payload = payload
                    self._input.pop_eof_matcher()
                    self._pop_message()
                    
                    
                    self._last = self._cur
                else:
                    
                    assert capturing_preamble
                    preamble.append(line)
            
            
            
            if capturing_preamble:
                defect = errors.StartBoundaryNotFoundDefect()
                self.policy.handle_defect(self._cur, defect)
                self._cur.set_payload(EMPTYSTRING.join(preamble))
                epilogue = []
                for line in self._input:
                    if line is NeedMoreData:
                        yield NeedMoreData
                        continue
                self._cur.epilogue = EMPTYSTRING.join(epilogue)
                return
            
            
            if not close_boundary_seen:
                defect = errors.CloseBoundaryNotFoundDefect()
                self.policy.handle_defect(self._cur, defect)
                return
            
            
            
            if linesep:
                epilogue = ['']
            else:
                epilogue = []
            for line in self._input:
                if line is NeedMoreData:
                    yield NeedMoreData
                    continue
                epilogue.append(line)
            
            
            
            if epilogue:
                firstline = epilogue[0]
                bolmo = NLCRE_bol.match(firstline)
                if bolmo:
                    epilogue[0] = firstline[len(bolmo.group(0)):]
            self._cur.epilogue = EMPTYSTRING.join(epilogue)
            return
        
        
        lines = []
        for line in self._input:
            if line is NeedMoreData:
                yield NeedMoreData
                continue
            lines.append(line)
        self._cur.set_payload(EMPTYSTRING.join(lines))

    def _parse_headers(self, lines):
        
        lastheader = ''
        lastvalue = []
        for lineno, line in enumerate(lines):
            
            if line[0] in ' \t':
                if not lastheader:
                    
                    
                    
                    defect = errors.FirstHeaderLineIsContinuationDefect(line)
                    self.policy.handle_defect(self._cur, defect)
                    continue
                lastvalue.append(line)
                continue
            if lastheader:
                self._cur.set_raw(*self.policy.header_source_parse(lastvalue))
                lastheader, lastvalue = '', []
            
            if line.startswith('From '):
                if lineno == 0:
                    
                    mo = NLCRE_eol.search(line)
                    if mo:
                        line = line[:-len(mo.group(0))]
                    self._cur.set_unixfrom(line)
                    continue
                elif lineno == len(lines) - 1:
                    
                    
                    
                    self._input.unreadline(line)
                    return
                else:
                    
                    
                    defect = errors.MisplacedEnvelopeHeaderDefect(line)
                    self._cur.defects.append(defect)
                    continue
            
            
            
            i = line.find(':')

            
            
            
            if i == 0:
                defect = errors.InvalidHeaderDefect("Missing header name.")
                self._cur.defects.append(defect)
                continue

            assert i>0, "_parse_headers fed line with no : and no leading WS"
            lastheader = line[:i]
            lastvalue = [line]
        
        if lastheader:
            self._cur.set_raw(*self.policy.header_source_parse(lastvalue))


class BytesFeedParser(FeedParser):
    

    def feed(self, data):
        super().feed(data.decode('ascii', 'surrogateescape'))