a dumb and slow but simple dbm clone for database spam spam dir contains the index a text file spam bak may contain a backup of the index also a text file while spam dat contains the data a binary file xxx to do seems to contain a bug when updating reclaim free space currently space once occupied by deleted or expanded items is never reused support concurrent access currently if two processes take turns making updates they can mess up the index support efficient access to large databases currently the whole index is read when the database is opened and some updates rewrite the whole index support opening for readonly flag m the ondisk directory and data files can remain in mutually inconsistent states for an arbitrarily long time see comments at the end of setitem this is only repaired when commit gets called one place commit gets called is from del and if that occurs at program shutdown time module globals may already have gotten rebound to none since it s crucial that commit finish successfully we can t ignore shutdown races here and commit must not reference any globals the directory file is a text file each line looks like r d dn key pos siz where key is the string key pos is the offset into the dat file of the associated value s first byte and siz is the number of bytes in the associated value the data file is a binary file pointed into by the directory file and holds the values associated with keys each value begins at a blocksizealigned byte offset and is a raw binary 8bit string value the index is an inmemory dict mirroring the directory file handle the creation mod by jack create data file if needed read directory file into the inmemory index dict write the index dict to the directory file the original directory file if any is renamed with a bak extension first if a bak file currently exists it s deleted caution it s vital that commit succeed and commit can be called from del therefore we must never reference a global in this routine use latin1 since it has no qualms with any value in any position utf8 though does care sometimes append val to the data file starting at a blocksizealigned offset the data file is first padded with nul bytes if needed to get to an aligned offset return pair starting offset of val lenval write val to the data file starting at offset pos the caller is responsible for ensuring that there s enough room starting at pos to hold val without overwriting some other value return pair pos lenval key is a new key whose associated value starts in the data file at offset pos and with length siz add an index record to the inmemory index dict and append one to the directory file see whether the new value is small enough to fit in the padded space currently occupied by the old value the new value doesn t fit in the padded space used by the old value the blocks used by the old value are forever lost note that index may be out of synch with the directory file now setval and addval don t update the directory file this also means that the ondisk directory and data files are in a mutually inconsistent state and they ll remain that way until commit is called note that this is a disaster for the database if the program crashes so that commit never gets called the blocks used by the associated value are lost xxx it s unclear why we do a commit here the code always xxx has so i m not changing it setitem doesn t try to xxx keep the directory file in synch why should we or xxx why shouldn t setitem open the database file filename and return corresponding object the flag argument used to control how the database is opened in the other dbm implementations supports only the semantics of c and n values other values will default to the semantics of c value the database will always opened for update and will be created if it does not exist the optional mode argument is the unix mode of the file used only when the database has to be created it defaults to octal code 0o666 and will be modified by the prevailing umask modify mode depending on the umask turn off any bits that are set in the umask the on disk directory and data files can remain in mutually inconsistent states for an arbitrarily long time see comments at the end of __setitem__ this is only repaired when _commit gets called one place _commit gets called is from __del__ and if that occurs at program shutdown time module globals may already have gotten rebound to none since it s crucial that _commit finish successfully we can t ignore shutdown races here and _commit must not reference any globals for _commit for _commit the directory file is a text file each line looks like r d d n key pos siz where key is the string key pos is the offset into the dat file of the associated value s first byte and siz is the number of bytes in the associated value the data file is a binary file pointed into by the directory file and holds the values associated with keys each value begins at a _blocksize aligned byte offset and is a raw binary 8 bit string value the index is an in memory dict mirroring the directory file maps keys to pos siz pairs handle the creation mod by jack create data file if needed read directory file into the in memory index dict write the index dict to the directory file the original directory file if any is renamed with a bak extension first if a bak file currently exists it s deleted caution it s vital that _commit succeed and _commit can be called from __del__ therefore we must never reference a global in this routine nothing to do use latin 1 since it has no qualms with any value in any position utf 8 though does care sometimes may raise keyerror append val to the data file starting at a _blocksize aligned offset the data file is first padded with nul bytes if needed to get to an aligned offset return pair starting offset of val len val write val to the data file starting at offset pos the caller is responsible for ensuring that there s enough room starting at pos to hold val without overwriting some other value return pair pos len val key is a new key whose associated value starts in the data file at offset pos and with length siz add an index record to the in memory index dict and append one to the directory file see whether the new value is small enough to fit in the padded space currently occupied by the old value the new value doesn t fit in the padded space used by the old value the blocks used by the old value are forever lost note that _index may be out of synch with the directory file now _setval and _addval don t update the directory file this also means that the on disk directory and data files are in a mutually inconsistent state and they ll remain that way until _commit is called note that this is a disaster for the database if the program crashes so that _commit never gets called the blocks used by the associated value are lost xxx it s unclear why we do a _commit here the code always xxx has so i m not changing it __setitem__ doesn t try to xxx keep the directory file in synch why should we or xxx why shouldn t __setitem__ open the database file filename and return corresponding object the flag argument used to control how the database is opened in the other dbm implementations supports only the semantics of c and n values other values will default to the semantics of c value the database will always opened for update and will be created if it does not exist the optional mode argument is the unix mode of the file used only when the database has to be created it defaults to octal code 0o666 and will be modified by the prevailing umask modify mode depending on the umask turn off any bits that are set in the umask
import ast as _ast
import io as _io
import os as _os
import collections.abc

__all__ = ["error", "open"]

_BLOCKSIZE = 512

error = OSError

class _Database(collections.abc.MutableMapping):

    
    
    
    
    
    
    
    
    _os = _os       
    _io = _io       

    def __init__(self, filebasename, mode, flag='c'):
        filebasename = self._os.fsencode(filebasename)
        self._mode = mode
        self._readonly = (flag == 'r')

        
        
        
        
        
        self._dirfile = filebasename + b'.dir'

        
        
        
        
        self._datfile = filebasename + b'.dat'
        self._bakfile = filebasename + b'.bak'

        
        self._index = None  

        
        self._create(flag)
        self._update(flag)

    def _create(self, flag):
        if flag == 'n':
            for filename in (self._datfile, self._bakfile, self._dirfile):
                try:
                    _os.remove(filename)
                except OSError:
                    pass
        
        try:
            f = _io.open(self._datfile, 'r', encoding="Latin-1")
        except OSError:
            if flag not in ('c', 'n'):
                raise
            with _io.open(self._datfile, 'w', encoding="Latin-1") as f:
                self._chmod(self._datfile)
        else:
            f.close()

    
    def _update(self, flag):
        self._modified = False
        self._index = {}
        try:
            f = _io.open(self._dirfile, 'r', encoding="Latin-1")
        except OSError:
            if flag not in ('c', 'n'):
                raise
            self._modified = True
        else:
            with f:
                for line in f:
                    line = line.rstrip()
                    key, pos_and_siz_pair = _ast.literal_eval(line)
                    key = key.encode('Latin-1')
                    self._index[key] = pos_and_siz_pair

    
    
    
    def _commit(self):
        
        
        
        if self._index is None or not self._modified:
            return  

        try:
            self._os.unlink(self._bakfile)
        except OSError:
            pass

        try:
            self._os.rename(self._dirfile, self._bakfile)
        except OSError:
            pass

        with self._io.open(self._dirfile, 'w', encoding="Latin-1") as f:
            self._chmod(self._dirfile)
            for key, pos_and_siz_pair in self._index.items():
                
                
                entry = "%r, %r\n" % (key.decode('Latin-1'), pos_and_siz_pair)
                f.write(entry)

    sync = _commit

    def _verify_open(self):
        if self._index is None:
            raise error('DBM object has already been closed')

    def __getitem__(self, key):
        if isinstance(key, str):
            key = key.encode('utf-8')
        self._verify_open()
        pos, siz = self._index[key]     
        with _io.open(self._datfile, 'rb') as f:
            f.seek(pos)
            dat = f.read(siz)
        return dat

    
    
    
    
    def _addval(self, val):
        with _io.open(self._datfile, 'rb+') as f:
            f.seek(0, 2)
            pos = int(f.tell())
            npos = ((pos + _BLOCKSIZE - 1) // _BLOCKSIZE) * _BLOCKSIZE
            f.write(b'\0'*(npos-pos))
            pos = npos
            f.write(val)
        return (pos, len(val))

    
    
    
    
    def _setval(self, pos, val):
        with _io.open(self._datfile, 'rb+') as f:
            f.seek(pos)
            f.write(val)
        return (pos, len(val))

    
    
    
    def _addkey(self, key, pos_and_siz_pair):
        self._index[key] = pos_and_siz_pair
        with _io.open(self._dirfile, 'a', encoding="Latin-1") as f:
            self._chmod(self._dirfile)
            f.write("%r, %r\n" % (key.decode("Latin-1"), pos_and_siz_pair))

    def __setitem__(self, key, val):
        if self._readonly:
            raise error('The database is opened for reading only')
        if isinstance(key, str):
            key = key.encode('utf-8')
        elif not isinstance(key, (bytes, bytearray)):
            raise TypeError("keys must be bytes or strings")
        if isinstance(val, str):
            val = val.encode('utf-8')
        elif not isinstance(val, (bytes, bytearray)):
            raise TypeError("values must be bytes or strings")
        self._verify_open()
        self._modified = True
        if key not in self._index:
            self._addkey(key, self._addval(val))
        else:
            
            
            pos, siz = self._index[key]
            oldblocks = (siz + _BLOCKSIZE - 1) // _BLOCKSIZE
            newblocks = (len(val) + _BLOCKSIZE - 1) // _BLOCKSIZE
            if newblocks <= oldblocks:
                self._index[key] = self._setval(pos, val)
            else:
                
                
                
                self._index[key] = self._addval(val)

            
            
            
            
            
            
            

    def __delitem__(self, key):
        if self._readonly:
            raise error('The database is opened for reading only')
        if isinstance(key, str):
            key = key.encode('utf-8')
        self._verify_open()
        self._modified = True
        
        del self._index[key]
        
        
        
        
        self._commit()

    def keys(self):
        try:
            return list(self._index)
        except TypeError:
            raise error('DBM object has already been closed') from None

    def items(self):
        self._verify_open()
        return [(key, self[key]) for key in self._index.keys()]

    def __contains__(self, key):
        if isinstance(key, str):
            key = key.encode('utf-8')
        try:
            return key in self._index
        except TypeError:
            if self._index is None:
                raise error('DBM object has already been closed') from None
            else:
                raise

    def iterkeys(self):
        try:
            return iter(self._index)
        except TypeError:
            raise error('DBM object has already been closed') from None
    __iter__ = iterkeys

    def __len__(self):
        try:
            return len(self._index)
        except TypeError:
            raise error('DBM object has already been closed') from None

    def close(self):
        try:
            self._commit()
        finally:
            self._index = self._datfile = self._dirfile = self._bakfile = None

    __del__ = close

    def _chmod(self, file):
        self._os.chmod(file, self._mode)

    def __enter__(self):
        return self

    def __exit__(self, *args):
        self.close()


def open(file, flag='c', mode=0o666):
    

    
    try:
        um = _os.umask(0)
        _os.umask(um)
    except AttributeError:
        pass
    else:
        
        mode = mode & (~um)
    if flag not in ('r', 'w', 'c', 'n'):
        raise ValueError("Flag must be one of 'r', 'w', 'c', or 'n'")
    return _Database(file, mode, flag=flag)