BeeDict Module

The BeeDict module provides two high level on-disk dictionary implementations. The first (BeeDict) can work with arbitrary hashable key objects, while the second (BeeStringDict) uses limited sized strings as basis providing slightly better performance. Both variants need pickleable Python objects as keys and values.

Data transfer to and from the dictionaries is done in the same way as for in-memory dictionaries, e.g. d['key'] = 1; print d['key']; del d['key], so usage should be transparent to Python programs using either in-memory or on-disk dictionaries. Not all dictionary methods are understood though.

BeeDict Objects

BeeDict objects are on-disk dictionaries which use a hash-to-address index. Both Keys and values must be pickleable and can have arbitrary size (keys shouldn't be too long though); keys have to be hashable.

Hash collisions are treated by sequential reads of all records with the same hash value and testing for equality of keys. This can be expensive !

BeeDicts use a BeeStorage.BeeKeyValueStorage instance as storage object and a BeeIndex.BeeIntegerIndex instance as index object.

BeeDict objects are constructed using:

BeeDict(name,min_recordsize=0,readonly=0,recover=0, autocommit=0,validate=0)

Create an instance using name as basename for the data and index files. Two files will be created: name.dat and name.idx.

min_recordsize is passed to the BeeStorage as indicator of the minimum size for data records. readonly can be set to true to open the files in read-only mode, preventing any disk modifications.

To open the dictionary in recovery mode, pass a keyword recover=1. Then run .recover() and reopen using the normal settings. The AutoRecover() wrapper can take care of this action for you automatically.

If autocommit is true the cache control will do an automatic .commit() whenever the transaction log overflows.

If validate is true, the dictionary will run a validation check after having successfully opened storage and index. RecreateIndexError or RecoverError exceptions could be raised in case inconsistencies are found.

BeeDict Instance Methods

BeeDict objects have the following methods:

flush()

Flush buffers to disk.

close()

Flush buffers and close.

This does not issue a .commit(), so the current transaction is rolled back.

commit()

Commits all changes and starts a new transaction.

rollback()

Rolls back the current transaction. All changes are undone.

changed()

Return true in case the current transaction includes changes to the database, false otherwise.

has_key()

Check if the dictionary has an item indexed by key.

Successfully found items are put in the cache for fast subsequent access.

get(key,default=None)

Get item indexed by key from the dictionary or default if no such item exists.

This first tries to read the item from cache and reverts to the disk storage if it is not found.

cursor(key,default=None)

Return a BeeDictCursor instance for the dictionary.

In case the key is not found, default is returned instead.

Note that cursors operate with the data on disk meaning that any uncommitted changes will not be seen by the cursor.

garbage()

Determine the amount of garbage in bytes that has accumulated in the storage file.

This amount would be freed if .collect() were run.

collect()

Run the storage garbage collector.

Storage collection can only be done for writeable dictionaries and then only if the current transaction does not contain any pending changes.

This can take a while depending on the size of the dictionary.

recover()

Recover all valid records and recreate the index.

keys()

Return a list of keys.

The method does not load any data into the cache, but does take notice of uncommitted changes.

values()

Return a list of values.

The method does not load any data into the cache, but does take notice of uncommitted changes.

items()

Return a list of items.

The method does not load any data into the cache, but does take notice of uncommitted changes.

BeeDict Instance Attributes

BeeDict objects have the following read-only attributes:

name

Basenname of the dictionary. The implementation uses two files for the on-disk representation: name.dat and name.idx.

closed

Closed flag.

readonly

Read-only flag.

autocommit

Autocommit flag.

FirstKey

Special key object useable to represent the first key in the (sorted) B+Tree index.

LastKey

Special key object useable to represent the last key in the (sorted) B+Tree index.

BeeDictCursor Objects

BeeDickCursor objects are intended to iterate over the database one item at a time without the need to read all keys. You can then read/write to the current cursor position and thus modify the dictionary in place.

Note that modifying the targetted dictionary while using a cursor can cause the cursor to skip new entries or fail due to deleted items. Especially deleting the key to which the cursor currently points can cause errors to be raised. In all other cases, the cursor will be repositioned.

BeeDictCursor objects are constructed using the BeeDict .cursor() method.

BeeDictCursor Instance Methods

BeeDictCursor objects have the following methods:

position(key,value=None)

Position the index cursor to index[key]. If value is given, index[key] == value is assured.

key may also be FirstKey or LastKey (in which case value has to be None).

next()

Moves to the next entry in the dictionary.

Returns true on success, false if the end-of-data has been reached.

prev()

Moves to the previous entry in the dictionary.

Returns true on success, false if the end-of-data has been reached.

read()

Reads the object from the dict to which the cursor currently points.

read_key()

Reads the key object from the dict to which the cursor currently points.

write()

Writes the object to the dict under the key to which the cursor currently points.

The new data is not written to disk until the dictionary's current transaction is committed.

BeeDictCursor Instance Attributes

BeeDictCursor don't have any useful attributes. Use the instance methods to query the key and value objects.

BeeStringDict Objects

BeeStringDict objects are on-disk dictionaries which use a limited size string to address index. Values must be pickleable and can have arbitrary size.

Since hash collisions cannot occur this dictionary type may have some performance advantages over the standard BeeDict dictionary.

BeeStringDict objects are constructed using:

BeeStringDict(name,keysize=10,min_recordsize=0,readonly=0, recover=0,autocommit=0,validate=0)

Create an instance using name as basename for the data and index files. Two files will be created: name.dat and name.idx.

keysize gives the maximal size of the strings used as index keys. min_recordsize is passed to the BeeStorage as indicator of the minimum size for data records. readonly can be set to true to open the files in read-only mode, preventing any disk modifications.

To open the dictionary in recovery mode, pass a keyword recover=1. Then run .recover() and reopen using the normal settings. The AutoRecover() wrapper can take care of this action for you automatically.

If autocommit is true the cache control will do an automatic .commit() whenever the transaction log overflows.

If validate is true, the dictionary will run a validation check after having successfully opened storage and index. RecreateIndexError or RecoverError exceptions could be raised in case inconsistencies are found.

Note that the keysize is currently not stored in the dictionary itself -- you'll have to store this information in some other form. This may change in future versions.

BeeStringDict Instance Methods

BeeStringDict objects have the following methods:

commit()

Commits all changes and starts a new transaction.

rollback()

Rolls back the current transaction. All changes are undone.

changed()

Return true in case the current transaction includes changes to the database, false otherwise.

has_key()

Check if the dictionary has an item indexed by key.

Successfully found items are put in the cache for fast subsequent access.

get(key,default=None)

Get item indexed by key from the dictionary or default if no such item exists.

This first tries to read the item from cache and reverts to the disk storage if it is not found.

cursor(key,default=None)

Return a BeeStringDictCursor instance for the dictionary.

In case the key is not found, default is returned instead.

Note that cursors operate with the data on disk meaning that any uncommitted changes will not be seen by the cursor.

garbage()

Determine the amount of garbage in bytes that has accumulated in the storage file.

This amount would be freed if .collect() were run.

collect()

Run the storage garbage collector.

Storage collection can only be done for writeable dictionaries and then only if the current transaction does not contain any pending changes.

This can take a while depending on the size of the dictionary.

recover()

Recover all valid records and recreate the index.

keys()

Return a list of keys.

The method will raise an error if there are uncommitted changes pending. Output is sorted ascending according to keys.

values()

Return a list of values.

The method will raise an error if there are uncommitted changes pending. Output is sorted ascending according to keys.

items()

Return a list of items.

The method will raise an error if there are uncommitted changes pending. Output is sorted ascending according to keys.

BeeStringDict Instance Attributes

BeeDict objects have the following read-only attributes:

name

Basenname of the dictionary. The implementation uses two files for the on-disk representation: name.dat and name.idx.

closed

Closed flag.

readonly

Read-only flag.

autocommit

Autocommit flag.

FirstKey

Special key object useable to represent the first key in the (sorted) B+Tree index.

LastKey

Special key object useable to represent the last key in the (sorted) B+Tree index.

BeeStringDictCursor Objects

BeeDickCursor objects are intended to iterate over the database one item at a time without the need to read all keys. You can then read/write to the current cursor position and thus modify the dictionary in place.

Note that modifying the targetted dictionary while using a cursor can cause the cursor to skip new entries or fail due to deleted items. Especially deleting the key to which the cursor currently points can cause errors to be raised. In all other cases, the cursor will be repositioned.

BeeStringDictCursor objects are constructed using the BeeStringDict .cursor() method.

BeeStringDictCursor Instance Methods

BeeStringDictCursor objects have the following methods:

position(key,value=None)

Position the index cursor to index[key]. If value is given, index[key] == value is assured.

key may also be FirstKey or LastKey (in which case value has to be None).

next()

Moves to the next entry in the dictionary.

Returns true on success, false if the end-of-data has been reached.

prev()

Moves to the previous entry in the dictionary.

Returns true on success, false if the end-of-data has been reached.

read()

Reads the object from the dict to which the cursor currently points.

read_key()

Reads the key object from the dict to which the cursor currently points.

write()

Writes the object to the dict under the key to which the cursor currently points.

The new data is not written to disk until the dictionary's current transaction is committed.

BeeStringDictCursor Instance Attributes

BeeStringDictCursor objects have the following read-only attributes:

key

Key string which dereferences to the current cursor position.

Functions

These functions are available:

AutoRecover(Class,*args,**kws)

Wrapper that can be used around the dictionary constructors to provide automatic recovery whenever needed (if possible).

Example: diskdict = AutoRecover(BeeDict.BeeDict, 'test')

Constants

These constants are available:

Error

Baseclass for errors related to this module. It is a subclass of exceptions.StandardError.

RecreateIndexError

This error is raised in case the index for a dictionary was not found and/or needs to be recreated by running recovery. It is a subclass of Error.

RecoverError

This error is raised in case the storage for a dictionary was found to be in an inconsistent state. It is a subclass of Error.

FirstKey

Special index key object representing the key of the first entry in the index (B+Tree's sort their data).

LastKey

Special index key object representing the key of the last entry in the index (B+Tree's sort their data).

Package Structure

[BeeBase]
       Doc/
       [mxBeeBase]
              test.py
       BeeBase.py
       BeeDict.py
       BeeIndex.py
       BeeStorage.py
       showBeeDict.py
 	

Entries enclosed in brackets are packages (i.e. they are directories that include a __init__.py file). Ones without brackets are just simple subdirectories that are not accessible via import.

The package imports all symbols from the BeeBase sub module which in turn imports the extension module, so you only need to 'import BeeBase' to start working.

Support

Copyright & License

History & Future

Things that still need to be done:

• Provide more examples.

• Test the package some more.

• Write more documentation.

Version 2.0.0 was the intial release.