RegistryAdapters

Rbot allows plugins to store arbitrary data in a persistent key-value storage, every plugin creates its own database file under ~/.rbot/registry_tc (the used database engine is appended, so in this case _tc for a tokyocabinet database). The adapter that is used determines the underlying database system, it depends on the configuration of the bot.

The corresponding configuration is core.db (currently supported values: tc dbm daybreak sqlite). The value can be changed in ~/.rbot/conf.yaml or using <user> config set core.db tc. If such a setting is missing, you can just add one, for instance core.db: tc.

Supported Registries are:

• Barkeley DB (bdb in older rbots, now dbm) The old bdb adapter is no longer supported by newer ruby versions, particulary because DBM is now in the standard library of ruby. DBM provides a wrapper to (among others) Barkeley DB. However, since it is very prone to data loss (in the event of crashes or power outages) that needs manual repairs I strongly recommend not using BDB or DBM if at all possible.

• Tokyocabinet (tc) Its crash resilient and relatively easy to install, this is the new default if available.

• Sqlite3 (sqlite) Well supported SQL database, but its not really optimized for just key-value storage, so its really slow compared to the others.

• Daybreak (daybreak) A simple and fast in-memory(!!) database.

Backup, Restore and Migration Script

It is important to note that files written by database systems are platform and version dependent.

That means that even minor version differences in the used database system are (most of the time) incompatible to each other. If you plan on moving the bot to another system, upgrading the database system or creating a backup you need to first save the database in a platform independent format.

I've written a script to help you perform those tasks.

Internally it builds an in-memory hash of the complete databases, then uses Marshal to export this to a binary file. The values itself are Marshal serialized and not touched, ruby itself does maintain backwards-compatibility in its Marshal format so we don't have to worry about it. (At least AFAIK)

Please make backups first and let me know of any problems:

~/rbot % ruby bin/rbotdb
RBot Registry Backup/Restore/Migrate
[Ruby: 2.0.0 | DBM: Berkeley DB 4.5.20: (September 27, 2013) | BDB: - | TokyoCabinet: 1.4.48 | Daybreak: 0.3.0 | SQLite: 1.3.8]
Usage: rbotdb COMMAND [OPTIONS]

Commands:
     backup: store rbot registry platform-independently in a file.
     restore: restore rbot registry from such a file.

Options:
    -t, --type TYPE                  format to backup/restore. Values: bdb, tc, dbm, daybreak, sqlite.
    -p, --profile [PROFILE]          rbot profile directory. Defaults to: ~/.rbot.
    -r, --registry [REGISTRY]        registry-path to read/write, Optional, defaults to: <PROFILE>/registry_<TYPE>.
    -f, --file [DBFILE]              cross-platform file to backup to/restore from. Defaults to: ./backup_2014-03-06_132732.rbot.

Backup and Restore Example

% cd ~/rbot
% ruby bin/rbotdb backup -t sqlite -p ~/.rbot2
RBot Registry Backup/Restore/Migrate
[Ruby: 2.0.0 | DBM: Berkeley DB 4.5.20: (September 27, 2013) | BDB: - | TokyoCabinet: 1.4.48 | Daybreak: 0.3.0 | SQLite: 1.3.8]
Using type=sqlite profile=/home/apoc/.rbot2 registry=nil
Found registry types: bdb=0 tc=0 dbm=7 daybreak=0 sqlite=7
Using registry type: sqlite
Read 7 registry files, with 14 entries.
Written registry to ./backup_2014-03-06_133203.rbot

That has created a backup of the registry from ~/.rbot2/registry_sqlite to a single file at ~/rbot/backup_2014-03-06_133203.rbot. If you want to restore it again (maybe on another system) to the default location of ~/.rbot using the tokyocabinet db:

% cd ~/rbot
% ruby bin/rbotdb restore -t tc -f ./backup_2014-03-06_133203.rbot
RBot Registry Backup/Restore/Migrate
[Ruby: 2.0.0 | DBM: Berkeley DB 4.5.20: (September 27, 2013) | BDB: - | TokyoCabinet: 1.4.48 | Daybreak: 0.3.0 | SQLite: 1.3.8]
Using type=tc profile=/home/apoc/.rbot
Read 7 registry files from backup file.
Using registry type: tc
Restore successful!

Make sure the registry directory does not already exist, it will fail with an error if it (~/.rbot/registry_tc) does.

Database Size / Optimization

Most database engines will constantly append to its database file and thus the files will increase in size over time with every insert and update, this is independent of the actual data they hold.

A good example for that is the urlplugin, it will cache data about the last 100 urls mentioned in a channel, thus constantly updating its data and consistently grow in size over time. To remedy this, all database engines provide some kind of optimization/vacuum operation that will "defragment" the database.

There several solutions for rbot:

• Automatic: After every restart / rescan the bot will automatically perform this optimization, this can cause problems with very old database files where it can take a long time to optimize. In this case its advised to manually optimize. (This does not work for DBM databases)

• Manual: This is database dependent, for tokyocabinet you can perform this using: find ~/.rbot/registry_tc -iname '*.tdb' | xargs -n1 tcbmgr optimize

• rbotdb: You can use the rbotdb migration script to first backup the databases, then restore it again. This will have the same effect as optimizing:

% cd ~/rbot
% ruby bin/rbotdb backup -t tc
[...]
Written registry to ./backup_2014-03-05_134335.rbot
% rm -R ~/.rbot/registry_tc # better: move it to some safe place
% ruby bin/rbotdb restore -t tc -f backup_2014-03-05_134335.rbot
[...]
Restore successful!