Postfix OpenLDAP LMDB Howto

Introduction

Postfix uses databases of various kinds to store and look up information. Postfix databases are specified as "type:name". OpenLDAP LMDB implements the Postfix database type "lmdb". The name of a Postfix OpenLDAP LMDB database is the name of the database file without the ".lmdb" suffix. OpenLDAP LMDB databases are maintained with the postmap(1) command.

This document describes:

How to build Postfix with OpenLDAP LMDB support.
How to configure LMDB settings.
Missing pthread library trouble.

Note:: The Postfix LMDB client implementation introduces unexpected failure modes that don't exist with other Postfix databases. Don't just yet abandon CDB.

Building Postfix with OpenLDAP LMDB support

Postfix normally does not enable OpenLDAP LMDB support. To build Postfix after you installed OpenLDAP LMDB from source code, use something like:

% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
    AUXLIBS="-L/usr/local/lib -llmdb"
% make

Solaris needs this:

% make makefiles CCARGS="-DHAS_LMDB -I/usr/local/include" \
    AUXLIBS="-R/usr/local/lib -L/usr/local/lib -llmdb"
% make

The exact pathnames depend on how OpenLDAP LMDB was installed.

Configure LMDB settings

Postfix provides a configuration parameter that controls how large an OpenLDAP LMDB database may grow.

lmdb_map_size (default: 10 MBytes per table). This setting controls how large any OpenLDAP LMDB database may grow. It must be set large enough to accommodate the largest table that Postfix will use.

Missing pthread library trouble

When building Postfix fails with:

undefined reference to `pthread_mutexattr_destroy'
undefined reference to `pthread_mutexattr_init'
undefined reference to `pthread_mutex_lock'

Add the "-lpthread" library to the "make makefiles" command.

% make makefiles .... AUXLIBS="... -lpthread"

Source code for OpenLDAP LMDB is available at http://www.openldap.org. More information is available at http://highlandsun.com/hyc/mdb/.

Unexpected failure modes of Postfix LMDB databases.

As documented below, conversion to LMDB introduces a number of failure modes that don't exist with other Postfix databases.

Unexpected postmap(1)/postalias(1) "database full" errors.

Problem:

The "postmap lmdb:filename" command fails with an MDB_MAP_FULL error. This problem does not exist with other Postfix databases.

Background:

LMDB databases have a hard size limit (configured with the lmdb_map_size configuration parameter).

When executing "postmap lmdb:filename", the Postfix LMDB database client stores the new data in a transaction which takes up space in addition to the existing data, and commits the transaction when it closes the database. Only then can the space for old data be reused.

Impact:

This failure does not affect Postfix availability, because the old data still exists in the database.

Mitigation:

When the postmap(1) or postalias(1) command opens an LMDB file larger than lmdb_map_size/3, it logs a warning and uses a larger size limit instead:

warning: filename.lmdb: file size 15024128 ≥ (lmdb map size limit 16777216)/3 -- using a larger map size limit

This can be used to automate recovery and avoid the need for human intervention. Just keep running "postmap lmdb:filename". After each failure it will use a 3x larger size limit, and eventually the "database full" error will disappear.

Prevention:

Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest LMDB file size.

Unexpected Postfix daemon "database full" errors.

Problem:

Postfix daemon programs fail with "database full" errors, such as postscreen(8), tlsmgr(8) or verify(8). This problem does not exist with other Postfix databases.

Impact:

This failure temporarily affects Postfix availability. The daemon restarts automatically and tries to open the database again as described next.

Mitigation:

When a Postfix daemon opens an LMDB file larger than lmdb_map_size/3, it logs a warning and uses a larger size limit instead:

warning: filename.lmdb: file size 15024128 ≥ (lmdb map size limit 16777216)/3 -- using a larger map size limit

This can be used to automate recovery and avoid the need for human intervention. Each time the daemon runs into a "database full" error, it restarts and uses a 3x larger size limit. The "database full" error will disappear, at least for a while.

Prevention:

Monitor your LMDB files and make sure that lmdb_map_size > 3x the largest LMDB file size.

Non-obvious recovery with postmap(1)/postalias(1)/tlsmgr(8) from a corrupted database.

Problem:: You cannot rebuild a corrupted LMDB database simply by running postmap(1) or postalias(1), or by waiting until the tlsmgr(8) daemon restarts automatically. This problem does not exist with other Postfix databases.
Background:: The Postfix LMDB database client does not truncate the database file. Instead it attempts to create a transaction for a "drop" request and subsequent "store" requests. That is obviously not possible with a corrupted database file.
Impact:: Postfix does not process mail until someone fixes the problem.
Recovery:: First delete the ".lmdb" file by hand, then rebuild the file with the postmap(1) or postalias(1) command, or wait until the tlsmgr(8) daemon restarts automatically.
Prevention:: Arrange your file systems such that they never run out of free space.