stalwart/UPGRADING/v0_15.md

Upgrading from v0.14.x to v0.15.x

Stalwart v0.15.x introduces breaking changes to both the database schema and some configuration options.
Upgrading to this version requires a schema migration, which is performed automatically when Stalwart starts for the first time on v0.15.x.

Because this migration modifies how data is stored and indexed, it is important to understand what will change, what will be migrated, and how the upgrade may impact your deployment—especially for larger installations.

What's changed

Version 0.15.x introduces significant internal improvements focused on performance, storage efficiency, and accuracy:

• Optimized database schema: The database schema has been redesigned to use less storage space and significantly reduce the number of read and write operations required for common tasks.
• Rewritten search layer: The search subsystem has been completely rewritten to use a more efficient and scalable indexing strategy.
• Native full-text search for SQL backends: When using PostgreSQL or MySQL as the backend, Stalwart now leverages the database’s native full-text search capabilities, replacing the previous custom full-text search implementation.
• New spam classifier engine : The spam classifier has been rewritten to use the FTRL-Proximal algorithm instead of the previous Naive Bayes implementation. This change improves classification accuracy, reduces memory usage, and reduces storage requirements for training data.

What will be migrated

The migration process runs automatically at startup and will migrate the following data:

• E-mail metadata, including flags, folders, and parsed message representations. (The raw e-mail content stored in the blob store is not migrated.)
• Encryption-at-rest settings, which now also include a spam training privacy option
• MTA message queue metadata (The actual message contents are not migrated.)
• Maintenance tasks
• Blob links (The underlying blobs themselves are not migrated.)
• Search indexes, which will be rebuilt using the new indexing strategy

Important considerations

• For deployments with 1,000 or more mailboxes, the migration may take a considerable amount of time, depending on the volume of stored data.
• During migration, Stalwart runs in read-only mode:
  • No new e-mail can be received
  • No outbound e-mail can be sent
• It is strongly recommended to perform this upgrade during a maintenance window.
• By default, the migration process is multithreaded and uses two threads for each available CPUs. You can control the number of threads by setting the following environment variable NUM_THREADS=<number>

Note: If you do not require any of the features introduced in v0.15.x, consider waiting for the next major release, which will introduce a proxy-based architecture allowing zero-downtime upgrades.

Upgrading steps

Binary installation

• Stop Stalwart in every single node of your cluster. If you are using the systemd service, you can do this with the following command:

  $ sudo systemctl stop stalwart
  

• Backup your data following your database system's instructions. For example, if you are using RocksDB or SQLite, you can simply copy the data directory to a backup location. If you are using PostgreSQL or MySQL, you can use the pg_dump or mysqldump commands to create a backup of your database.

• Download the latest binary for your platform from the releases page and replace the binary in /opt/stalwart/bin.

• Start the service. In a cluster, you can speed up the migration process by starting all nodes at once.

  $ sudo systemctl start stalwart
  

Containerized

• Stop the Stalwart container in every single node of your cluster. If you are using Docker, you can do this with the following command:

  $ docker stop stalwart
  

• Backup your data following your database system's instructions. For example, if you are using RocksDB or SQLite, you can simply copy the data directory to a backup location. If you are using PostgreSQL or MySQL, you can use the pg_dump or mysqldump commands to create a backup of your database.

• Pull the latest image and restart the container:

  $ docker pull stalwartlabs/stalwart:latest
  $ docker start stalwart
  

Post-upgrade steps

After the upgrade and migration complete, several follow-up steps are required or recommended:

• Upgrade the webadmin: Upgrade the webadmin interface by navigating to Manage → Maintenance → Update Webadmin

• Update the spam rules: Download and apply the latest spam rules from the webadmin Manage → Maintenance → Update Spam rules

• Update search settings: Review the updated documentation for search settings, as some configuration options have changed. In particular, the Elasticsearch backend now uses different authentication settings than previous versions.

• Rebuild search indexes: All search indexes must be rebuilt to take advantage of the new indexing strategy. This can be done from the webadmin interface Manage → Maintenance.

• Recalculate disk quotas for all accounts: This step is not required immediately, but it is recommended to perform it at some point after the upgrade. The new version includes additional metadata in quota calculations, so recalculating ensures accurate disk usage reporting.

  $ curl -X DELETE https://myserver.org/api/store/quota/<account_name> -u <admin_user>:<admin_pass> -k
  

• Delete deprecated spam classifier keys: Remove deprecated spam classifier keys from the memory store. These are the keys starting with the integer prefixes 12 to 16 and 17 to 18:

  • If you are using Redis:

    $ for code in {12..18}; do
        char=$(printf "\\x$(printf '%02x' $code)")
        redis-cli --scan --pattern "${char}*" | xargs -r redis-cli DEL
    done
    

  • If you are using your database as the in-memory store:

    $ /opt/stalwart/bin/stalwart --config /opt/stalwart/etc/config.toml --console
    Stalwart Server v0.15.2 Data Store CLI
    
    > delete y\x0c\x00 y\x12\xff
    > delete m\x0c\x00 m\x12\xff
    > exit
    

  • If you are using your database as the in-memory store with Docker:

    $ docker stop stalwart
    $ docker run -it --rm \
        -v <STALWART_DIR>:/opt/stalwart \
        --entrypoint /usr/local/bin/stalwart \
        stalwartlabs/stalwart:latest \
        --config /opt/stalwart/etc/config.toml --console
    Stalwart Server v0.15.2 Data Store CLI
    
    > delete y\x0c\x00 y\x12\xff
    > delete m\x0c\x00 m\x12\xff
    > exit
    
    $ docker start stalwart
    

Troubleshooting

Interrupted or stopped migration

If the migration process is interrupted or stopped, it can be resumed automatically by simply restarting Stalwart.

Data corruption detected error

If you see an error message similar to: Data corruption detected. This indicates that another node wrote data using the old format while the migration was in progress. This usually happens when the cluster was not fully stopped before starting the upgrade.

In order to resolve this issue, follow these steps:

1. Stop all Stalwart nodes.
2. Ensure all nodes are upgraded to v0.15.x.
3. Start the nodes again.

Forcing a migration

If the migration does not resume because the node responsible for it already marked it as completed, you can force migration using environment variables:

• Force re-migration of MTA queue metadata: FORCE_MIGRATE_QUEUE=4
• Force re-migration of blob links: FORCE_MIGRATE_BLOBS=4
• Force re-migration of a specific account: FORCE_MIGRATE_ACCOUNT=<account-id>
• Force re-migration of all data: FORCE_MIGRATE=4

Use these options with care and only when necessary.