# Database schema management in Quassel IRC

## Introduction

Quassel IRC supports two database backends for storing chat history, user
sessions, etc:

* [PostgreSQL][postgres-home], in the [`PostgreSQL`](PostgreSQL/) folder
* [SQLite][sqlite-home], in the [`SQLite`](SQLite/) folder

For the most part, Quassel strives to have consistency between the two
backends, offering the same user experience and similar `.sql` query files,
though there are times when one database supports features that another does
not.

All of the current schema version queries are stored at the top level of the
backend subdirectory, with schema upgrade queries stored in
`[backend]/version/##` folders.

At compile time, Qt reads [`sql.qrc`][file-qrc-sql] to know which queries to
include.  This list can be automatically updated on Linux and macOS using
[`updateSQLResource.sh`][file-sh-updateresource]; for now, on Windows you'll
need to install some form of Bash.

## Managing the database outside of Quassel

Whenever possible, it's recommended to use Quassel itself to make changes, such
as deleting chats or changing user passwords.

However, some tasks aren't yet possible in Quassel, and you should
[read the documentation on managing the database][docs-wiki-db-manage] if the
need arises.

## Making changes to the database
### Changes to existing queries, no new/moved files

If you're only modifying existing queries in a way that does **not** require
any schema changes (*e.g. `ALTER TABLE`, `CREATE TABLE`, `CREATE INDEX`
statements*), you can just modify the `.sql` files in the appropriate backends.

However, any database schema changes must fulfill a number of requirements.

### New queries, schema changes, etc

> One does not simply 'upgrade the schema'...

When changing Quassel's database schema, you need to follow several steps to
handle all cases (*upgrade, Postgres migration, etc*).  Some of these steps may
not apply to all schema upgrades.

1.  **Run [`upgradeSchema.sh`][file-sh-upgradeschema] script on *both*
PostgreSQL and SQLite directories**

Make sure you're on the Git branch you want to modify

*Example:*
```sh
./upgradeSchema.sh "PostgreSQL"
./upgradeSchema.sh "SQLite"
```

Or, you can manually create new `[backend]/version/##` folders for both
`PostgreSQL` and `SQLite`.  Pick the next higher number from the largest
version number for each (*the exact version numbers will usually differ; that's
fine*).

2.  **Modify queries and setup scripts to handle your change**

The specifics depend on your change; in general, you'll want to modify any
query files that select from or insert to a modified table.  Then, modify the
`setup_###_[...].sql` files to include your changes on new database installs.

> *Example: modifying the `ircserver` table to add column `test`*
>
> Modify all query/setup `.sql` files that touch the `ircserver` table for
> *both* `PostgreSQL` and `SQLite`.

3.  **Create upgrade scripts for *both* PostgreSQL and SQLite**

These should go in the newest `[backend]/version/##` folders that were created
in Step 1.

Outside of special circumstances, do **not** modify the files in lower-numbered
version folders.  Existing Quassel cores have already run those statements and
will not run them if changed.

> *Example: modifying the `ircserver` table to add column `test`*
>
> Add the file `upgrade_000_alter_ircserver_add_test.sql` with contents:
>
> ```sql
> ALTER TABLE ircserver
> ADD COLUMN test [additional column-specific details]
> ```

4.  **Update the pair of migration scripts for moving from SQLite to
PostgreSQL**

For any table changes, you'll need to update the relevant
`SQLite/migrate_read_[table].sql` to read the existing data, and the
`PostgreSQL/migrate_write_[table].sql` to insert this data.

> *Example: modifying the `ircserver` table to add column `test`*
>
> Modify `SQLite/migrate_read_ircserver.sql` to select from new column
>
> Modify `PostgreSQL/migrate_write_ircserver.sql` to insert to new column

5.  **Update the SQL resource file; re-run CMake if needed**

**The easy way:**

Run [`updateSQLResource.sh`][file-sh-updateresource] in this directory.

**The manual way:**

Add the new SQL queries to [`src/core/sql.qrc`][file-qrc-sql], update all
changed existing files.

> *Example: modifying the `ircserver` table to add column `test`*
>
> Add the new upgrade scripts:
> ```diff
> +  <file>./SQL/SQLite/version/19/upgrade_000_alter_ircserver_add_test.sql</file>
> +  <file>./SQL/PostgreSQL/version/18/upgrade_000_alter_ircserver_add_test.sql</file>
> ```
>
> Add/update non-upgrade scripts, if any:
> ```diff
> +  <file>./SQL/SQLite/update_buffer_persistent_channel.sql</file>
> +  <file>./SQL/PostgreSQL/update_buffer_persistent_channel.sql</file>
> +  [etc]
> ```

6.  **Update the migration logic in
[`src/core/abstractsqlstorage.cpp`][file-cpp-abstract], and the storage
backends [`postgresqlstorage.cpp`][file-cpp-postgres] and
[`sqlitestorage.cpp`][file-cpp-sqlite]**

Make sure to read the data from SQLite in the right types and order, then write
it to PostgreSQL also with the right types and order.  SQLite and PostgreSQL
may use different database representations of certain types.

> *Example: modifying the `ircserver` table to add column `test`*
>
> Modify struct `IrcServerMO` in [`abstractsqlstorage.h`][file-h-abstract],
> adding an entry for `test` of the appropriate data-type.
>
> Modify `SqliteMigrationReader::readMo(IrcServerMO &ircserver)` in
> [`sqlitestorage.cpp`][file-cpp-sqlite] to read from the new column and store
> it in the migration object.  You may need to convert from SQLite's looser
> types.
>
> Modify `PostgreSqlMigrationWriter::writeMo(const IrcServerMO &ircserver)` in
> [`postgresqlstorage.cpp`][file-cpp-postgres] to write to the new column from
> the data in the migration object.

7.  **Update any affected queries in storage backends
[`postgresqlstorage.cpp`][file-cpp-postgres] and
[`sqlitestorage.cpp`][file-cpp-sqlite], and any related synchronized
`src/common` classes.**

> *Example: modifying the `ircserver` table to add column `test`*
>
> Update `network.h` to add new column to Server structure:
> ```diff
>    QString proxyPass;                                    // Existing code
> +  Typename test;                                        // New column 'test'
>    [...]
> -  Server() : port(6667), ..., proxyPort(8080) {}
> +  Server() : port(6667), ..., proxyPort(8080), test("defaultValue") {}
> ```
>
> Modify reading data in `[PostgreSql/Sqlite]Storage::networks(...)`:
> ```diff
>    server.proxyPass = serversQuery.value(10).toString(); // Existing code
> +  server.test = serversQuery.value(11).toType();        // New column 'test'
>    servers << server;                                    // Existing code
> ```
>
> Modify writing data in `[PostgreSql/Sqlite]Storage::bindServerInfo(...)`:
> ```diff
>    query.bindValue(":proxypass", server.proxyPass);      // Existing code
> +  query.bindValue(":test", server.test);                // New column 'test'
> ```

8.  **If protocol changed (*add a setting, etc*), add a new `Feature` flag**

Newer clients need to detect when they're on an older core to disable the
feature.  Use the `Feature` enum in [`quassel.h`][file-h-quassel].

In client-side code, test for a feature with...

```cpp
if (Client::isCoreFeatureEnabled(Quassel::Feature::FeatureName)) { ... }
```

9.  **Test everything!  Upgrade, migrate, new setups, new client/old core,
old client/new core, etc.**

More specifically, you should try the following combinations, especially if you
change the protocol.  Check if any data or settings get lost or corrupted, or
if anything unusual shows up in the log.  Restart the Quassel core and client,
to check that data persists.

---

**Linux/macOS**

*Fresh configuration (initialize the database and settings from scratch)*

Test conditions  | Result  | Remarks
-----------------|---------|-------------
SQLite, new core, new client | ❓ **TODO** |
SQLite, new core, old client | ❓ **TODO** |
SQLite, old core, new client | ❓ **TODO** |
SQLite, new monolithic | ❓ **TODO** |
Postgres, new core, new client | ❓ **TODO** |
Postgres, new core, old client | ❓ **TODO** |
Postgres, old core, new client | ❓ **TODO** |

*Migration (set up SQLite, then `--select-backend PostgreSQL`)*

Test conditions  | Result  | Remarks
-----------------|---------|-------------
SQLite → Postgres, new core, new client | ❓ **TODO** |

*Upgrading existing (set up a copy from `master`, then upgrade to your branch)*

Test conditions  | Result  | Remarks
-----------------|---------|-------------
SQLite, old → new core | ❓ **TODO** |
SQLite, old monolithic → new monolithic | ❓ **TODO** |
Postgres, old → new core | ❓ **TODO** |

**Windows**

*Fresh configuration (initialize the database and settings from scratch)*

Test conditions  | Result  | Remarks
-----------------|---------|-------------
SQLite, new core, new client | ❓ **TODO** |
SQLite, new core, old client | ❓ **TODO** |
SQLite, old core, new client | ❓ **TODO** |
SQLite, new monolithic | ❓ **TODO** |

*Upgrading existing (set up a copy from `master`, then upgrade to your branch)*

Test conditions  | Result  | Remarks
-----------------|---------|-------------
SQLite, old → new core | ❓ **TODO** |
SQLite, old monolithic → new monolithic | ❓ **TODO** |

*If someone figures out how Postgres runs on Windows with Quassel, please
update this...*

---

Yes, this looks excessive, and maybe it is.  But it's easy to overlook
some minor typo that breaks the client/core for a certain configuration.
People may get unhappy and rioting might happen in the streets.  And we don't
want that, do we?

Thank you for reading this guide and good luck with your changes!

[postgres-home]: https://www.postgresql.org/
[sqlite-home]: https://sqlite.org/
[docs-wiki-db-manage]: https://bugs.quassel-irc.org/projects/quassel-irc/wiki/Database_Management
[file-cpp-abstract]: ../abstractsqlstorage.cpp
[file-h-abstract]: ../abstractsqlstorage.h
[file-cpp-postgres]: ../postgresqlstorage.cpp
[file-cpp-sqlite]: ../sqlitestorage.cpp
[file-sh-updateresource]: updateSQLResource.sh
[file-sh-upgradeschema]: upgradeSchema.sh
[file-qrc-sql]: ../sql.qrc
[file-h-quassel]: ../../common/quassel.h

