High Availability and Scaling

Cluster options

Single-node cluster: A single-node cluster only has one node which is called the primary node. This node accepts customer connections and performs read/write operations. This is a single point of truth as well as a single point of failure.

Multi-node cluster: In addition to the primary node, this cluster contains standby nodes that can be promoted to primary if the current primary fails. The nodes are spread across availability zones. Currently, we use warm standby nodes, which means they don't serve read requests. Hot standby functionality (when the nodes can serve read requests) might be added in the future.

Database scaling

Existing clusters can be scaled in two ways: horizontal and vertical.

Horizontal scaling is defined as configuring the number of instances that run in parallel. The number of nodes can be increased or decreased in a cluster.

Scaling up the number of instances does not cause a disruption. However, decreasing may cause a switch over, if the current primary node is removed.

Note: This method of scaling is used to provide high availability. It will not increase performance.

Vertical scaling refers to configuring the size of the individual instances. This is used if you want to process more data and queries. You can change the number of cores and the size of memory to have the configuration that you need. Each instance is maintained on a dedicated node. In the event of scaling up or down, a new node will be created for each instance.

Once the new node becomes available, the server will switch from the old node to the new node. The old node is then removed. This process is executed sequentially if you have multiple nodes. We will always replace the standby first and then the primary. This means that there is only one switchover.

During the switch, if you are connected to the DB with an application, the connection will be terminated. All ongoing queries will be aborted. Inevitably, there will be some disruption. It is therefore recommended that the scaling is performed outside of peak times.

You can also increase the size of storage. However, it is not possible to reduce the size of the storage, nor can you change the type of storage. Increasing the size is done on-the-fly and causes no disruption.

Replication modes

The synchronization_mode determines how transactions are replicated between multiple nodes before a transaction is confirmed to the client. IONOS DBaaS supports three modes of replication: Asynchronous (default), Synchronous and Strict Synchronous. In either mode the transaction is first committed on the leader and then replicated to the standby node(s).

Asynchronous replication does not wait for the standby before confirming a transaction back to the user. Transactions are confirmed to the client after being written to disk on the primary node. Replication takes place in the background. In asynchronous mode the cluster is allowed to lose some committed (not yet replicated) transactions during a failover to ensure availability.

The benefit of asynchronous replication is the lower latency. The downside is that recent transactions might be lost if standby is promoted to leader. The lag between the leader and standby tends to be a few milliseconds.

Caution: Data loss might happen if the server crashes and the data has not been replicated yet.

Synchronous replication ensures that a transaction is committed to at least one standby before confirming the transaction back to the client. This standby is known as synchronous standby. If the primary node experiences a failure then only a synchronous standby can take over as primary. This ensures that committed transactions are not lost during a failover. If the synchronous standby fails and there is another standby available then the role of the synchronous standby changes to the latter. If no standby is available then the primary can continue in standalone mode. In standalone mode the primary role cannot change until at least one standby has caught up (regained the role of synchronous standby). Latency is generally higher than with asynchronous replication, but no data is lost during a failover.

At any time there will be at most one synchronous standby. If the synchronous standby fails then another healthy standby is automatically selected as the synchronous standby.

Caution: Turning on non-strict synchronous replication does not guarantee multi node durability of commits under all circumstances. When no suitable standby is available, the primary node will still accept writes, but does not guarantee their replication.

Strict synchronous replication is the same as synchronous replication with the exception that standalone mode is not permitted. This mode will prevent PostgreSQL from switching off the synchronous replication on the primary when no synchronous standby candidates are available. If no standby is available, no writes will be accepted anymore, so this mode sacrifices availability for replicated durability.

If replication mode is set to synchronous (either strict or non-strict) then data loss cannot occur during failovers (e.g. node failures). The benefit of strict replication is that data is not lost in case of a storage failure of the primary node and a simultaneous failure of all standby nodes.

Synchronization mode considerations

Please note that synchronization modes can impact DBaaS in several ways:

Aspect	Asynchronous	Synchronous
primary failure	A healthy standby will be promoted if the primary node becomes unavailable.	Only standby nodes that contain all confirmed transactions can be promoted.
Standby failure	No effect on primary. Standby catches up once it is back online.	In strict mode at least one standby must be available to accept write requests. In non-strict mode the primary continues as standalone. There is a short delay in transaction processing if the synchronous standby changes.
Consistency model	Strongly consistent (expect for lost data.)	Strongly consistent (expect for lost data.)
Data loss during failover	Non-replicated data is lost.	Not possible.
Data loss during primary storage failure	Non-replicated data is lost.	Non-replicated data is lost in standalone mode.
Latency	Limited by the performance of the primary.	Limited by the performance of the primary, the synchronous standby and the latency between them (usually below 1ms).

The performance penalty of synchronous over asynchronous replication depends on the workload. The primary handles transactions the same way in all replications modes, with the exception of COMMIT statements (incl. implicit transactions). When synchronous replication is enabled, the commit can only be confirmed to the client once it is replicated. Thus, there is a constant latency overhead for each transaction, independent of the transaction's size and duration.

Changing the commit guarantees per transaction

By default, the replication mode of the database cluster determines the guarantees of a committed transaction. However, some workloads might have very diverse requirements regarding accepted data loss vs performance. To address this need, commit guarantees can be changed per transaction. See synchronous_commit (PostgreSQL documentation) for details.

Caution: You cannot enforce a synchronous commit when the cluster is configured to use asynchronous replication. Without a synchronous standby any setting higher than local is equivalent to local, which doesn't wait for replication to complete. Instead, you can configure your cluster to use synchronous replication and choose synchronous_commit=local whenever data loss is acceptable.