Techly Business Continuity and Disaster Recovery Policy

Purpose

This policy sets out how Techly Operations Pty Ltd (Techly) protects Pyron's data against loss and restores the service after a failure, and it is plain about what Techly cannot yet recover from quickly.

Scope

Pyron's production service: its database, stored files, infrastructure and configuration, and the people who run it.

What is backed up

• The database. Backed up continuously by AWS, and restorable to any moment in the last 35 days. The backups are encrypted and held in the AWS Melbourne region. The database is protected against deletion, and a final snapshot is taken if it is ever deleted.
• Stored files. When a file is replaced or deleted, its earlier version is kept for 30 days.
• Infrastructure. Every cloud resource, network rule, alert and access assignment is defined in code, so it can be rebuilt from Techly's source repository.
• Software. Each release is a built image in Techly's container registry, and can be rebuilt from its source.

Recovery objectives

• Recovery point: about five minutes. A restore can recover the database to within about five minutes of a failure, because its transaction logs are backed up that often.
• Recovery time: measured at each restore test. Until the first restore test has run, Techly does not know how long a full restore takes, and makes no promise about it. There is no service level agreement.

Restore testing

Every quarter, the Technical Director restores the production database from its backups into a new, separate database, checks that it is complete and readable, measures how long it took, and then deletes it. The result and the time taken are recorded, and the recovery time above is updated from it. A failed test is treated as an incident.

The first restore test is planned, with a scripted procedure, so that anyone following the runbook gets the same result.

Known limits

• One availability zone. The production database runs in a single availability zone, with no standby. If that zone fails, Pyron is unavailable until the database is restored into another zone. Techly has accepted this at its current scale, and will add a standby when a customer's agreement calls for one.
• One region. Backups are held in Melbourne only. Copying them to the AWS Sydney region is planned. Until then, the loss of the whole Melbourne region would stop Pyron until that region recovers.
• One engineer. Pyron is operated by one engineer. Its infrastructure, runbooks and procedures are written down in Techly's source repository so that another engineer could take over.

When something fails

1. The failure is handled as an incident under the Techly Incident Management Policy: an alert pages the on-call engineer, who follows the runbook for that alert.
2. The engineer restores the service, fixing forward through the normal change process, or restoring data from backup if data was lost or damaged.
3. Affected customers are told by email what happened, what was affected and when the service is back.
4. If a restore was needed, the restored data is checked before the service is reopened.

Who is responsible

• The Technical Director owns this policy, runs the restore tests and leads recovery.
• The Managing Director keeps customers informed during a long outage when the Technical Director is restoring the service.

Review

The Technical Director reviews this policy every year, after every restore test that changes the recovery time, and after any incident that needed a restore.