“How do you manage software updates so that your players are protected while minimizing the chance of downtime due to complications?”
The management of software updates needs to start very early in the development cycle. Planning is everything to ensure a smooth update. Key considerations include:
- Prioritization of updates – Carefully prioritize updates needed into several buckets with differing importance. Security updates are deemed critical and high importance. New feature deployments need to be prioritized using various considerations including customer impact, value, risk etc. Once prioritized, these updates are typically implemented in a CI/CD fashion. Sending these updates to the field is measured and packaged to reduce impact.
- Test, Test and Test – Employ massive test grids to test out all field configurations and update scenarios. An important test case is back-out testing in case of failures. Being able to understand every failure mode and testing the appropriate back-out process is key. It is also important to understand how to collect logs from such failures when it happens in the field.
- Dog-fooding and Canary testing – Once the package is ready for deployment, select in-office and in-field devices that are of lower priority to test out the upgrades. This is typically 2 to 5 percent of your in-field devices. It is a good idea to include local markets where support is readily available in this initial cycle. Monitoring this initial release is key to the success of this upgrade.
- Reporting – There is a need to ensure that we have ample reporting defined to track progress and anomalies. Sometimes, having a war-room setup for this upgrade project with a dashboard (live) is a good idea.
- Upgrade away…