“How do you manage software updates so that your players are protected while minimizing the chance of downtime due to complications?”
Managing software updates while protecting media players is not easy. PATTISON Outdoor has more than 4,000 media players of varying ages in the field. Some are a day old, and others are 15 years in the field happily running away. Updates represent a tremendous requirement to manage “legacy” issues when deploying updates. While we use certain automation tools to deploy the patches and updates, there’s a very structured and refined testing process that happens BEFORE the automated push. As we learned many years ago, three days before Christmas, a hastily pushed software update no matter how small it appears, can brick your media players across the country… costing thousands in service calls.
So before we ‘launch’ the update, we employ a technique of targeting specific media players to test in the field. A series of test rings, each a little larger, that eventually contain each ‘flavor’ of hardware and software we have out there. Each ring is tested, and verified. If a player fails to respond, we stop and start over or limit the deployment.
What this process costs in time (this takes a solid day to test, if not more), it saves in mistakes. We sacrifice speed for (near) perfection every day. There are always edge cases that go sideways, so an informed NOC (Network Operations Center) on Slack doesn’t hurt to have on the ready either. In the odd case where an emergency push has to go out, and time doesn’t allow the full range of testing, it requires three keys turned simultaneously – by the Chief Software Architect, Senior Software Engineer and finally, myself (buck stops here).
It’s a process under constant strain and review, but after 15 years, it has become something we’re pretty good at.