The stadium lights ripped the darkness over an empty field.
They weren’t supposed to be on. The lights at Princeton University’s stadium, recently upgraded, should have followed an automated cycle, reducing the need for human oversight.
Instead, the lights went to war.
That’s how Jay Dominick, the vice president for information technology and the chief information officer for the Office of the Vice President for Information Technology at Princeton University, described to me what happened when I followed-up with him after he spoke at the Conference on Security and Privacy for the Internet of Things, held Oct. 16, 2016 at Princeton University.
The lights weren’t entirely replaced, and therein lies the problem — and the lessons for any larger enterprise-level project that brings analog projects into the age of the Internet of Things.
The lights flipped on well after midnight because, as Dominick explains, the technology behind the bulbs couldn’t communicate. It’s not something anyone could have predicted or tested for, like they might a software upgrade, before going live.
“The network guys run out there and run disparate packets and say, ‘Yep, the network works, the lights tested and work’,” he says. “And at 3 a.m. the lights go on, and they start whole process again.
“Eventually, through a rigorous process of elimination, you wind up figuring out what you think might have happened, and then it’s the argument about who’s going to fix it.
“We wound up creating a new network for the new lights,” he says. “The new lights liked to talk to each other quite a bit. They were very chatty. And when they would get to talking, the old lights couldn’t process the packets fast enough, so they failed into some obscure state, and that failure would put the old lights in a failure which turned them on.”
“It was essentially a DDOS [Distributed Denial of Service] attack,” he says — just within the same system.
The broader lesson here, he says, is that you can’t upgrade an enterprise Internet of Things system as it were an iPhone — expecting all the parts to run perfectly, out of the box.
“When the next generation comes out as an upgrade to the operating system, now we have a change management process that might not have been familiar to the operational tech world — how does the new software interact with the old software, how do the new lights interact with the old lights?
“There’s just a lot of friction where we’re seeing with these large-scale electromechanical, formerly analog systems now all automated, on a network, and suddenly there are IT people and operations technology people trying to figure out how it all works together, and sometimes it just doesn’t.”
Princeton faced a second problem with a fire alarm system that failed across campus. Today, campus fire alarms have panels that report their status via fiber optic connections to a central controller. The buildings are supposed to ping the central system frequently — if they don’t, the system assumes that building alarm is broken.
Every time communication failed, the university would have to send a person with a walkie talkie to monitor the building while the staff figured out why communication halted, Dominick says.
Dominick cites four key lesson from the light war:
1. Change management has to change. Take the fire alarm example, Dominick says. “In the analog world, if you have continuity on the table, you’re good. Things were largely electromechanical devices that either worked or didn’t work… Now, as we begin to put processors with software and communication stacks at both ends, you tend to get into typical IT problems, which is how you engage in change management. How do you do version control between different parts of the software stack that are going in at different parts, and how do you manage that change?”
2. IT leaders and operational experts need to talk. A lot. “For us, it was taking some recognition by our operational technology friends that they’ve become dependent on IT to get their work done, which unfortunately usually comes up when something goes wrong.
“Now our facility colleagues, our public safety colleagues and IT realize how totally interdependent we are. It would have been nice if that had been a self-realization without having to have been pushed to that realization [when something went wrong] but that’s how it works — you respond to stimuli in the environment.”
3. Talk to vendors. “The intersection of operational technology and information technology is full of friction. This shows up in things like lighting systems that were installed a dozen years ago or so that have a certain set of performance characteristics getting upgraded, and the IT change control not being well understood, either by the vendor or the operational technology folks,” Dominick says.
The light issue resolved, in part, thanks to “very complex discussions with the vendor.” Much of the technology involved in enterprise-level IoT projects likely started as a consumer product or consumer-based technology, Dominick says. Talking to vendors about processes, testing, upgrades and security can help head of issues, Dominick says.
4. The Internet of Things needs a roadmap. There are no guidelines or universally accepted best practices for IoT, Dominick says.
“Whether it’s IEE (Institute of Electrical and Electronics Engineers) or the NIST (National Institute of Standards and Technology), they have got to come together for the rules of the road for how the different products are going to inter-operate. End-to-end security, trust standards, operations — there are some out there working on that,” he says. [Ed. note: NIST released cybersecurity guidelines in mid-November, but it’s not wholesale IoT guidelines: http://www.federaltimes.com/articles/nist-unveils-internet-of-things-cybersecurity-guidance]
So far, technology has quickly outpaced many enterprise agencies’ ability to ensure reliability. Enterprise leadership must serve as their own watchdogs — and ensure the lights don’t go to war.
Editor’s note: For more information, see our report “The Internet of Things (IoT): An Overview – Understanding the Issues and Challenges of a More Connected World“.