Avoiding disaster may seem like a basic requirement for most jobs, but in cybersecurity that's easier said than done. 

With malicious actors ready to attack at the first sign of a glitch or misstep, the enterprise keeps cybersecurity professionals on their toes, ready to defend against any attacker. If a mistake happens, it can be catastrophic to the business. 

Shining a spotlight on the oft-overlooked wins that happen each day in cybersecurity departments, Cybersecurity Dive asked security executives to share an almost-disaster that they helped the company avoid. 

(The comments below have been lightly edited for length and clarity.)

Mike Hamilton, CISO at CI Security and former CISO of Seattle

---

"When I told the FBI this I got the sideways wink and, 'We can't confirm that.'"

Mike Hamilton

CISO of CI Security and former CISO of Seattle

We caught some 'intelligence briefs' coming in through e-mail that were good fakes, but booby-trapped with malware. In looking at the distribution of recipients, noticed that most were in Seattle City Light, the city's energy utilities and one of the largest municipals. I spoke with the CIO of SCL to determine what roles the traps were being sent to, and most were power marketers.

Power marketers are the people that negotiate trades of energy over the grid. It has to be priced, transmission lines need to be scheduled, etc. 

The address used to email the supervisor was an instant message address, which had been aliased to email. Now how did bad guys know the IM address of the supervisor of SCL power marketers? (Aside: back then, and I'm not making this up, trades of energy over the grid were negotiated using AOL Instant Messenger.)

After much thought we figured it out. Some time before, China hacked Google. It was a big deal… all over the news. Google makes their own power, and thus has power marketers — presumably with the IM contacts for others. China lifted this contact info and used it to gain access to West Coast energy generation. When I told the FBI this I got the sideways wink and, "we can't confirm that."

Dawn Cappelli, VP of global security and CISO at Rockwell Automation

---

"We have the satisfaction of stating with confidence that our security controls saved our IP from being shared with a supplier whose customer information ended up being breached."

Dawn Cappelli

VP and CISO at Rockwell Automation

Our Third Party Risk Program used to focus on IT: primarily engineering service providers as well as any product or service with access to our network or confidential information. However, over the past three years we increased the scope of our program to include product supply chain (including software supply chain), manufacturing supply chain, and critical partners.

As a result, we recently conducted a Third Party Risk Assessment (TPRA) for a manufacturing company that we were considering using to produce product prototypes. In order to perform that task they would have required access to confidential Intellectual Property (IP) for manufacturing those prototypes.

Our TPRA revealed that the company lacked the required security posture for companies with access to our IP, and as a result we awarded the contract to another company. A few months later, the company we rejected experienced a data breach, and some of their customer information was stolen.

One of the difficult things about security is calculating return on investment (ROI). Our goal is to prevent a cyberattack, but it is usually difficult to predict what would have happened if we had not invested in security controls. In this case we have the satisfaction of stating with confidence that our security controls saved our IP from being shared with a supplier whose customer information ended up being breached.

In addition, the supplier reached out to us after the breach for suggestions on how to secure their practices internally, and we met with them to discuss how they can mitigate their risks. It is crucially important that we — the security community — work together to raise the security posture of our entire ecosystem, especially the small and medium companies in the global supply chain.

Tim Bandos, CISO and VP of managed security services at Digital Guardian

---

"Key lessons learned from this breach are to acquire visibility across all endpoints, both existing and newly installed, to ensure a cyberattack can be prevented — or at least detected — as soon as possible."

Tim Bandos

CISO and VP of managed security services at Digital Guardian

One of our managed detection and response (MDR) customers was breached by an external ransomware group via an insecure remote desktop server that the customer just stood up within their organization. Unfortunately, our endpoint agent wasn't installed, so we lacked visibility into the initial compromise of the system.

That same day though, the threat actor moved laterally to a set of domain controllers in order to steal usernames and passwords. They had also collected information pertaining to the customer's disaster recovery procedures and exfiltrated them to an external site.

All of this activity was detected and we immediately responded by determining the scope of devices accessed by the group and any additional indicators left behind. This type of behavior is common for ransomware operators to initially gather intel on their target, steal information, and return to deploy their malware. Fortunately, we saved this customer by neutralizing the attack and preventing any ransomware from being installed and encrypting their data.

Key lessons learned from this breach are to acquire visibility across all endpoints, both existing and newly installed, to ensure a cyberattack can be prevented — or at least detected — as soon as possible. If a remote desktop server is required to be internet-facing, additional measures should be taken to harden the device such as enforcing complex passwords, enabling two-factor authentication, and changing the default listening port 3389.

This customer narrowly escaped a ransomware campaign that would've surely caused great harm to their business.

Kristen Sanders, CISO at the Albuquerque Bernalillo County Water Utility Authority

---

"I ran a few tests to confirm, and indeed the public Wi-Fi had access to our internal network. I couldn't believe it!"

Kristen Sanders

CISO at the Albuquerque Water Utility Authority

---

I began my career as a network engineer, but I always had a passion for security. I loved to poke around and test to ensure we were leveraging our technology to the best of our abilities. I was running some tests on a free public Wi-Fi network that my former employer was providing. I wanted to ensure that our splash pages were functioning correctly and we were providing acceptable bandwidth for our customers.
 
All of my tests were going very smoothly. Before I disconnected, I decided I would try to reach some of our internal resources. I knew this wouldn't work because this was all tested during the initial deployment. However, I had this feeling that I just needed to do it. I sent a ping to one of our internal servers, and it was successful. I ran a few tests to confirm, and indeed the public Wi-Fi had access to our internal network. I couldn't believe it!

Thankfully we were running centrally managed Cisco wireless APs. I was able to disable the Public Wi-Fi within a minute as opposed to touching each individual device. I alerted management immediately of what I had found and put together an official email with an action plan for remediation.
 
I quickly learned that day the value of constantly checking for vulnerabilities. Yes, those same tests had failed previously. However, someone had made a network configuration change since that time. We need to continuously monitor and validate our controls. This was the result of a simple misconfiguration that could have resulted in a major data breach.

It's also crucial that your employees feel empowered to make immediate decisions in these types of situations. I knew that disabling the public Wi-Fi could potentially upset executive management, but I also knew that my manager would understand and support my decision.