On Tuesday 16th January 2018, the leaders in ICS security met in Miami at S4x18 to discuss and share ideas about the future of the industry. This year, the organizers have put together an exciting new project called the ICS Detection Challenge. ICS anomaly detection is a relatively new field, and one of the most common complaints among stakeholders is the lack of opportunities to compare the different solutions. This challenge addresses that concern. As stated by S4, “The ICS Detection Challenge was created to provide an objective test of the growing class of passive ICS detection solutions in the market.”
S4 Challenge Phase I: Asset Inventory Day
Since SecurityMatters is completely committed to the cyber resilience of its customers, we decided to take on the challenge and compete for the top spot among only four companies brave enough to do so. Our challenge team consists of some of the best and brightest minds in the industry, including one of our customers from the oil & gas industry. On the first day of the contest, they scored extremely well in Phase I, which was devoted to asset identification.
Contestants taking on this challenge were provided with pcap files from span ports on multiple ICS switches and tasked with documenting the system as extensively as possible. These pcap files came from an actual industrial control system in the oil/gas sector and, although anonymized, were a representative, real world test, which was especially challenging considering the limited time and information available to the contestants.
Cyber asset identification was at various specificity levels, with extra points given for additional specificity. There was also a time bonus given for correctly identifying cyber assets in a short time frame to acknowledge the difference between automated analysis and human analysis. The scoring was conceived to balance breadth of view with accuracy.
At the end of the first day, the asset inventory challenge has shown that all products involved have strong core capabilities, but obviously none supports all vendors and equipment.
Apart from our strong finish, the first day of this event has demonstrated that there are three key elements that set SecurityMatters apart from other ICS security vendors:
- Customers are the core of our business. Sure, different technology can provide similar results, like in this case, but it’s the human factor that makes the difference. The amazing relationship we have with our customers was on display at this event, where our team of engineers worked shoulder to shoulder with one of our new customers and enjoyed their shared work (and fun!) throughout the day.
- SilentDefense performed very well in quickly identifying the types and manufacturer of a vast range of devices. Our solution demonstrated a very good balance of speed, accuracy and complete situational awareness, and our ability to integrate vulnerability analysis has been praised as one key differentiator of SilentDefense.
- With so many vendors and protocols out there, customization capabilities and team skills are fundamental in this market, as every ICS is unique and must be treated as such. SecurityMatters’ team is the absolute leader in this, and as experienced by many of our customers, the product can extend and adjust to their systems and protocols in a matter of days. Our engine not only has the capability to develop brand new, customer-specific detection scenarios, but also to extend our built-in asset inventory capabilities and protocol support on the fly.
That’s all for now- we are excited to take on the second part of the challenge and good luck to all the participants!
S4 Challenge Phase II: Anomaly Detection Day
This year’s S4 conference, taking place as we write in Miami, will be remembered as the event where the first real benchmark of ICS cyber security solutions took place. The goal of the “ICS Detection Challenge” was to provide asset owners with objective information of how helpful advanced network monitoring solutions can be to support them in their daily work and to detect cyber threats and intrusions. The indisputable result is that these solutions not only proved themselves able to save days, if not months, of effort for industrial operators when performing comprehensive asset inventory, but also that they can detect the most advanced cyber attacks and operational threats to these networks.
Before going into details about the results and lessons learned from the challenge, we would like to express our gratitude to the event organizers – Dale Peterson, Eric Byres and aeSolutions. We have lived a couple of days of suspense and fun, and our team enjoyed every single bit of this. Certainly, there can be discussions about how to make the next edition even better, but as the first event of its kind, this has exceeded every expectation. So, kudos to S4 for organizing this groundbreaking event and allowing us the opportunity to challenge ourselves.
Now, back to the challenge. Phase II of the “ICS Detection Challenge” was organized similarly to the first one: at 8:30AM, the organizers started replaying PCAPs containing traffic recorded from multiple industrial networks, which the participants had to analyze through their real-time network monitoring solutions. The goal of this Phase, however, was completely different from that of Phase I. Rather than extracting asset information, the participants had to leverage their solutions to detect anomalous and undesired behavior. In fact, the PCAP files contained a large variety of penetration attempts, malware, illegitimate process modifications, as well as simulated operator mistakes. Teams could win points by correctly identifying the anomalous activity, with additional points being earned based on specificity and timeliness of the results, as an answer form was collected every hour.
After a long night of discussion and preparation for Phase II, our team woke up full of energy and excited by the challenge ahead. Phase II went by in the blink of an eye, and while the rest of the team across the US and Europe lived the thrill with them, our challenge team in Miami was having fun at the challenge desk, as they uncovered one threat after the other. When the results were announced a few hours later, SilentDefense had an incredibly strong finish, leading in 6 of the 11 categories assessed and standing out to the judges as the best security tool.
“It has been two intense days!”, says Christiaan Schade, Head of Development at SecurityMatters, “The organizers have been very creative in what they put in these PCAPs. Luckily, we have SilentDefense on our side, which made things much simpler. When the organization asked us to end one and half hours before time, we handed in our results without any problem, because we detected a vast range of anomalies.” Eric, an ICS/SCADA Network and Security Engineer at a large US based oil & gas company who joined our team for the challenge confirms, “We have detected Stuxnet, Havex, several reconnaissance attempts, unusual write operations, flooding on ModbusRTU and ModbusTCP. We even spotted a certificate with 10 years validity, which would violate any company’s security policy. It has been great fun!”
Aside from the fun of the game, it is important to highlight what the results of this challenge mean for critical infrastructure and manufacturing operators worldwide. We think that it is very important to set the right context and build awareness to stress why passive network monitoring technology would very beneficial for critical infrastructure and manufacturing organizations. The results of Phase I, focused on automated asset inventory, have proven that today’s technologies are sufficiently mature to save months of human effort in identifying network assets and classifying them based on their risk and exposure. The anomaly detection phase, Phase II, has shown how important it is to complement traditional security technology, such as firewalls, with something more advanced that can detect threats at their earliest stage, along with lateral movement and exfiltration attempts typical of malware and zero-days. It goes without saying that network monitoring technology would have enabled early identification of the most famous malware in recent years, from Stuxnet down to the latest threat, Triton, before it crippled the targeted organization. These two days are absolute proof that network monitoring technologies would help to create a safer and more resilient environment.
Finally, we would like to conclude with an important consideration. Although this was an extremely realistic simulation, it was focused on a very limited subset of capabilities that a product should have. There is a big set of equally important characteristics and features of a product that users need in their daily operation that could not be tested in such a short exercise, like stability, reliability and completeness of the solution. This is what makes a product usable and likable in a real deployment, especially if large scale. A major pitfall of “lab” PoCs and evaluations is that these characteristics are overlooked, and they will only surface when the real deployment occurs. For this reason, it is important that end users who want to evaluate and compare multiple ICS network monitoring solutions dedicate the right amount of time and resources to these tests, possibly simulating a real deployment in their environment. This ensures a real, comprehensive evaluation and confirms that the chosen solution is really the best fit for their needs, with no unpleasant surprises later.
To learn more about SilentDefense and its capabilities, check out our Solution Brief.
