What Project Memoria Foretold about TCP/IP Security and Supply Chain Vulnerabilities
Project Memoria was the largest study about the security of TCP/IP stacks, conducted by Vedere Labs and partners in the cybersecurity industry. It started from a collaboration with JSOF to understand the impact of Ripple20 and led to the discovery of almost 100 vulnerabilities in 14 TCP/IP stacks, divided into five phases: AMNESIA:33, NUMBER:JACK, NAME:WRECK, INFRA:HALT and NUCLEUS:13.
Last November, a year after the AMNESIA:33 disclosure, we concluded our research and summarized the lessons learned from this endeavor, including insights on legacy software, silent patching, predictability of vulnerabilities and vendor responsiveness. Two years since that initial disclosure, it’s clear that Project Memoria is even more relevant today. It foreshadowed the persistent problems the industry is facing with supply chain vulnerabilities and why the recommended mitigation strategies provided with each disclosure can’t be ignored.
Supply chain vulnerabilities: the good, the bad and the ugly
Since TCP/IP stacks are important supply chain components used by many software and device vendors, it’s no surprise that the vulnerabilities found during Project Memoria ended up affecting hundreds of different products, from network switches to VoIP phones to patient monitors to gas turbines. As researchers, our goal in disclosing vulnerabilities is not just to bring them to light but to work with CISA, vendors and security professionals to ensure they cannot by exploited. Unfortunately, when we consider some of the longer-term effects from three points of view, it may look more like a spaghetti western:
- The good: Project Memoria has led not only to fixes of individual issues but also to a body of work that provides guidance on how to avoid repeating the same mistakes. This body of work continues to influence further research.
- The bad: Some of these vulnerabilities are now exploited by threat actors; vendor response continues to be slow and, in many cases, vague.
- The ugly: The number of exposed devices running the vulnerable services disclosed by Project Memoria has decreased in some cases but remained stable or even increased in others, which shows that more attention must be put into network segmentation efforts.
The good – long-term positive effects of vulnerability research
Vulnerability research is not only about finding specific problems and fixing them in specific products. More importantly, it’s about understanding why things go wrong and how to avoid repeating mistakes in the future. Two phases of Project Memoria in particular have been very successful at achieving this long-term positive effect: AMNESIA:33 and NAME:WRECK.
AMNESIA:33 has been used as the motivation and a testing suite for several academic developments, such as a set of vulnerability anti-patterns in communication stacks, a tool for easier firmware patching in real-time embedded devices and a new design for resilient IoT devices based on virtualization. These efforts help pave the way for devices with fewer vulnerabilities, easier patching procedures and more secure designs.
NAME:WRECK went beyond the academic with two direct contributions for software developers:
- The set of anti-patterns we distilled for DNS clients became an informational RFC. A request for comments (RFC) is an official peer-reviewed document published by the IETF, the internet standards-setting body, that provides guidance to those implementing networking applications. RFC9267 (“Common Implementation Anti-Patterns Related to DNS RR Processing”) details our findings so that developers can avoid the vulnerabilities we identified in the future.
- The Chromium project, which provides the code for the Chrome browser, adopted these same anti-patterns as part of their testing suite, to make sure that the browser is not vulnerable to those issues as it continues to evolve. Hopefully, that means that the type of flaw identified with NAME:WRECK will not happen again.
The bad – exploits and vendor response
In November 2021, CISA published a binding operational directive to reduce the risk of known exploited vulnerabilities to U.S. federal agencies. The directive established the Known Exploited Vulnerabilities (KEV) catalog, which is constantly updated with new vulnerabilities and a deadline for agencies to patch them. The reasoning behind this directive is that less than 4% of CVEs are actually exploited, so it is important to patch the ones that are known to have been exploited quickly.
On March 3, 2022, the first vulnerability on an embedded TCP/IP stack was added to the KEV catalog: CVE-2020-11899, part of Ripple20, with a two-week deadline for patching. This shows that attackers are willing to exploit these vulnerabilities in the real world, but it also presents a challenge: how to patch all affected devices?
A successful patching effort depends, at a minimum, on vendors acknowledging that their devices are vulnerable and providing patches. However, when we concluded Project Memoria, only 81 out of 422 vendors potentially affected by our findings had published any public advisory, either by acknowledging the vulnerabilities or showing how they were not affected.
In the year since, 14 more vendors have published advisories. That brings the total number of advisories we track to 95, which is still only 22.5% of the 422 identified vendors. If this trend continued, it would take more than 20 years for all vendors to publish some response. That would be hopeful: the number of responses per year is very likely to decrease because the vendors who were more likely to assess their products have already done so.
In a race for last place, two advisories stand out:
- Juniper published an advisory about CVE-2020-7461 (which originally affects the FreeBSD DHCP client) on April 13, 366 days after the publication of NAME:WRECK and 588 days after the original publication of the FreeBSD advisory. This is the new record for slowest response in Project Memoria, surpassing Schneider Electric’s response to AMNESIA:33, which took 308 days.
- Nihon Kohden on August 5 published a letter stating that the Nucleus real-time operating system (RTOS) is used by “some Nihon Kohden devices.” The letter came 269 days after the public disclosure of NUCLEUS:13 and 319 days after we first shared the information with the vendor. Still, the letter does not specify which devices are vulnerable, which vulnerabilities apply to those devices or what customers should do to mitigate them. We are unaware if the manufacturer has provided further guidance in private communication to their customers, but this type of public statement simply obscures the risk that organizations are exposed to, which is totally contrary to the premises of coordinated vulnerability disclosure.
The ugly – device exposure
When we published the last phase of Project Memoria, NUCLEUS:13, we noticed that the number of devices exposed on the internet running the Nucleus FTP server and RTOS had decreased by 13% and 25%, respectively, when compared to the release of NAME:WRECK, six months earlier.
This was a good sign. Often the devices running this type of operating system are safety-critical and should not be directly accessible on the open internet. We hypothesized that the decrease was a positive result of our research.
Using the same queries on the Shodan search engine in August 2022, one year after first noticing the decline, we still see a sharp decrease of exposed devices running the Nucleus FTP server, as shown in Figure 1. However, the number of devices running the Nucleus RTOS seems to have stabilized at around 1100-1200, as shown in Figure 2, which is still less than when we started our research.
On the other hand, the number of devices running NicheStack – the stack found vulnerable in INFRA:HALT – has increased by almost 50% in the same two-year time frame observed for Nucleus, as shown in Figure 3.
Figure 1 – Devices running the Nucleus FTP Server (Shodan query: “220 Nucleus FTP”)
Figure 2 – Devices running the Nucleus RTOS (Shodan query: “Operating System: Nucleus PLUS”)
Figure 3 – Devices running NicheStack (Shodan query: “interniche”)
These mixed results show that it is difficult to know if vulnerability research really had an impact in reducing the number of exposed devices. They also show that there is still a lot of work to do in terms of reducing the risk of IoT devices to organizations.
Conclusion – supply chain vulnerabilities are here to stay and mitigation must go beyond patching
Project Memoria came at a time when initiatives for understanding the complexity of software supply chains and how to tame that complexity with tools such as software bills of materials (SBOMs) and automated vulnerability disclosure were starting to gain traction.
Right after we concluded the project, in December 2021, the Log4Shell vulnerabilities were disclosed by a researcher from Alibaba Cloud, which brought the topic of supply chain vulnerabilities to the forefront of cybersecurity discussions. Log4Shell is a set of vulnerabilities affecting a logging library used by thousands of vendors and figured among the most exploited in 2022. It was declared endemic by a newly formed Cyber Safety Review Board, meaning that it will remain an important attack vector for the foreseeable future.
Similar to Log4j, the vulnerabilities in Project Memoria will probably remain an unsolved problem for a long time, due to two issues we outlined in this post:
- Often there are no patches available for vulnerable devices because vendors take a very long time to publish them.
- Vulnerable devices continue to be exposed directly to the internet.
But even before patching and reducing exposure, simply identifying vulnerable assets is hard enough without the appropriate tools, since devices don’t advertise the TCP/IP stack they use. Case in point, the Forescout Frontline team continues to find vulnerable devices during threat hunting engagements that would be much harder to pinpoint without using the Forescout Continuum Platform. In two recent engagements, the team identified five devices vulnerable to NUCLEUS:13 and almost 500 vulnerable to Ripple20.
For individual organizations, mitigation measures such as device visibility, segmentation and exploit detection help with supply-chain vulnerabilities. However, those measures require mature cybersecurity tools that can handle a growing and changing digital terrain instead of focusing on individual parts of the network. The same vulnerabilities in Project Memoria affected devices as diverse as medical equipment and building automation controllers. A security tool that has visibility on only one of those groups could allow for large blind spots on today’s heterogeneous networks. At the same time, one of the most important takeaways from the project is that simply identifying vulnerable devices is not enough if no further action can be taken. Organizations must adopt security tools that allow for detection of threats and automated, orchestrated response.