As I’m currently developing the ‘next gen’ state-full fuzzing framework @ERNW [called dizzy, to be released soon 😉 ], I will give you an updated set of fuzzing scripts from the ‘old world’.
Some of you will remember the 2008 release of sulley_l2, which was a modified version of the sulley fuzzing framework, enhanced with Layer 2 sending capabilities and a hole bunch of (L2) fuzzing scripts. All the blinking, rebooting, mem-corrupting ciscos gave us some attention. Back from then, we continued to write and use the fuzzing scripts, so the hole collection grew.
There is a common misconception that the sheer amount of data coupled with multiplexed channels (e.g. WDM technology) make successful eavesdropping attacks on high speed Ethernet links – like those connecting data centers – highly unlikely. This is mainly based on the assumption that the amount of resources (e.g. RAM, [sufficiently fast] storage or CPU power) needed to process large files of captured data is a limiting factor. However, to the best of our knowledge, no practical evaluation of these assumptions has so far been performed.
Therefore we conducted some research and started writing a paper (to be released as a technical report shortly) that aims to answer the following questions:
– Can the processing of large amounts of captured data be done “in a feasible way” ?
– How much time and which type of hardware is needed to perform this task?
– Can this be done with readily available tools or is custom code helpful or even required? If so, how should that code operate?
– Can this task be facilitated by means of public cloud services?
We performed a number of tests with files of different sizes and entropy. Tests were both carried out with different sets of dedicated hardware and by means of public cloud services. The paper describes the tools used, the various test setups and, of course, the results. A final section includes some conclusions derived from the insights provided by the test sets.It is assumed that an attacker has already gained access enabling her to eavesdrop on the high speed data link. A detailed description how this can be done can be found e.g. here or here. The focus of our paper is on the subsequent extraction of useful data from the resulting dump file. It is further assumed the collected data is available in standard pcap format.
We’ll summarize some of the stuff in a series of three blog posts, each discussing certain aspects of the overall research task. In the first one we’ll describe the tools and hardware used. In the second we’ll give the results from the test lab with our hardware while the third part describes the tests performed in the (AWS) cloud and provides the conclusions. Furthermore we’ll give a presentation of the results, including a demo (probably the extraction of credit card information from a file with the size 500 GB which roughly equates to a live migration of 16 virtual machines with 32 GB RAM each) at the Infoguard Security Lounge taking place on 8th of June in Zug/Switzerland.
Last but not least before it get’s technical: the majority of the work was performed by Daniel, Hendrik and Matthias. I myself had mostly a “supervisor role” 😉 So kudos to them!
COTS packet analysis tools
A number of tests utilizing available command-line tools (tethereal, tshark, tcpdump and the like) were performed. It turned out that, performance-wise, “classic” tcpdump showed the most promising results. During the following, in-depth testing phase two problems of tcpdump showed up:
– It’s single-threaded so it can’t use multiple processors of a system (for parallel processing). Given the actual bottlenecks to be related with I/O anyway (see below) this was not regarded to be major problem.
– Standard pcap filters do not allow for “keyword search” which somehow limits the attack scenarios (attacker might not be able to search for credit card numbers, user names etc. but would have to perform an IP parameter based search first and then hand over to another tool which might cause an unacceptable delay in the overall analysis process). To address this limitation Daniel wrote a small piece of code that we – not having found an elegant name like Loki so far 😉 – called pcap_extractor.
Pcap_extractor
This is basically the fastest possible implementation of a pcap file reader. It opens a libpcap file handle for the designated input file, applies a libpcap filter to it and loops through all the filter matching packets, writing them to an output pcap file. Contrary to tcpdump and most other libpcap based analysis tools, it provides the possibility to search for a given string inside the matching packets, for example a credit card number or a username. If such a search string is applied, only packets matching the libpcap filter and containing the search string are written to the output file.A call to search a pcap file for iSCSI packets which contain a certain credit card number and write them to the output file would then look like:
The source code of pcap_extractor can be downloaded here.
Identifying the bottleneck(s)
While measuring the performance of multiple pcap analysis tools the profiling of system calls indicated that the tools spend between 85% and 98% of the search time on waiting for I/O. In case of the fastest tool that means 98% of the time the tool does nothing, but waiting for dump data. So the I/O bandwidth turned out to be the major bottleneck in the initial test setups.
Actual lab setup
The final test system was designed to provide as much I/O bandwidth as possible and was composed of:
Intel Core i7-990X Extreme Edition, 6x 3.46 GHz
12GB (3 * 4GB) DDR3 1600MHz, PC3-12800
ASRock X58 Extreme6 S1366 mainboard
4 * Intel 510 Series Elm Crest SSD 250GB
The mainboard and the SSDs were chosen to support SATA3 with a theoretical maximal I/O bandwidth of 6 Gbit/s. FreeBSD was used as operating system.
===
In this post we’ve “prepared the battle ground” (as for the tools and hardware to be used) for the actual testing, in the next one we’ll discuss the results. Stay tuned & have a great day
I can’t help myself. And I fully understand that some of you, dear readers, might get a bit annoyed by always hearing the same tune from our side. This post is, surprise!, about yesterday’s Microsoft Patch Tuesday which – as can be seen here and here – disclosed quite a number of vulnerabilities in various Microsoft components. To make the point evoked in this post’s title I’d like to draw your attention to two particular bulletins, both rated as critical.
The advisory states that “this security update resolves a publicly disclosed vulnerability in Microsoft .NET Framework. The vulnerability could allow remote code execution on a client system if a user views a specially crafted Web page using a Web browser that can run XAML Browser Applications (XBAPs)”.
Looking at the “Workarounds” section, it turns out that the configuration of some specific parameters within Internet Explorer (those are: Loose XAML, XAML browser applications, XPS documents, Run components not signed with Authenticode, Run components signed with Authenticode) would prevent a successful attack, including potentially future ones against the vulnerable components. Disabling those parameters (amongst others) is exactly what this document suggests.
To quote from the advisory itself: “this security update resolves a privately reported vulnerability in Microsoft Windows GDI+. The vulnerability could allow remote code execution if a user viewed a specially crafted image file using affected software or browsed a Web site that contains specially crafted content”.
Here, in the “Workarounds” section disabling metafile processing is listed as a potential one. Which, in turn, we’ve recommended here.
So, to cut the chase: once more proper hardening could have been your friend, at least for those two “critical” ones.And yes, we’ve already taken the potential business impact of these measures into account. We can safely state that in many environments there’s practically none. But not having to worry about some of yesterday’s advisories and maybe even avoiding getting owned (for MS11-029 Microsoft estimates that it’s “likely to see [a] reliable exploit developed in [the] next 30 days”) might have some benefit in pretty much every organization. Think about it!
Recently Micele and I were researching for our talk about the current state of security of Multifunction Devices (MFDs). Since we’re both seasoned pentesters who are quite familar with MFDs, we were really surprised that very little new research is going on on the topic of MFD security. While diving deeper into the topic, we found a very simple explanation for this: As in 2002, it is still possible to download print or scan jobs using PJL, many devices still offer default FTP or Telnet access, and, of course, stored files can be recovered from MFD hard drives — on an enterprise wide scale. To even strengthen our impression of the current state of MFD security, most devices crashed or did go wild while performing some scans — and we do not talk about fuzzing here.
This devastating result lead to the question how MFDs can be secured. Since there are a lot of MFD hardening resources out there, even from vendors, we decided to put together a comprehensive hardening guide for MFDs. To raise the level of awareness, we put together a lot of examples on attacks on MFDs and then focused on the development of our own MFD security guide which is based on the seven sisters. The result of this approach can be found here. And of course, soon there will be a ERNW newsletter to cover this topic in a more academic and structured way 😉
Lots of stuff has been written about this blog post from RSA describing the (potential) details of the attack, so I will refrain from detailed comments on this piece that Marsh Ray nicely called “some of the most egregious hyperbole I’ve read in infosec”.
Just one short note. Presumably the attack, in an early stage, used a “spreadsheet [that] contained a zero-day exploit that installs a backdoor through an Adobe Flash vulnerability (CVE-2011-0609)”.
TROOPERS11 was a blast! We received great feedback from all attendees and speakers. This really pushes ourselves towards the next goals and an even better security conference in 2012.
We’re happy that everybody got home safely with new ideas and inspirations in mind. On a side note: The awesome TROOPERS badge caused trouble for some of you with the airport security 😉 I really hope everybody could find a way to take it back home. It will hopefully find its way to an adequate place right next to your old memorabilia (cup of the first won soccer match, your college degree or photos from your first ballet show). Regard it as the proof of your latest achievement and tell everybody proud and loud: WE ARE TROOPERS.
Best regards,
Florian
PS: Videos and photos are coming soon. Stay tuned.
Some of you may have heard of the break-in at RSA and may now be wondering “what does this mean to us?” and “what can be done?”. Not being an expert on RSA SecurID at all – I’ve been involved in some projects, however not on the technical implementation side but on the architecture or overall [risk] management side – I’ll still try to contribute to the debate 😉
Feel free to correct me either by comment or by personal email in case the following contains factual errors.
Fundamentals
My understanding of the way RSA SecurID tokens work is roughly this:
a) The authentication capabilities provided by the system (as part of an overall infrastructure where authentication plays a role) are based on two factors: a one-time-password (OTP) generated in regular intervals by both a token and some (backend) authentication server and a PIN known by a user.
c) the OTP generation process takes some initialization value called the “seed” and the current time as input and calculates – by means of some algorithm at whose core probably sits a hash function – the OTP itself.
d) the algorithm seems publicly known (there are some cryptanalytic papers listed in the Wikipedia article on RSA SecurID and a generator – needing the seed as input – has been available for some time now). Even if it wasn’t public we should assume that Kerckhoff’s principle exists for some reason 😉
e) So, in the end of the day, an OTP of a given token at a given point of time can be calculated once the seed of this specific token is known.
This means: to some (large) degree, the whole security of the OTP relies on the secrecy of the seed which, obviously, must be kept. [For the overall authentication process there’s still the PIN, but this one can be assumed to be the “weaker part” of the whole thing.]
Flavors
RSA SecurID tokens, and those of other vendors as well, are sold in two main variants:
– as hardware devices (in different sizes, colors etc.) Here the seed is encoded as part of the manufacturing process and there must be some import process of token serial numbers and their associated seeds into the authentication server (located at the organization using the product for authentication), and some subsequent mapping of a user + PIN to a certain token (identified by serial number, I assume). The seeds are then generated on the product vendor’s (e.g. RSA’s) side in an early stage of the manufacturing process and distributed as part of the product delivery process. Not sure why a vendor (like RSA) should keep those associations of (token) serial numbers and their seeds (as I said, I’m not an expert in this area so I might overlook sth here, even sth fairly obvious ;-)) once the product delivery process is completed, but I assume this nevertheless happens to some extent. And I assume this is part of the potential impact of the current incident, see below.
– as so-called “soft tokens”, that are software instances running on a PC or mobile device and generating the OTP. For this purpose, again the seed is needed and to the best of my knowledge there’s, in the RSA space at least, two ways how the seed gets onto the device:
generate it as part of “user creation” process on the authentication server and subsequent distribution to users (by email or download link), for import. For obvious reasons not all people like this, security-wise.
generate it, by means of an RSA proprietary scheme called Cryptographic Token Key Initialization Protocol (CT-KIP) in parallel on the token and the server and thereby avoid the (seed’s) transmission over the network.
Btw: In both cases importing the seed into a TPM would be nice, but – as of mid 2010 when I did some research – this was still in a quite immature state. So not sure if this currently is a viable option.
Attacks
For an attacker going after the seed I see three main vectors:
compromise of an organization’s authentication server. From audits in the past I know these systems often reside in network segments not-too-easily accessible and they are – sometimes – reasonably well protected (hardening etc.). Furthermore I have no idea how easy it would be to extract the seeds from such a system once compromised. Getting them might allow for subsequent attacks on remote users (logging into VPN gateways, OWA servers etc.), but only against this specific organization. And if the attacker already managed to compromise the organization’s authentication server this effort might not even be necessary anymore.
compromise of the (mobile) devices of some users of a given organization using soft tokens and copy/steal their seeds. This could potentially be done by a piece of malware (provided it manages to access the seed at all, which might be difficult – protected storage and stuff comes to mind – or not. I just don’t know 😉
This is the one usually infosec people opposing the replacement of hard tokens by soft tokens (e.g. for usability reasons) warn about. There are people who do not regard this as a very relevant risk, as it requires initial compromise of the device in question. Which, of course, can happen. But why “spend energy” on getting the seed then as the box is compromised anyway (and any data processed on it). I’m well aware of the “attacker can use seed for future attacks from other endpoints” argument. One might just wonder about the incentive for an attacker to got after the seeds…
It should be noted that binding the (soft) token to a specific device, identified by serial number, unique device identifier (like in the case of iPhones) or harddisk ID or sth – which can be done in the RSA SecurID space since some time, I believe since Authentication Manager 7.1 – might to some degree serve as a mitigating control against this type of attack.
attack vendor (RSA) and hope to get access to the seeds of many organizations which can then be used in subsequent targeted attacks. I have the vague impression that this is exactly what happened here. Art Coviello writes in his letter:
“[the information gained by the attackers] could potentially be used to reduce the effectiveness of a current two-factor authentication implementation as part of a broader attack.”
I interpret this as follows: “dear customers, face the fact that some attackers might dispose of your seeds and the OTPs calculated on those so you’re left with the PIN as the last resort for the security of the overall authentication process”.
I leave the conclusions to the valued reader (and, evidently, the estimation if my interpretation holds or not) and proceed with the next section.
Mitigating Controls & Steps
First, let’s have a quick look at the recommendations RSA gives (in this document). There we find stuff like “We recommend customers follow the rule of least privilege when assigning roles and responsibilities to security administrators.” – yes, thanks! RSA for reminding us, this is always a good idea 😉 – and equally conventional wisdom including pieces like “We recommend customers update their security products and the operating systems hosting them with the latest patches”. And, of course, it’s pure coincidence that they mention the use of SIEM systems twice… being a SIEM vendor themselves ;-))
From my part I’d like to add:
– in case you use RSA SecurID soft tokens, binding individual tokens to specific devices seems a good idea to me. (yes, this might mean that users using several devices have several, different, instances then. and, yes, I understand that in the shiny new age of user-owned funky smartphone gagdets used for corporate information processing, this might be a heavy burden to ask your users for 😉
– some of you might re-think their (sceptical) position as for hard tokens: in the RSA SecurID space soft tokens can be “seeded” by means of CT-KIP, so no 3rd party is involved or disposes of the seeds. I’m not aware of such a feature for hard tokens.
– whatever you do, think about the supply chain of security components, which parties are involved and which knowledge they might accumulate.
– replacing proprietary stuff by standard based approaches (like X.509 certificates) should always be reconsidered.
– whatever you do, authentication-wise, you should always have a plan for revocation and credential replacement. This should be one of the overall lessons learned from this incident and the current trend that well-organized attackers will go after authentication providers and infrastructures (see, for example, this presentation from the recent NSA Information Assurance Symposium).
Last but not least I’d like to draw your attention to this upcoming presentation on the current state of authentication at Troopers. I’d be surprised if Steve and Marsh would not include the RSA incident in their talk 😉
Reading this advisory I’m quite tempted to emit another rant on the relationship of heavy use of 3rd party components, lack of (security) quality assurance and services running at times where they’re not needed (see second workaround here). I’ll refrain from that for today. Just wanted to let you know that the underlying vulnerability in Struts2 was initially discovered by Meder Kydyraliev who gives this talk at Troopers in two weeks. He’ll certainly describe the inner workings of this one, and others… 😉
Just a short addition to the previous posts ([1], [2]) on IPv6 security today. In the last two days I had the opportunity to sharpen my understanding of some aspects of IPv6 behavior in (Windows-) LANs. Actually I gave an IPv6 workshop for some members of the “Project Services” team of Hamburg-based computer & competence IT-solutions provider [btw: thanks to Mr. Wendler of CuC for organizing it, and thanks to Mr. Cassel for the breakfast…].
Those guys were amazingly adept with Wireshark (I got a ‒ long-needed ‒ refresher on display filters ;-)) and networking technologies in general so it was a workshop in the purest sense: lots of practical hands-on, lots of tinkering, lots of enligthening discussions.
I pretty much like every part of my work (I mean, from my humble perspectice infosec is the most exciting discipline anyway, isn’t it ;-)) but workshops like this one are sth I particularly enjoy. Huge personal progression and getting paid for it 😉
Ok, enough enthusiasm… let’s get back to earth. Based on the stuff we did I’d like to raise two points.
a) I found out that the latest MS Windows versions all seem to dispose of a parameter allowing to disable the processing of router advertisements at all. Being an old-fashioned networking guy I’m not sure if I like this (given it seems a violation of core IPv6 architecture fundamentals), still ‒ wearing the hat of a network security practitioner ‒ I recognize there are certainly use cases where this comes in handy, e.g. DMZ segments with a mostly (and deliberately so) static configuration approach.
The parameter can be set on an interface level by
netshint ipv6 set int [index] routerdiscovery=disabled
Checking the actual state of it can be done by
When set the box (interface) in question will not process router advertisements anymore which might provide some protection against RA based attacks (notably the spoofed RA attack described in this post). The configuration of the basic IP parameters must then either be done manually or by means of DHCPv6. It should be noted that DHCPv6 currently does not provide an option to distribute a default route/gateway (this IETF draft on Default Router and Prefix Advertisement Options for DHCPv6 was seemingly discontinued, see also the extensive discussion in RFC 6104) so in case of a DHCPv6 based (stateful) configuration approach the respective systems won’t have a default route (which, again, might be helpful for their security, in certain scenarios).
Obviously, once you use this one for hardening purposes, you should closely keep track of the affected systems. Else troubleshooting might became a nightmare …
b) I had a closer look at the router advertisements generated by fake_router6 from the THC-IPV6 attack suite. Those are generated with a “High” (value: 01) router preference. So going with a high router preference on one’s own might just provide equal terms. Of course we still recommend to use this approach (discussed in this post, and in RFC 6104) to protect from “mislead entities emitting router advertisements on the local link”.
this post is the sequel of this one on the IPv6 security feature called “RA guard”. As announced in that post I recently got a 4948 on ebay. After installing the appropriate image and noticing that RA guard was still unavailable I found out it should have been a 4948E which is capable of “doing IPv6 in hardware” (as opposed to the “simple 4948” only supporting IPv6 in a “software switched” way. and pls note there’s also the informal term 4948-E denoting a 4948 running an “enhanced image”).
Well… we’ll certainly find some useful function for that device – and I just initiated the acquisition of a 4948E – but I was just eager to get my hands on the practical use and implications of RA guard. Looking at the Cisco Feature Navigator for “IPv6 Basic RA Guard” it seems that it’s still the same platforms supporting it as eight weeks ago, so there was only one option left: get one of our Cat 65Ks from the basement and perform the testing with it.
Given ERNW’s strong emphasis on sports pretty much immediately some kind-of “strong man contest” evolved going like “who can lift and carry the device without further assistance”
so we had quite some fun just putting the box into operation.
[for those interested we can provide the number and nature of modules installed at the time of that contest ;-)]
Ok, back to the technical side of things. Here’s what we did:
Step 1) Perform RA based attacks with “RA Guard” being absent.
For that purpose we used Van Hauser’s THC-IPV6 attack suite (which can be found here) and performed two specific attacks.
1a) Use
fake_router6 eth0 2001:db8:dead:beef::/64
to send spoofed RAs which resulted in an additional prefix and associated default gateway being learned on the victim system (default install of current MS Windows).
It should be noted that the additional (“spoofed”) default gateway had a better (= lower) metric in the routing table.
1b) Use
flood_router6 eth0
to send arbitrary spoofed router advertisements (with the eventual purpose of a DoS condition). This worked equally well. Right after starting the attack the CPU load of the attacked system ‒ and for that matter, the load of other systems on the local link as well ‒ went to 100% (and stayed there for hours after stopping the emanation of packets on the attacker’s host).
[you might notice that this is not the “oldest and weakest laptop we took from our lab shelf”…]
In short: both RA based attacks we tried worked like a charm.
So this is absolutely something every security responsible of a IPv6 enabled network should be concerned about…
Step2: put lab device into action, enable RA guard and check if it helps.
[which, btw, “silently ignores” any WS-X6248-RJ-45 modules present in the box. Of course, I’m aware that all of you valued readers always carefully read the release notes of this (or other) images before deploying them…]
We then enabled RA guard on the access-ports to-be by
Router(config)#int range f1/24 – 36
Router(config-if-range)#switchport
Router(config-if-range)#switchport mode access
Router(config-if-range)#ipv6 nd raguard
and performed the attacks again.
Without success! Seemingly the box efficiently (and silently) discarded all RA packets arriving on those ports.
So we can certainly state that RA guard fully performs as expected. One minor annoyance we noticed: there seems no log message of any kind while the attack is in progress.
However, earlier – when performing step 1a – we had noticed sth like this on the router legitimately serving the local network with RAs:
Thus there might still be a way to detect RA based attacks. It’s just not on the box providing protection from them but on another one.
If you want to see these attacks (and the – quite simple – configuration of RA guard) in practice you can do so either at our workshop on “IPv6 security in LANs” at Troopers or at the Heise IPv6 Kongress in May. At both occasions we’ll demonstrate this stuff.
We think that RA guard is pretty much the only way to protect against RA based attacks in an operationally feasible way (as opposed to, say, filtering ICMP type 134 by means of port based or VLAN ACLs).
To mitigate the risk of “RA interference” caused by potentially misconfigured systems (e.g. Windows systems running 6-to-4 with a public IPv4 address and Internet Connection Sharing enabled; more on this in a future post on IPv6 tunnel technologies) or by systems “accidentally inserted into the local network” (employee connecting SOHO DSL router. which, again, certainly never happens in your network…), there might be another configuration tweak that could help. The standards track RFC 4191 on Default Router Preferences and More-Specific Routes introduced the concept of a preference assigned to router advertisements distributed by routers on the local link.
For that purpose the format of RA messages is extended and two of the previously unused bits within the byte containing the flags are used for a “Prf” (Default Router Preference) value.
The configuration is quite simple (on Cisco routers where the feature was introduced in IOS Version 12.4(2)T) and goes like this:
Router(config)# interface f0/1
Router(config-if)# ipv6 nd router-preference {high | medium | low}
Hosts receiving RAs tagged with a high preference flag will prefer them over RA messages emitted with the default value (“medium”). We’ll play around with this in a more detailed fashion at some occasion and keep you posted. In the interim you might look here to get some background on the way router preference works.
The next post of our series is planned to cover the (in the current Windows world) ever-present tunnel technologies and their security implications.
