WASC Web Application Security Statistics Published
Written by Chuck Willis
Thanks to Veracode’s Blog for pointing me to the Web Application Security Consortium (WASC) Web Application Security Statistics that were recently published.
Overall, I think that the paper has some very interesting data and statistics. As Chris Wysopal at Veracode pointed out, it provides some good evidence to back up the seemingly common sense idea that white box testing (where the testers have access to source code, design documents, and internal resources) is more likely to find certain issues than black box testing. I believe that this is the case for most, but not all, types of issues. Again, the study appears to support this notion by showing that some issues types (such as Insufficient Authorization) are more likely to be found by black box testing.
I think that this study validates the approach that Mandiant takes toward conducting web application assessments. We always try to convince our clients to let us use both black box and white box techniques. When combined, these approaches allow us to find and validate different types of issues in different ways and provide better coverage in less time. It also allows us to easily eliminate false positives through manual testing.
A couple words of caution when reading the WASC paper, however. First, the titles of some of the tables and graphs are correct, but could be misinterpreted. For example, P. 9 is titled “The probability to detect the most risky vulnerabilities in Web applications (% Sites BlackBox & WhiteBox)”. What this figure is showing is the percentage of web sites tested with the different techniques which were found to have the issue shown, not the likelihood of actually detecting the issue if it exists.
So, it could be that only 44% of the sites subjected to white box testing had Credential/Session Prediction issues, in which case the white box technique was “perfect”. It could also be that 88% of those sites had the issue and the white box technique only found half of them. In all, this study did not appear to look at “false negatives” in determining what issues were missed, which is understandable since that is very difficult to account for in a study of this type.
The other word of caution I would propose is that there is no mention at all in the document of false positives, making it unclear how many of the findings included in the study were real issues in the sites tested. False positives can be very common when using automated processes, including external web application scans and source code scans. I would expect that the black box statistics in the paper would have accounted for false positives to some degree since manual effort was included, but that is just an assumption.
In summary, a great study with some good numbers, but I’d take them all with a grain of salt and use them as trends and ballpark figures rather than as ground truth.
Chuck
Tags: Black Box, Statistics, Veracode, WASC, Web Application, White Box
. 02 Nov 09 | Thoughts
Leave a Reply
You must be logged in to post a comment.
Recent Posts
- Combat the APT by Sharing Indicators of Compromise
- DOD Cyber Crime: New Audit Viewer/Memoryze
- M-Trends: Advanced Persistent Threat Malware
- M-Trends: The Advance of the Persistent Threat
- MANDIANT in Miami at the SCADA Security Scientific Symposium
Follow us on Twitter
- Mandiant: MANDIANT People #FF Correction it is @wendilou2
- Mandiant: MANDIANT People #ff Cont @sroberts
- Mandiant: MANDIANT People #FF @petersilberman @nickharbour @robtlee @0x73davis @wendilou @coryaltheide