SoftwareSecurity2014/Group 1/Code Scanning Reflection

Level 1B requirements

OWASP Application Security Verification Standard 2009 defines the following Level 1B security verification requirements:

V3 – Session Management Verification Requirements

Index	Target	Explanation
V3.6	Session IDs	Verify that the session ID is never disclosed other than in cookie headers. Pay attention particularly for URLs, error messages or logs. This includes verifying that the application does not support URL rewriting of session cookies.

V10 – Communication Security Verification Requirements
No requirement for Level 1B

V11 – HTTP Security Verification Requirements

Index	Target	Explanation
V11.1	Data validation	Verify that redirects do not include unvalidated data.
V11.2	HTTP requests	Verify that the application accepts only a defined set of HTTP request methods, such as GET and POST.
V11.3	Charset	Verify that every HTTP response contains a content type header specifying a safe character set (e.g. UTF-8).

Fortify

Fortify produces a huge number of issues. Frankly, we believe a large number of these to be false positives. For instance, it complains about passwords being stored in HTML files, which we were unable to find. We did find hard coded plain text strings, however, that the tool likely mistook for passwords. (Erik:I believe Fortify complains if it eg. finds the sting password in some comment.) In addition, although various ways for grouping reported issues are provided (including OWASP top-10 categorization), none of them overlaps with the ASVS requirements, which limits the ease of use when trying to verify the latter.

The quick mode is indeed a much quicker scan. It achieves this partly by ignoring all potential issues below the 'high' classification, although we also found a difference in the critical issues. Some are not reported, but in other cases more appear to be found compared to the regular scan.

Fortify's memory warnings are puzzling and reduces our confidence in the scanning results. We remain skeptical even though the command line version produces about the same number of issues. An option to increase memory allocation through the GUI would be useful. Although Fortify's refusal to scan certain files for being too complex is indicative of poor code quality in FluxBB, it also highlights the tool's limitations.

Overall, Fortify can help identify egregious errors, but it cannot actually verify the ASVS requirements. Just because it fails to locate issues pertaining to a particular requirement doesn't mean there aren't any.(Erik: Note that for some requirements Fortify might be more helpful that the V3, V10, and V11 you look at. Still, no reason why a static analysis tool should not be able to spot the presence/absence of the HTTPOnly flag, which might help in checking V11) This is a general problem with automated tools. Furthermore, the memory problems and complexity issues point to limitations in HP's scanning engine. Finally, the sheer number of results means there will be a lot of tedious weeding through non-issues.

Correlating scanning results to ASVS requirements

Because the results are numerous, it was not practical look at each one of them analytically. Instead, we selected four relevant Level 1B requirements to filter useful results. In total, we identified only 3 instances related to requirement V11.1 but none related to requirements V3.6, V11.2 or V11.3. (Erik: Were these 3 instances false or true positives? And do you have any confidence in it that Fortify would find all/more/many/most problems related with V11.1?)

RATS

RATS is a simple utility (read: a glorified GREP script) that is fast but limited in its analysis. Its results and warnings are generally at a "lower level" than the OWASP ASVS requirements. No reference to sessions or HTTP requests is made as RATS is not aware of those things. Input and output validation, buffer overflow and TOCTOU warnings are essential to improving the security of any application, but none of the warnings we received is inside the scope of our analysis (V3, V10, V11).

PHPLint

PHPLint was not designed to provide comprehensive security verification, only to prevent potential errors that result from weak-type variables and structures. As such, its benefit for use in OWASP security verification is very limited.

Doxygen

We ran Doxygen for documentation generation on the source tree of the project. This may be useful later on when inspecting the source code for bugs and vulnerabilities. Unfortunately the output generated by Doxygen is incorrect because it does not properly parse the PHP code. Even though the code is correct (the application can be used on a webserver), we believe that Doxygen has very strict requirements with regards to code formatting.

PHP Code Sniffer

As 99% of the errors identified by the program were related to code formatting, we agreed that the output from this tool was not relevant to our case. Furthermore, the output is sent directly to stdout and is not properly formatted, which makes interpretation much harder than Fortify or even RATS.