Software Security/Group 2/Code Scanning

SWAAT

After running the SWAAT tool on the PHPBB2 source code we have got a lot of findings. During the scan SWAAT examined 76 files, containing a total of 37113 lines.

Risk level	Number of findings
High	14
Medium	2222
Low	16

The tool gave us some more detailed information about the findings. It gave us a name, description and location of the problem.

Detailed evaluation of the results.

Fortify

We performed a static source code analysis using Fortify 360. The Fortify Software company is founded in 2003. In September 2010 the company was acquired by Hewlett-Packard Company (HP). ((??Erik: is this really relevant?)) Fortify Software creates products that identify and remove security vulnerabilities from software applications. The program we used was "Fortify 360 v3.0.0 Analyzers and Apps".

This analyzer detects more than 470 types of software security vulnerabilities across 18 development languages in source code (including PHP 5). It also ranks these vulnerabilities by severity of risk. Also guidance is given on how vulnerabilities could be fixed.

In our project we are only interested in SQL vulnerabilities, so we will only look into a specific portion of the results we have summarized below.

After running Fortify on the source code we have got a lot more possible risks than after we ran the SWAAT tool. Initially Fortify ran out of memory, causing us to add the -Xmx flag (described here) to the command line arguments in order to get complete results.

Risk level	Number of findings
Critical	3543
High	2488
Medium	1
Low	62

Detailed analysis of the results

CodeSecure

After running CodeSecure on the phpBB2 web application the tool reported a ...

We will not go into detail about the pros and cons of the tool. For a detailed review of the tool please head over to the evaluation page.

Scan Summary

Duration	Did not complete
Scanned Files	101
Scanned Lines	43,130
Entry Points	62
Vulnerable Entry Points	24
Resulting Vulnerabilities	2,571
Vulnerable Statements	53
Vulnerable Files	16

Detailed analysis of the results

Comparison

False positives

Fortify and CodeSecure report quite a few more risks than the SWAAT tool. The obvious first question here should be: "do Fortify and CodeSecure report a lot of false positives or does SWAAT just not detect these true positives?". The answer to this question is no! Fortify and CodeSecure just do a more thorough scan. They do more than a simple search for keywords. ((?Erik: And SWAAT only does such a simple search?))

Overlapping results

Given the shared purpose of the tools, there is a surprising lack of overlap in their findings. Our initial expectations were that Swaats results would be largely subsumed by those of Fortify and maybe raise a few arguable additional points.

usage of the 'eval' function

Both Fortify and Swaat report the usage of the eval function. Curiously fortify doesn't mention the usage in includes\bbcode.php, line 61, whereas Swaat does. CodeSecure does not report about this.

Sensitive information handling

Both Fortify and Swaat raise bells, as soon as variable names or comments mention certain words. Words like 'user', 'password' or 'session'. The tools do not point to exactly the same lines of code, as Fortify seems more concerned with these terms in the comments, while Swaat reacts more to variable names. CodeSecure does not report about those words. An exception to this is the usage of the variable "$query", which Fortify immediately chucks in a critical risk SQL Injection category.

Swaat unique results

Though Swaat does only report a very finite of distinct warning categories, there are still some findings that are unique to this tool with regard to Fortify.

Usage of HTML hidden fields

Swaat warns for the usage of hidden fields, arguing that users can easily view their contents and even manipulate them. As long as the information contained in it is not security critical the usage of these fields has its place.

Race conditions

For various functions that operate on handles Swaat generates a low severity warning. Specifically the tool reports that usage of the functions "is_dir", "is_readable" and "is_link" is subject to race conditions. (text)Searching the fortify results, none of these functions is mentioned.

Fortify unique results

After some inspection we found out that Fortify tends to generate an item for each (possible) vulnerability, although lots of these items are actually the result of a single (possible) vulnerability. There are significantly less distinct issues than the number Fortify reports. A positive side to this story is that Fortify helps indicate these items by labeling it as a shared sink.

In contradiction with our initial expectations, the easiest method of conveying what results are unique to Fortify is listing the items that overlap. Indeed, so limited - rather in flavor than in number - are the findings of Swaat, that reading the previous sections and viewing the fortify results themselves serves as an illustration.

CodeSecure unique results

CodeSecure starts its report with a lot of statistics about the results of the scan. It tells you where most vulnerabilities are and shows some nice diagrams. Compared to Fortify, the tool does not really find any new vulnerabilities. Compared to the SWAAT tool almost everything is a vulnerability that was not found by SWAAT. Again the report of CodeSecure has indicated this since it is much longer and much more detailed as the report that was generated by SWAAT.

At the end of the CodeSecure report there is a list with suggestions about how to fix most vulnerabilities (with some examples). Those suggestions are based on the different categories of vulnerabilities that were found during the scan.

Ease of analyzing each tool's results

Swaat results are a simple HTML report with no interactive components whatsoever. The grouping is done by severity, but beyond that items are listed in the order in which they where found. Besides listing the location and type of error, there is a small snipped of code context and a textual explanation of why this piece of code was raised as a warning.

Fortify on the other hand does not only contain a lot of features helping the user understand what kind of vulnerability has been found. It also allows for grouping by severity, OWASP's Top Tens, PCI, Category, Source, Taint Flag and so forth. Besides these categories it allows the user to label an item in five categories: Not an issue, Reliability Issue, Bad Practice, Suspicious and Exploitable. It also keeps track of how many of the items you have audited by using these labels. Fortify also allows for filtering of results to preset user categories and also assigning a vulnerability to yourself for easier viewing. After a scan it provides you with a detailed trace of how the vulnerability could be exploited, often in several different ways and, just as important, tells the user why the item is a (possible) security vulnerability.

Besides the very extensive grouping options, fortify provides the user with a rich repository of information. first there is a short summary of the issue, greatly resembling those that Swaat provides. Then the user can opt to read a detailed explanation on the issue type, with other examples and why this is an issue in the first place. The next tab provides generic solutions to the reported vulnerability. There is even a little sequence diagram of calls that a statement will generate in different layers of the code.

In all fairness, we have to say that the large summary of the issue and the generic solutions provided by Fortify will in most cases provide no valid information as developers will already be familiar with most cases. But in the few cases this is not so it will prove quite enlightening. If nothing else it can function to re-raise awareness of the possible consequences and remind developers of the default ways of securing certain procedures.

CodeSecure can produce a nice report in pdf format. In our case the report contained 128 pages. Like Fortify is the report also contains different categories of issues. CodeSecure also adds some nice figures to it in order to show some statics of the security issues in the source code. After this the the tool start reporting all the issues en shows you the problem and in which file it is and why it is dangerous. Unfortunately the tool is less interactive compared to Fortify. This means you cannot click on the issues and immediately jump to the source code. Compared to SWAAT, the tool gives the same kind of output (just a report in which you have to read) but it is much more structured than the SWAAT report. It has for example a table of contents so you know where to look and it contains much more information about the issues.

So after all, SWAAT does not give us any interesting results and when a result is interesting it is also covered by Fortify and CodeSecure. Those last two tools seem to give very similar results. They complain about almost the same things and give a very detailed report. So we think those tools are appreciable above SWAAT. Nevertheless those advantages come at a (pretty high) price.