SoftwareSecurity2014/Group 5/Code Scanning Reflection

Fortify

Fortify contains a nice user interface that helps to do all the administration of the issues it has found. This saves quite some time because it produced a huge amount of issues. We tried to get the SSC server running but we couldn't get it to work. As a result we couldn't work on the analysis with multiple people at the same time and had to send each others a message when someone wanted to analyse issues with Fortify. It is remarkable that for such an expensive tool they did not even include syntax highlighting for php code.

The tool Fortify gave a large amount of warnings. As expected, most of these warnings were false positives. Checking all those warnings is a very depressing, dull and time consuming task. (Erik:Still, from your discussion of the results on the other wiki page I get the impression that you could weed out quite a lot of obvious false positives quite quickly. Is that impression correct?) Luckily Foritfy provides a data-flow diagram that provides information on which user input was causing the warning. A positive feature of Fortify is that it gives a detailed description with examples for every issue. This makes it a lot easier to understand why it is a possible vulnerability. Another positive feature is that the person that evaluated a certain issue can leave a comment on why the issue is a false positive or a true positive so that others can easily understand the evaluation.

The way SMF is coded makes it harder for Fortify(and for the auditor!) to give good results. (Erik:Interesting observation!) SMF has a few global arrays where it stores a lot of data. When you assign user input to one key of that array, it is theoretically possible that it ends up in another key of the global variable. For example, it sometimes changes the $txt global(the loaded language file) and it replaces a text with a text with some user input. When another key of that language file is echo-ed it complains that it could echo user input. The problem there is that this line of code is then scattered in evidence traces among lots of issues. Luckily you can search for all issues that have a certain line in their evidence trace. It was often needed to search for the name of a key in all php files to see what happens with it.

RATS

RATS is a much simpler tool than Fortify. It simply scans your files for known bad functions and reports those. It had a lot of TOCTOU messages which were not relevant for our security requirement. Still it found some issues that were relevant.

RATS also gave a lot of mostly false positive warnings. Investigating these warnings was a very time consuming task. RATS does not provide any data-flow information. SMFs structure consists of a large amount of functions that use a large amount of global variables. This made it complicated to find the value of every parameter that was passed to the function that RATS complained about.

RATS can be a useful tool if you are not sure about functions that might be security vulnerable.

RIPS

We found it a bit more difficult to follow issues in RIPS. The errors are vague and there is barely any feedback, but by simply analyzing the code snippets and files associated with a issue we could identify the problem since we already encountered them in Fortify. There were mostly false positives by the look of the results and it is unlikely that RIPS will add anything to our project.

Regarding our requirement: Input Validation

The built-in analysis evidence of Fortify shows a nice trace of where the user input goes to when Foritfy finds an issue. Most of the issues Fortify reported are about user input ending up somewhere in the output.