SoftwareSecurity2014/Group 6/Code Scanning Reflection

(Erik: Fortify gives quite a small number of warnings for Roundcube, compared to what some other groups encounter. Eg, one group reports one complaint for every 5 lines of code. Is Roundcube a really small application? )

Reflection on Code Scanners

RATS

The name of this tool – rough auditing tool for security – already suggests that there can not be too high expectations in it. However, it can be very useful. It is also extremely fast, due to its simple way to scan source code based on patterns. The speed is a big advantage compared to some other tools, since it is easy to quickly check some code. On the other hand, it also produces many false positives, since in most cases it can not detect whether a vulnerability has been dealt with in the code. This is a large problem, since it is not very pleasing to have to check every time whether there is an actual problem in the reported line (especially when this has been done before for this line already). As far as we know, there is no way to make RATS ignore specific parts of the code. It would be useful to be able to annotate something as fixed and make RATS ignore this.

Pros	Cons
very fast (millions of lines of code in milliseconds) very easy to use very useful for inexperienced programmers that are not yet familiar with common security vulnerabilities of a specific language good to remind programmers of common vulnerabilities free to use easy to extend	many false positives RATS cannot be made to ignore fixed vulnerabilities (as far as we know) can only detect rather easy to spot vulnerabilities based on pattern matching explanation for the alerts is sometimes only rudimentary

RIPS

RIPS is especially designed to analyze PHP source code and search for different kinds of vulnerabilities. RIPS is available for free and runs quite fast: 200 seconds for the Roundcube project on a below average laptop. The output is quite extended. For each report of a possible vulnerability a structured tree of the relevant code is shown, so that you will have a clear view of the tainted used data that is passed along as it eventually reaches a sensitive function. From the analysis of our scan we could observe that RIPS sometimes incorrectly assumes that tainted data is passed to such a function, especially when there are complex code structures. This may also be due to the fact that RIPS does not support OOP, which is the programming structure in a major part of the Roundcube project's source code. All in all we think that RIPS is quite a useful tool to use. The level in detail is somewhat below that of Fortify, but the UI is very detailed. There are in-depth descriptions for each vulnerability and the possible attacks that could be initiated, although these descriptions are more generic. There is also a possibilty to generate an exploit based on an identified vulnerability, but we did not test this.

Pros	Cons
quite fast given the exstensiveness of the tool easy to use free to use extensive UI, providing detailed info about the vulnerability, a structured overview of the relevant source code, a graph of functions and even an in-built exploit writing functionality multiple verbosity levels and vulnerability/code filtering options	does not support object-oriented code the development of the current version is abandoned and although a new version is supposedly being developed, it is unclear when this is finished quite a lot of false positives, especially for code structures that become complex, such as multiple inner function calls, although these findings may also have been due to a lack of support for OOP

Fortify

One of the first things that is noticable is that fortify runs quite a while to analyse the source code (~33 min on a laptop that is already a few years old). Also all issues that we looked at where false positives (privacy violations in fact didn't reveal anything; hardcoded passwords weren't any; dangerous file inclusions were only configuration files and seemed to be implemented in a secure way; ...). This makes fortify really annoying to work with and not very usable. Since it takes so much time to analyze, one can not quickly have a look on the project. Dealing with all the false positives makes it very time consuming as well. We believe that developers don't enjoy fortify very much.

(Erik: Moreover, I think that Fortify would not be able to spot many issues wrt V4 and V9, as this typically requires some insight in what the web app is meant to do. Eg Fortify seems to focus a lot on lack of input validation, but for access control it can hardly say sensible things.

Pros	Cons
GUI to order vulnerabilities and visualize things detailed backtraces of security and privacy violations in form of graphs more in depth analysis compared to other tools many features	rather slow (compared to other tools) → usability issue complicated to acquire and install costly relatively complicated to use (several tools) in our case: almost all warnings were false positives