SoftwareSecurity2014/Group 9/Code Scanning Reflection

During the verification of phpBB a set of tools was used in order to perform a static code analysis. This article will reflect on the Code Scanning Tools, in the context of this verification process, including Fortify, RATS and RIPS. For each tool the strengths and weaknesses are listed below.

Fortify

Fortify is a powerful commercial tool developed by HP. Running the tool against the code base of phpBB yielded a total of 777 issues. To get a general impression of Fortify we analyzed the critical level warnings of phpBB v3.0.7 not related to Password Management (These can be found in the OWASP Verification Report). Of 38 non-password-management-related issues we have found 26 false positives so we have a false positive ratio of approximately 70%. Below follows a few shortcommings of Fortify, as well as how useable the tool was on authentication and cryptography related issues.

Sanitized data

Many of the false errors found were caused by Fortify's inability to recognize sanitized data or safe sources of input such as configuration files. Another downside of Fortify is that it stops following data traces at files or databases. If it would trace the code that can write to these locations it could form a better analysis of possible issues and actual sources of user input. (Erik: But note that doing that would be very hard, and can only be done simpy assuming that there are no other applications interaction with these files or databases.)

Unrecognized output buffering

Output buffering is a method in php that can prevent outputted data from actually being sent to the user. Fortify doesn't recognize output buffering and can therefore create false warnings if it feels unsafe data is sent to the user while it is actually captured by output buffering.

(Erik: But such false warnings would only occur if after data is buffered there is subsequent detection that this data is unsafe before the buffer is flushed, right? So that actually happens in the code?

Usability

Although Fortify is quite extensive and helpful to locate possible security issues, the way to operate it is not always trivial. It, for example, only gives line numbers in the traces of an issues, meaning it isn't always trivial to find the position of the issues in the line itself. Also, on elaborations of the issue, Fortify only gives general comments about detected issues, making it hard to identify the particular issue.

Some features however, do provide structured methods to analyse certain issues. The possibility to display diagrams of call traces and the extensive possibilities to group issues really helps in understanding the issue.

Some features that would, in our opinion, improve Fortify would for example be to include features like one can find in IDEs (i.e. clicking on a function call will lead to that function).

Authentication specific issues

Since Fortify offers grouping the issues by different categories, we could easily find password related issues in the categories Password Management' and Privacy Violation. The analysis of Fortify in those categories however, seems to only include searching on certain keywords like password. These passwords that were found were, however, often normal variable names or hidden in the comments, leading to a lot of false positives.

The only part where Fortify really helped was in discovering whether autocomplete is enabled on password forms. On those kind of issues it is far more helpful to use a tool as Fortify.

Cryptography specific issues

All cryptography related issues found in our code were related to hashes deemed insecure. The code also shows that Fortify is unable to detect whether hashed values have been salted, meaning it is not very good in detecting cryptography related issues.

RATS

RATS, the Rough Auditing Tool for Security, is a tool that aids the security analyst by scanning the code for points that may require extra attention. It scans for certain functions that have the potential to create security issues if used incorrectly. For each potentially dangerous function found it gives advice on how it should be used safely. The tool also estimates the risk level of the potential threat to help the analyst prioritise issues.

RATS can also find TOCTOU(Time of check, time of use) issues for which it tries to find checks and uses of variables. A weakness is that it only checks for variable names for TOCTOU issues. This causes the tool to link independent variables from different functions.

Usability

The tool gives a lot of warnings and does not aid in analysing the code involved in the potential issues it finds. This means the analysis still requires a lot of manual code checking to verify whether it is secure.

The library of dangerous functions that RATS uses can easily be extended to include more, or project-specific functions to check for. Making the analysis more human edditable when more dangerous functions are added.

Authentication & Cryptography specific issues

RATS with default settings doesn't really help in finding authentication and cryptography related issues. On the other hand its libraries could be extended to point out and advise on these kind of issues.

RIPS

RIPS is a tool, written in PHP, that detects potential security risks caused by tainted user input in PHP applications. RIPS allows for extensive configuration when initialising a scan where one can set the verbosity level and the vulnerability types to scan for:

Verbosity level: How deep the analysis goes.
User Tainted only
  1. file/DB tainted
  2. Show secured
  3. Untainted
  4. Debug mode
Vulnerability types: Which types of vulnerabilities the analysis should look for.
  1. All server-side
    1.1 Code Execution
    1.2 Command Execution
    1.3 Header Injection
    1.4 File Disclosure
    1.5 File Inclusion
    1.6 File Manipulation
    1.7 LDAP Injection
    1.8 SQL Injection
    1.9 XPath Injection
    1.10 other
  2. All client-side
    2.1 Cross-Site Scripting
    2.2 HTTP Response Splitting
    2.3 Unserialize / POP
A checkbox to scan all subdirectories as well

Running the tool against phpBB resulted in 56 issues found. Almost all of these issues were found to be false positives. Possibly due to the fact that object-oriented PHP is not yet supported in RIPS.

Usability

RIPS structures its issues according to the files that they were found in. For each issue a short outline is given that includes: the kind of issue that is given (Cross-Site Scripting, File Injection, etc.), a short description of the problem and the lines of code that are the cause of and lead to the insecure situation. It would be useful to know what the priority/severity of the issues are, but RIPS does not show this. Most of the issues that RIPS finds are false positive. It seems that RIPS deems any file or database input as potentially harmful user input and does not see whether the input is checked/sanitized. Also, it seems that RIPS sometimes omits code, which forces the user of RIPS to open the corresponding file and find the line of code to really understand what it does. For example in common.php on line 153

($secure)? 'https://' : 'http://')

becomes

'https://' : 'http://')

in RIPS.

Authentication & Cryptography specific issues

With the focus on tainted user input and as the list of vulnerabilities indicates, RIPS is not a very useful tool for finding authentication and cryptography issues in a PHP application.

Conclusions

Comparison of Fortify, RATS and RIPS

In comparing the tools, there are two important things to consider: The correctness of the analysis of the tool, and the ease of use of the tool. In the correctness of the analysis, Fortify stands out the most. Although it has quite a lot of false positives, it does find more true positives than the alternatives and that is, in the end, what you want to find. Also in the ease of use of the tool, Fortify wins. Although there are some improvements to be made, it does provide a few features that make it a lot easier to analyse the given issues. RIPS does provide some ways to analyse the issues, but is by far not as good as Fortify.

Fortify was, besides this, also the only tool that could focus on Authentication and Cryptographic issues, although it proved not too helpful in finding out whether a page is in the public or private domain. (Erik: What do you mean by public or private domain here? wether a page is publicly accessible or only by a specific user/the admin? If so, isn't that part of the internal logic of the web application, which you can hardly expect Fortify to know something about?)

Tools in general

All these tools can only offer a guideline to the security analyst and a lot of manual analysis is still required to verify all issues. Proper aiding in analysis, e.g. by showing relevant code, can greatly reduce the user's work.

In general these tools seem to focus a lot on sanitation of data. This is also one of the weaknesses of the tools because they don't really have any way to tell if data has been sanitized. (Erik: Still, tools can tell something about whether data has been sanatized if it knows some standard sanitisation functions that could/should be used.) This is the cause of most false positives.