SoftwareSecurity2013/Group 5/Code Scanning Reflection

Introduction

In this page, we will discuss the first phase of the application security verification project. This comprises of evaluating the architecture and usefulness of two given static analysis tools, RATS (Rough Auditing Tool for Security), an open-source solution and Fortify, an advanced, commercial solution from HP, in the context of web applications.

The evaluation is done on the example of the publicly available MediaWiki, a open-source wiki written in PHP. As the likelihood of finding a high amount of potential security vulnerabilities with static analysis tools, given roughly 1.2 million lines of code, we will focus on three chosen categories of security vulnerabilities. These are chosen according to the OWASP application security verification standard from 2009 and are as follows: session management, communication security and HTTP security. The depth of the analysis is limited to level 1B, encompassing only a manual review of warnings produced by the SCA tool.

RATS

RATS's functionality can be described as that of a simple pattern matching software. It is scanning the code and looking for particular patterns and keywords. Once a match with the internal blacklist is found, a warning is produced. These warnings have three different severity levels: high, medium and low.

It supports various programming languages, such as Python, Perl, Ruby, C, C++ and PHP. Unfortunately, Java, C\# and many more and not supported. Since the functionality of the program is very limited, it is only able to find common security related programming errors and TOCTOU race conditions. In addition to that, even the author states it is prone to false positives and false negatives. The former has been confirmed by our analysis. Although the amount of false-negatives cannot be directly deduced, due to the very trivial detection mechanism of security vulnerabilities it can be said with very high certainty, that whole families of vulnerabilities cannot be detected with it.

Finally, the produced output is very coarse. There is a simple categorization into three severity levels, where single messages provide a brief description of the issue found and its location.

Fortify

There is no public information available on the inner workings of Fortify, as the code is proprietary and belongs to HP. (Erik: Actually, Brian Chess, who founded Fortify, did write a whole book, called Secure Programming with Static Analysis, about the principles that are behind tools like this :-)Public sources only disclose that the source code is read, converted into an intermediate structure optimized for security analysis and plugged into the analysis engine, which consists of multiple specialized analyzers. The analysis is done with the help of secure coding rules and practices. These rulesets are updated on a regular basis.

Some of the languages supported are Java, C/C++, HTML, PHP, XML, SQL amongst others. Although python is supported, several files in our code couldn't be analyzed due to restrictions given by the student license. (Erik: What makes you say that? Did the tool produce output saying this? I was under the impression that the license we have is the full one, although maybe it does not inlcude the rule-set for python.)

The representation of scan results is visually appealing and structured. The warnings are classified by severity (critical, high, medium and low) and type (e.g. XSS, SQL Injection). All warnings are supplied with a detailed description on the possible reasons of the warning and recommendations on how to mitigate it. Next to the possibility of directly displaying the corresponding code, it is possible to visualize the complete function call chain. This allowed us for a much quicker finish when classifying the warnings according to the OWASP verification requirements mentioned in the introduction. The results can be outputted into a PDF document, allowing others, without Fortify, to read them. Finally, there is an option to display a message sequence diagram representing the data flow among different functions (grouped in the files that contain them), which greatly helps to understand the source of tainted data and the sink function, easing the work of the auditor; a picture of the call chain and the message sequence diagram is shown below (click for full size).

(Erik: your text about Fortify above does not say anything about whether you think the tool was any use for the security requirements you are looking at. Maybe the scan results are visually appealing and structured, but do you think they were useful?)

Notes: