SoftwareSecurity2012/Group 8/Code Scanning Reflection

RIPS

Installation

RIPS is easier to install than fluxbb, because it doesn't require a database. It is easily installed by moving it to the web root of a PHP-capable vhost configured in Apache.

Strengths

• Uses taint tracking which is a modern approach
• Seems to provide sane results (nothing easily discarded as a false positive)
• Has attractive and easily navigated output
• Has category-based selections for result filtering
• Is reasonably fast

Limitations

Its 'generate exploit' functionality is very simplistic. This feature really seems too optimistic to be attainable from a static context. Although it uses taint tracking it still seems to generate many false positives in more complex code like FluxBB.

RATS

Installation

Shipped in most GNU/Linux distributions. A matter of installing it through your native package manager.

Strengths

• Really fast
• Good for a first pass on new code to get a high-level idea

Limitations

It's fast and stupid, but great for a rough estimate of where to look. This tool only statically checks for potentially dangerous functions, it's like an automatic grep script.

YASCA

Installation

Once downloaded, YASCA is immediately ready for use by unpacking it and approaching it through the commandline. After that you can specify the path to the source code that needs to be analysed.

Strengths

• Can use of plugins, for example RATS, Pixy, PHPLint.
• The core version is fast with analysing.
• Output in HTML is simple, although some graphical representation of the data would be nicer.

Limitations

We already tested some of the YASCA plugins separately and therefore only focussed on the core version. Unfortunately this is limited in its analysis and reporting compared to other tools. It could benefit from providing more of the surrounding code around the bug or vulnerability and explaining more about the license restrictions.

We also tested YASCA on Mac OS X but this did not work because the script used windows line endings, fixing this led to more problems thus we decided to use it on a different platform.

CodeSecure

Installation

Installing and using CodeSecure is very straightforward. The package comes with enough documentation to help set everything up. Once installed CodeSecure can be approached in the browser (localhost:8080). From there it is a matter of selecting the right folder, options and hit the start scan button. Under linux CodeSecure strongly recommends that it is run as a seperate user, and it is installed as a service that runs constantly instead of as a program that you can run as required, the program takes up significant resources, so it is inefficient to always have it running.

Limitations

The only limitations we ran into was the license. This license only allows 10000 lines of code to analysed, but unfortunately FluxBB has more. Therefore we were forced to break FluxBB in parts and analyse the parts. This resulted in another problem (see Problems) and we had to keep breaking up in smaller and smaller parts, until we scanned one file at the time. Because CodeSecure uses dataflow analysis, it is necessary for all files to be included in the scan. Our results might not be valid or contain a lot of false positives because not all files were included.

The license also does not allow reports to be generated in HTML or PDF.

For our second attempt with CodeSecure we minified the source code of FluxBB by removing almost all the line breaks in the code. This resulted in going from approximately 43924 lines of code to only 5619. We were then able to zip the source code and upload it to the CodeSecure interface. The minified source is available here: [1] CodeSecure was capable of scanning the complete source code, and it identified 35 vulnerabilities, of which 34 were High Severity and 1 was Low Severity. As the license does not allow any reports to be made (other than in an almost unreadable XML format) we uploaded the webpage with results here: [2] and an example of one such errors description here: [3]. As you can see, the errors are mostly all SQL injections or XSS and the actual vulnerability report is not really helpful, especially if the source code is unformatted it makes it almost impossible to find the error CodeSecure is referring to.

Problems

We ran into a problem where CodeSecure would freeze during a scan (around 35%). We tried it on a different system, but there we had the same problem. To solve this we analysed a smaller amount of files, but this did not help until we scanned one file at a time. Both system ran Windows 7 (64bit) and hardware configurations were completely different.

Running CodeSecure under Linux seems to work correctly. The scans of the minified sourcecode run to completion and a list of vulnerabilities gets created.

CodeSecure is installed in Linux as a service that starts at boot by default. This means that CodeSecure will always systematically hog resources on the computer it is running on, and since it constantly uses more than 2.4GB of RAM it gives a significant performance hit to the rest of the system, even when the developer is not actively using it.

PHPLint

Installation

Whilst a patch was required for it to compile using OS X, something which was documented at [4], the compilation was painless. After compilation it could be executed from the command line passing a PHP file as a parameter.

First impression

From the documentation it was clear that PHPLint required the PHP source code to be augmented and that it could only scan a single file at a time. This, combined with the fact that the reported results are hard to parse didn't give a good first impression. PHPLint seems to be designed to aid a developer, not to scan for vulnerabilities.

Number of warnings

PHPLint generates a lot of warnings and errors since it must be run on each PHP file separately, with 98 of such files in FluxBB. Given that it produced 132 alerts for index.php it may very well be over 10000 alerts in total, too much to be useful. Whilst the number of alerts may be reduced by augmentation it is hard to correctly augment the source code for everyone except the original author and the number of alerts will likely still be over 8000. A second problem is that the alerts are not very detailed, a short message and the code is shown but it isn't possibly to search for a single type of message making it nigh impossible to automatically process the alerts.

Utility of result

This utility simply generates too many alerts with too little information about each alert to be even remotely useful. As FluxBB is not augmented there will be a high rate of false positives with the well known consequence that important alerts are drowned out. Manually sorting through over 10000 alerts just isn't feasible, nor is augmenting the source code ourselves. Combined with the fact that it can only be used to scan a single file at a time, and does not use information from included files (such as defined functions) in these scans it is quite useless.

Pixy

Installation

Installation was trivial, unzipping the downloaded file and executing the run-all.pl Perl script with a PHP file as a parameter was enough.

First impression

Pixy was listed as a dead project, and the lacking documentation did not help our first impression either but once installed it seemed easy to work with. Whilst, like PHPLint, it only allows one file to be scanned at a time Pixy does use the information included in the include files when scanning for vulnerabilities. It is also quite slow.

Number of warnings

With 53 warnings for a single file scanning FluxBB in its entirety would lead to over 5000 warnings, too many. However, this tool list the conditions for the vulnerability to be exploitable; "- conditional on register_globals=on" for example. Since register_globals is deprecated these can quickly be ignored reducing the number of warnings to 6, which is acceptable.

Utility of result

Whilst we haven't fully analyzed each warning the ones which weren't immediately disregarded do seem to be of importance at first glance. Since XSS errors are important to us as they indicate a lack of input validation this tool is quite useful.