SoftwareSecurity2012/Group 4/Code Scanning Try Out

We tested all tools listed on this page using Windows 7 Ultimate both 32 and 64 bit with Apache 2.2.14, MySQL 5.1.41 and PHP 5.3.1 (XAMPP for Windows 1.7.3). To test the effectiveness of the tools we ran them on a local FluxBB 1.4.8 installation with the New Private Message System modification by Visman (this is the modification we want to examine against all the security verification requirements listed on SoftwareSecurity2012).

RIPS

First Impression

We used RIPS version 0.53. The installation was very easy and the tool works perfectly on our test system. We tested the tool by scanning both FluxBB as whole and our modification specifically and this succeeded in one try.

The tool scans for many security vulnerabilities specific for web applications written in PHP. Some examples are: Code Execution, Command Execution, Cross-Site Scripting, Header Injection, File Disclosure, File Inclusion, File Manipulation, LDAP Injection, SQL Injection, Unserialize with POP and XPath Injection vulnerabilities using static code analysis with over a hundred test cases.

The tool supports five verbosity levels:

1. User tainted only
2. File/DB tainted + 1
3. Show secured + 1 and 2
4. Untainted + 1, 2 and 3
5. Debug mode

For us the first four levels are most useful. The levels are ordered on strictness. The higher the level, the more vulnerabilities the tool detects, but possibly also more false positives.

The GUI of the tool looks very nice, supporting a few settings (like setting the verbosity type and the vulnerability type) and many code styles. This way you can get the results you want, displayed how you want it. After scanning the GUI generates a nice result summary, discussing the types of vulnerabilities found and the amount of each type. You will get a very nice graphical overview of the categories of warnings that were found using a collection of pie charts and bar graphs. It also has some other (popup) windows giving an overview of the scanned files, the scanned functions and the (kinds of) user input used. The overall view of the security vulnerabilities is nice, has code highlighting, gives filenames and line numbers. It is possible to hide a specific warning or all warnings in a certain file, which is a nice extra feature making the result overview clear. It also supports some extra functions per warning, for example a help function (giving an explanation about the vulnerability type), a review code screen and a generate exploit option. In overall our first impression of RIPS is very good. Everything looks complete, nice and useful.

Number of warnings

We first ran the tool over the complete FluxBB project (including our modification). We did this for the first four levels. The total number of found vulnerabilities are given in the table below:

Verbosity level	Code Execution	Command Execution	Header Injection	File Disclosure	File Inclusion	File Manipulation	SQL Injection	Cross-Site Scripting	HTTP Response Splitting	Possible Flow Control	Unserialize	Total
Level 1	0	0	1	1	2	6	2	54	2	0	0	68
Level 2	1	0	1	7	9	12	2	286	3	0	18	339
Level 3	4	0	1	7	9	16	2	461	3	0	19	522
Level 4	143	1	2	32	373	34	26	3244	72	264	20	4211

After that we ran the tool over the files important for our modification (install_mod.php, pmsnew.php, plugins/AP_PMS_New.php and /include/pms_new/*). RIPS also include files listed (as require or include) in the file(s) selected. Therefore some of the found vulnerabilities are in other files, not belonging to our modification. The total number of vulnerabilities found are given in the table blow:

Verbosity level	Code Execution	Command Execution	Header Injection	File Disclosure	File Inclusion	File Manipulation	SQL Injection	Cross-Site Scripting	HTTP Response Splitting	Possible Flow Control	Unserialize	Total
Level 1	0	0	0	0	1	0	0	24	0	0	0	25
Level 2	0	0	0	0	1	2	0	24	0	0	0	27
Level 3	0	0	0	0	1	2	0	30	0	0	0	33
Level 4	17	0	0	5	74	16	0	593	15	77	0	797

Usefulness of results

As you can see the number of warnings increases when the verbosity level is higher. Especially on level 4, the number of warnings is very large. The amount is so large, that it probably is not useful any more and consists of many false positives. RIPS confirms that higher verbosity levels give a great increase of the number of false positives. RIPS even advises to run the code scanner on verbosity level 1, but we will use level 3 as explained in the code scanning section. On levels 1, 2 and 3 the tool reports much less warnings. For the complete FluxBB project the number of warnings at level 3 is still very large, possibly too large to work with in the timespan that is given for this project. However when you only consider the Private Message System modification, the number of total warnings (33) is a nice amount which can be worked with.

The security vulnerabilities found on verbosity level 3 in the PMS modification are mainly Cross-Site scripting vulnerabilities, which of course are very useful for security verification requirements V6 (Output validation HTML) and possibly V5 (Input validation). Besides XSS vulnerabilities also File Disclosure and File Manipulation vulnerabilities are found which are useful for security verification requirements V5 (Input validation) and V9 (Data Protection). Unfortunately there are no other vulnerabilities found using this verbosity level and therefore this level is not useful for the other security verification requirements.

On verbosity level 4 there are warnings in almost every possible vulnerability category, useful for almost every security verification requirement. Unfortunately the amount is too large. However when ignoring the XSS vulnerabilities, the amounts become workable again. Therefore the tool is (probably) useful for almost all the security verification requirements, and thus very useful for us.

Tracing back a warning to the source code is not always that easy. The program provides a nice interface and does mention the filename of the found vulnerability, including line numbers. This makes it easier. However figuring out why the program seems to thinks something is a vulnerability is not that easy. It tries to give some sort of trace where the untrusted/tainted data causing the vulnerability comes from, but this is not always clear at first sight. How the tool exactly works (or why it seems to think something is a vulnerability) is not always clear either. However there is paper on the working of RIPS which possibly could provide more insight on this subject.

We had a quick look at the first results and for some warnings tried to figure out if they were real or a false positive. Unfortunately we did find some false positives. For example there was an XSS warning about printing a GET variable, but this GET variable was escaped (casted to an integer) using the intval() PHP function (which in our eyes excludes the possibility of an XSS attack).

Yasca with plugins

We have tried the following plugins for the Yasca framework:

• RATS
• PHPlint
• Yasca Core

We will now describe our most important findings based on the guiding questions that are denoted at the Exercise page.

Yasca framework

First Impression

Compared to the RIPS GUI, the Yasca framework was a bit of a disappointment. The RIPS GUI consisted of a nice web interface with nice graphs and representations of the code that is currently inspected. The Yasca framework didn't have any of these. It is only a console application that you need to run with the path to your project as a parameter. The output will be a static HTML page that you can find in the installation directory of Yasca. This HTML page has some nice extra features as a possibility to hide certain categories of warnings and a nice option to show extra information about the warnings and an indented view of the currently inspected source code parts.

Raw Results of Yasca with PHPLint and RATS plugin

FluxBB forum

The next result files are the original results of the Yasca framework with the Yasca core, PHPLint and RATS plugin when run on the newly installed FluxBB forum without any modifications. The first file has HTML layout which is the most human readable and the second is CSV layout.

Forum Only HTML

Forum Only CSV

Private Messaging Modification files

The next result files are the original results of the Yasca framework with the Yasca core, PHPLint and RATS plugin when run on the files that were included in the Private Messaging Modification Package. The file has HTML layout.

PM Mod Only HTML

FluxBB forum and the Private Messages Modification installed

The next result files are the original results of the Yasca framework with the Yasca core, PHPLint and RATS plugin when run on the FluxBB installation which we had modified with the Private Messaging mod. The first file has HTML layout which is the most human readable and the second is CSV layout.

Forum and Mod HTML

Forum and Mod CSV

Diff file of the Forum and Mod CSV and the Forum Only CSV

We have also created the following CSV file which contains all the warnings that were present in the Forum and Mod CSV file but were not present in the Forum Only CSV. This would give us a list of warnings that were really created by installing the Private Messaging mod which we want to analyze for this project.

We think that this diff file gives a more precise list of possible vulnerabilities that are caused by installing the private messaging modification because it will also take the changes that we made manually in the fluxBB forum code files into account. These changes are not analyzed when we would only run on the code files that are for the Private messaging mod specifically (which we have also analyzed, see the section 'Private Messaging Modification files' above).

DIFFERENCE between Forum and Mod and Forum only CSV

RATS

First impression

The included results of RATS give some interesting results, though most of the results are just false positives. Luckily RATS doesn't give too many results, so checking all of them shouldn't be a problem. The most interesting results were the fopen scans. RATS checks for all fopen function occurrences and reports when the argument of the function call is dynamic. The parameter could be overwritten by user input. This could make local file inclusion possible. There are 11 of those warnings given by RATS. The rest of the warnings by RATS are TOCTOU (Time Of Check, Time Of Use) warnings. According to RATS, there is a race condition on file reads and writes. If you test whether the file exists before reading from it, the file can be deleted between checking and opening the file. In principle these warnings are correct, because a webserver often has a multi-threaded structure (with more apache workers for example) but there is no good way in PHP to solve them. There is no way to lock the file after checking for existence in one atomic operation. You can only lock a file after you have opened it. Therefore, you need to make the blocks where you write or read files as small as possible and perform these actions directly after the in-use or existence check. If this is done by a programmer, a remote attacker has probably no change of misusing the non-atomic TOCTOU structure because the timing that is required for this kind attacks is very, very specific. In this case, the file open statement happens immediately after the check for existence, so it is very unlikely this will cause problems and therefore these warnings could be considered false positives.

Number of warnings

Whole FluxBB installation with Private Messaging mod

Category	Number of warnings	Description
RATS mail	1	Warnings concerning interaction with mail servers.
RATS is_readable	2	Warnings concerning TOCTOU issues.
RATS is_dir	1	Warnings concerning TOCTOU issues.
RATS fsockopen	2	Warnings concerning user input validation.
RATS fopen	11	Warnings about possible file inclusion attacks.

Only Private Messaging mod files

Category	Number of warnings	Description
RATS mail	0	Warnings concerning interaction with mail servers.
RATS is_readable	0	Warnings concerning TOCTOU issues.
RATS is_dir	0	Warnings concerning TOCTOU issues.
RATS fsockopen	0	Warnings concerning user input validation.
RATS fopen	0	Warnings about possible file inclusion attacks.

Usefulness of results

The RATS scanner seems to focus on scanning functions that concern Input and Output. For security scanning this is a very important factor. When you do file readings, or socket readings or writings, you really have to make sure the input and output is validated to prevent that this IO point can be exploited. The scanner did not gave too much results in such a way that it gets infeasible to check all the warnings.

It gets clear that the scanner is used for more than one language and that the scanner isn't specifically made for PHP. For example, the TOCTOU warnings cannot be fixed for the PHP language. There just isn't a way to prevent that the state of a file changes between the check for file existence and availability and the opening of a file for reading or writing. Therefore, the TOCTOU warnings are in principle legit, but there is not a nice solution to fix these errors available for the PHP language.

PHPlint

First impression

Our first impression of PHPlint was that the installation of the plugin wasn't as easy as we had expected. There were some issues with the different paths of the executables and the system variable we needed to add to our set of Windows environment variables. The errors that were displayed during this process were not clear at all (only some path not found errors).

When the PHPlint plugin was installed, we also didn't get much information about the execution progress of the scan. We just had to wait a reasonably long time asking ourselves if the plugin was still working or that it encountered some infinite loop state.

Number of warnings

Whole FluxBB installation with Private Messaging mod

Category	Number of warnings	Description
PHPLint Findings Whole FluxBB with mod	11473	There are no categories among the PHPLint warnings so all warnings are just sorted on the filename of the file the warning is applicable to.

Only Private Messaging mod files

Category	Number of warnings	Description
PHPLint Findings mod files only	1939	There are no categories among the PHPLint warnings so all warnings are just sorted on the filename of the file the warning is applicable to.

Usefulness of results

The first thing that comes to mind when you look at the results that the PHPLint adds to the resulting HTML is that this is a lot of warnings. For a project of approximately 30.000 lines of code (see: fluxBB dev stats) PHPLint gives approximately 10.000 warnings. This resulted in a HTML page with a total size of 20 MB. Actually, I think that this is a nice example of a scanner with too much warnings. For every three lines, the scanner seems to give a warning or tip for one of these lines. This is too much to check for the programmer, so he will probably decide not to use these results during his analysis. Also there was no form of categorization possible, so the programmer isn't able to filter only certain types of warnings. The only distinction the PHPLint scanner made was the severity of the warning (which is indicated by the background color of the warning number field in the table). Approximately half of the result entries we had, were so called "Critical" The other half were the so called "Warnings" or "Informationals" Still, manually checking only the "Criticals" means checking approximately 5000 entries and this is still not very practical.

Though, maybe there is an option to set the level of detail for the PHPLint scanner as a parameter when the scanner is started. But because we used the scanner as a plugin of the Yasca framework, we did not found a possibility to do so (yet). Maybe this could give a more interesting selection of the warnings as result in such a way it gets feasible for the programmer to check the remaining warnings by hand. We could check if such an option is available in a later stadium of this project.

Yasca Core

First impression

Yasca core is the core scanner of Yasca. It has some built in checks that it will perform even if no plugins are installed. Yasca core will give some performance enhancement tips like using ++$i instead of $i++. This can be 20% faster according to Yasca. It also gave warnings about putting a function in a for-declaration (for(int $i=0; count($array); $i++). This function will be executed for every iteration. The count can be placed outside the for loop for faster performance.

Furthermore Yasca core gives possible Cross-Site Scripting vulnerabilities and some licensing and browser compatibility information.

The results are not overwhelming like the PHPlint results. The number of results is limited enough to check them all. So even if the results are false positive, it is doable.

Number of warnings

Whole FluxBB installation with Private Messaging mod

Category	Number of warnings	Description
W3 Core	10	Possible compatibility issues among widely used browsers.
Cross-Site Scripting	19	Warnings about possible XSS attack vulnerabilities.
Bug: Script Tag in JavaScript file	2	Warnings about syntactical errors.
Licensing Information	66	Summary of all found licensing declarations.
Performance tips	36	Programming tips to make the system faster (for example: use ++$i instead of $i++).

Only Private Messaging mod files

Category	Number of warnings	Description
W3 Core	0	Possible compatibility issues among widely used browsers.
Cross-Site Scripting	6	Warnings about possible XSS attack vulnerabilities.
Bug: Script Tag in JavaScript file	0	Warnings about syntactical errors.
Licensing Information	20	Summary of all found licensing declarations.
Performance tips	5	Programming tips to make the system faster (for example: use ++$i instead of $i++).

Usefulness of results

Yasca only warned about 19 possible Cross-Site Scripting attacks. This is nothing compared to the amount RIPS could detect. These are the only interesting warnings for Yasca core related to the topic of software security. Most of them are false positives though. Like intval($_POST['p']) even if you can put <script> tags in the variable, intval will output this as an integer, so the script tags will be gone.

The rest of the warnings by Yasca core are about licensing declarations, browser compatibility and performance enhancements. That is not interesting for this course at all, though the performance enhancement tips are not bad. It won't hurt to implement them.