SoftwareSecurity2012/Group 4/Code Scanning Reflection

In this page we give a short reflection on the code scanners we tried. This will be a short summarization of the Trying out code scanners page combined with our findings after completing the Manual code analysis .

RIPS

We tested the PMS module with RIPS. First of all the RIPS code scanner has a nice GUI, which looks nice and is well structured. It also offers a nice summarization of the results, with a diagram showing the results per module. The several ways to examine the findings (for example to show parts of the source code) and short documentation about the findings was also very useful.

Then the scanning itself. On the RIPS website, it says that verbosity level 1 is recommended. Higher verbosity levels give more results. We used verbosity level 3, because level 4 gave us such a large amount of warnings (797), that it was not really useful to check them all. In total RIPS gave us 33 warnings for all the PMS files on verbosity level 3. All of them were false positives as we showed on the code scanning page. Therefore we think RIPS doesn't do a very good job. If RIPS gives us 33 false positives on verbosity level 3, verbosity level 4 probably only gives us far more false positives, because the warnings on level 3 are more critical than the ones on level 4. Also RIPS doesn't check intermediate functions for tainting. If there is an echo of a $_GET or $_POST variable, RIPS does not test for functions that might filter the contents between getting the variable from input fields and setting it as output. This could be done by mapping against standard PHP functions like intval().

After performing the manual code analysis we noticed there are no really severe problems with the code. This might explain the large number of false positives. We notice that when running RIPS on verbosity level 1, it finds very few errors, which is consistent with our finding that there are very few serious problems. We did not find any serious security problems that RIPS missed (this is probably mainly because we did not find any very serious security problems at all), so we could say RIPS does not report lots of false negatives. When taking this into account, we actually could say RIPS does a good job when scanning our module, as long as the verbosity level is not set too high (probably verbosity level 1 is fine).

We did not learn anything new about specific security vulnerabilities while using RIPS. All security vulnerabilities RIPS found were already known to us.

Yasca

The user interface of Yasca was clearly not as nice as for RIPS. While RIPS offered a easy to use web interface, Yasca only had a command line interface. Yasca has several output options. The nicest was HTML. The output was not bad, but we liked the output of RIPS much more. At least the output was good enough to understand what the problem was, why it was reported and where it occurred. A negative point about Yasca was we needed to we even had to modify some bat files to get the extensions (RATS and PHPLint) working.

Again, we did not really learn any specific security vulnerabilities from using Yasca.

We will now discuss the results of the code scanning for each extension separately.

Yasca core

The Yasca core scanner gives some more warnings than just the security warnings. It checks for performance issues as well as licensing restrictions too. This is useful, but not in the context of this course. The security warnings however were all false positive. It gives some of the same warnings as RIPS. So it also doesn't know which functions can filter the contents of the variables. Unfortunately Yasca also didn't provide any useful warnings. However taken into consideration the code actually was secure in overall, this probably makes sense. However still Yasca does not seem to be a very good tool for security scanning, since it gives lots and lots of warnings, many of them not even security related. For this reason we prefer RIPS over Yasca (also because of the much nicer GUI of RIPS).

RATS

RATS didn't find anything specific for the PMS module. Considering the results of our manual review (no very serious security problems found), this could be a positive sign (since there were no false positives reported). However it also could be the case RATS just does not report anything at all, resulting in a lot of false negatives. When looking at the results for the whole FluxBB project, we consider RATS to be relatively useful. Although in the results it is clear RATS is not meant for PHP specifically. Therefore we consider the results of RIPS to be better.

PHPLint

PHPLint gave many warnings. 1863 in total. Most of them are not even security related. PHPLint complains about standard PHP functions. For these reasons we think PHPLint is not a very useful tool for security scanning.