SoftwareSecurity2014/Group 7/Code Scanning Reflection

Fortify

Fortify was excellent in terms of its documentation and user interface which is perhaps to be expected of such a widely deployed commercial tool. The commands to run the scan were well documented and the scan was completed relatively quickly. More importantly, the warnings were very easy to analyse (if they were true of false positives) with the function call diagram in auditworkbench that traces the flow of tainted information, across classes, into the vulnerable function.

The organisation of issues first according to the sink and then the source, also facilitated analysis. For example, a single sink might have 6 sources (e.g. of tainted information) which were related. Usually. analyzing and fixing one of them would similarly apply to the rest. Hence, althoguh the large number of issues (228) highlighted seemed intimidating initially, they could be cut down relatively quickly.

For each vulnerability identified, Fortify also delivered useful information describing the problem and ways to fix it. With this information, we easily understood the issues and how to fix them. It was also nice that issues are classified according to severity so that reviewers can prioritize the higher severity levels. It seems that Fortify's classification was relatively accurate for the Typo3 project.

Overall, this was the best tool reviewed. It would definitely be useful to eliminate vulnerabilities related to the security requirement V5 Input Validation.

RIPS

Running the scan via RIPS was very easy, however it took very long as opposed to Fortify. Furthermore, we found the following disadvantages of using RIPS compared to Fortify:

• Difficult to analyse if the issues are true of false positives due to the more unwieldy user interface.
• Have to manually scan the code and related sections in order to find what the error is
• The reported errors don’t occur in the files which are reported, but in included files. This makes it very hard to find the exact line of code that is reported, which is necessary (see previous point) for analysing the reported error.
• Nevertheless several true positives were identified which were similar to the Fortify tool
• The number of false positives were also not overwhelming and manageable to sort through
• Overall, the tool is useful and would help us to eliminate several vulnerabilities related to the security requirement V5 Input Validation.

Other Tools

We were put off by the tedious installation and configuration process of several free tools such as Yasca and RATS and hence did not test them further. PHPlint did not detect any errors because the authors had already used this tool with the project.

On the other hand, PHP Codesniffer swamps us with trivial false positives which does not lend any help to analyse the V5 Input Validation security requirement.

Overall, we did not enjoy using these tools and would not choose them to help us in an actual code analysis.

Conclusion

Similar to our conclusion for code scanning, the tools (especially Fortify) does help to find many instances where input validation is required (e.g. when tainted user input reaches a vulnerable function). Without such automated tools, this task would be tedious or near impossible for projects with large code base.

However, the tools cannot tell whether input validation has actually been done or not. More importantly, it also cannot check if input validation is done correctly. (Erik: I agree, but still, in some cases you might hope that a tool could check this, if it knows the correct php functions to be used to sanitise arguments of SQL queries,) To check this, other testing methods are required e.g. functional testing or penetration testing.

In conclusion, the tools definitely help to check against the security requirement (V5 Input Validation) by spotting mistakes we would otherwise not spot, and sharply cutting down the time involved. We would definitely choose to have such a tool (especially Fortify) to help us with this task, than without. Nevertheless, it is not a panacea and still needs a skilled human reviewer to interpret the results and should be used with other testing methodology for a more complete result.