SoftwareSecurity2014/Group 2/Reflection

Reflection on the whole process

What difficulties did you encounter, either with the code or the ASVS? Can you think of ways to reduce or avoid these difficulties?

Some of the requirements seem to overlap. A more detailed description of the requirements would be very helpful. In retrospect we chose a too large project to manually check for vulnerabilities. It would have been better if we had chosen a smaller project or reduced of the scope of the project.

How useful is this ASVS? Could the ASVS be clearer, more complete, or structured in a better way?

The Application Security Verification Standard is a good starting point for a security analysis, but although we used the ASVS as a checklist, we cannot guarantee that we have found all vulnerabilities in the codebase. As mentioned before it would've been nice to have more detailed descriptions of the requirements. At the moment, the descriptions are pretty minimal which can lead to different interpretations of the standard. (Erik: Good point. My impression is that they tried to keep the ASVS short, and not inflate it to some huge document like some of the other documents they made with OWASP, but now it it short to the point that it is not always very clear.)

The ASVS is geared towards experienced security reviewers, who already know how to achieve the requirements. Without additional material it isn't all that useful for developers.

How useful are the tools you used?

For the manual code review we mainly used simple tools like grep, find, etc. They were useful for basic searches in the codebase but had no additional functionality aimed at security analysis. For information about program flow etc. we had to rely on our own insight.

Without tool support, data flow analysis is just impossible to do by hand. Therefore a tool like Fortify is indispensible, even for manual code review. That being said, we found a bug in Fortify with relation to PHP namespaces which renders the results of the data flow analysis unreliable. The Fortify support team confirmed that it is a bug, so the results of our code review must be considered with this in mind.

What were the bottlenecks in doing the security review in your experience?

Our success in finding vulnerabilities in the code was tied to our experience with PHP. In this case we all had varying experience in PHP, so we had to rely on PHP documentation and the StackOverflow archives for reference. As we mentioned earlier, it often proved to be impossible to verify every single usage of a potentially unsafe function simply because there were hundreds of them.

Is splitting up the work as we did,with different groups looking at different security requirement, a sensible way to split the work? Or are there more practical ways?

It would probably be best to make 'expert teams', so that people can work on the requirements they are most knowledgeable about. It might even be worth considering choosing one big project for everyone verify, so that as a group we produce a more complete verification. However, as we have noted before, there is quite a big potential for overlap between the different requirements, so this might lead to repeated work.

What could or should developers do to facilitate a security assessment? Do you think that experiencing security from the perspective of a reviewer rather than a developer has changed the way you would design or implement a web application yourself?

It would make life easier for security assessment team if developers would document the security choices they made during implementation. Also, it proved to be very convenient when OwnCloud consistently used single functions for things like query execution or HTML output. By creating one point of sanitisation and making sure all data flows through there, it is much easier to verify the security of the codebase as a whole, by inspecting such an individual function. Creating such 'standard practices' is especially useful for large codebases with multiple developers. (Erik:good points!)

The dynamic scripting-like nature of PHP makes it a very bad language for static analysis. Especially for large projects, and especially for security-critical applications you want a language that is designed with this kind of verification in mind. A language with proper type system and higher-order functions would eliminate the need for unsafe language constructions like call_user_func_array. While it may be possible to write safe PHP code, ideally you would want a language where safety is guaranteed by the compiler. (Erik: Indeed. Btw,note that when it came to your problems in trying to find client side validation PHP also does not help, with all the server-side processing intermingled with HTML & javascript output)

Any other interesting, frustrating, boring, educational, etc. aspects of the whole experience.

-