Software Security/CodeScanners

Assignment

As a first step of the code review project I want each group to try out Fortify and at least one other source code analysis tool on the PHPBB source code.

Concrete steps for this:

1. Pick one of the source code analysis tools from the list
   The free version available from CodeScan apparently only looks at SQL injection, so it would make sense for group 7 to try that one.
2. Run them over the [phpbb source code]
   If you find a tool cannot be installed (on a particular platfrom ,or not at all) or say crashes when you try it on phpbb, please document this in the wiki, on a wiki page for that tool, to prevent others from wasting time on this.
3. Consider the feedback you get from the tools. What kind of feedback do you get? How much feedback do you get, a few lines, a few screenfuls, or tons and tons of warnings? Does it look meaningful/useful? Is (some of) the feedback useful for any security requirements mentioned in the ASVS, and if so which ones? More in particular, does the tool provide feedback that might be useful for any security requirements your group is looking at? Maybe you already note that the two tools you try report very similar things? How easy it is to trace back a problem in the source code given the feedback from the tool? can you understand how these tools actually work? Do you spot obvious false positives? Can you see things that the tools are great at/useful for/hopeless at? Etc.
4. Look at the results of the code scanners from the point of view of the security requirements (V?) that your group looks at. Are any of the warnings relevant for this, and if so, which ones ? If not, can you image some code scanning tool that would give feedback relevant for your security requirements? Or is there some fundamental reason why a code scanner can not do this?
5. On you own group wiki-page, or a subpage, write a small section (say a screenful or A4) discussing these issues for each of the tools, and - if applicable - a short comparison between them. Deadline: April 20 so that we have a chance to look at it before the lecture on April 27 and we can discuss and compare our findings during that lecture.

   Overall goal of the section on your group page should be to give a rough impression of what the tools can do and how useful this might be, for which purposes. Of course, a big issue is how accurate and complete the feedback from the tools is, but that is something that I do not expect you to look at now. Hopefully, that might be clearer at the end of the whole project.
   If the tool produces output that you can stick up on the web or in the wiki for others to have a look at, that is fine. Of course, that's no substitute for the discussion of the tools.

6. Also, don't forget to keep a log of what you have done on your group log page! This should also be a useful means for synchronising work between members of the group. In the log also record your decisions on who will do what, to make sure everyone is clear on this.

Once this is done, if the code scanners did provide something useful for your security requirements, then you can start start checking if these warnings are false positives; this would amount to a Level 1B evaluation in the ASVS approach. If not, then you have to think of another way to get started verifying your security requirements, and you have to move on to a Level 2B type of evaluation. I won't have expected you have finished all this by April 27, but you should have some ideas on how to get started.

---

Overall impressions of the source code analysis tools

The table below gives a very brief overview of everyone's impressions of the tools and their usefulness, both in general and specifically for the security requirements you are looking at. Try to stick to a three word evaluation (e.g. great, good, very much, a bit, not much, marginally, not at all, not sure, not sure yet, ...). Motivation of this and possibly a more detailed judgement should be somewhere on your group's "reflection on code scanning page".

Group	Fortify			Yasca			SWAAT			Pixy			RATS			CodeScan			CodeSecure			PHP-SAT
	works?	useful?	useful for us?	works?	useful?	useful for us?	works?	useful?	useful for us?	works?	useful?	useful for us?	works?	useful?	useful for us?	works?	useful?	useful for us?	works?	useful?	useful for us?	works?	useful?	useful for us?
1	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
2	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
3	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
4	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
5	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
6	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
7	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
8	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
9	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
10	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
11	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-