SoftwareSecurity2013/Group 12/Reflection

ASVS

At the first sight the ASVS seemed well organized because of the various sets of requirements and the coherence between these requirements. However, when we looked at these requirements one by one it was not straightforward how to verify some of these requirements. For example for V11.3, Verify that every HTTP response contains a content type header specifying a safe character set (e.g., UTF-8) it was not clear what lines in the code would satisfy this and what we were exactly looking for. Because of this it wasn’t easy to inspect the code in an efficient way for this requirement. Checking the code manually line by line was not doable in this case. Another difficulty we encountered was that there was no mapping between categories in Fortify and requirement sets in ASVS, making it time expensive to find out to what requirement of ASVS belongs to what category in Fortify.

ASVS was clear about the requirements and what was meant with them, although, as mentioned above, it could be difficult to find out how to test them. We did not notice any incompleteness about the requirements of ASVS. We think the requirements were structured very well and logical in their categories, so we do not have any suggestions for that.

Groupwork

According to us, splitting up the work among different groups is a good way to split the work. By giving each group one or more sets of requirements to verify, it makes it doable to focus on certain parts of the code. In our case, we split our team of five students in different small subgroups, by splitting our three sets of requirements. In this way we were able to give each other feedback on their parts independently.

Fortify

Fortify seemed to be a useful tool when we started using it. It took some time to look at all the warnings, but since they were categorized, we could select only the categories that were useful for our sets of requirements. We had about five different types of warnings that were useful for our requirements in part 1b and most of these warnings appeared more than once. After checking these warnings manually we found out that most of them were false positives. Because of this we also had to do a manual verification. Furthermore, since the warnings of the tool didn’t map exactly to the requirements, there was no case in which we could only trust the warning and come up with a verdict. As a consequence, we had to check all the requirements manually.

Out of scope requirements

Not all the sets of requirements were easy to do with a manual inspection of the code. For HTTP Security and Session Management the code itself could tell us a lot. But in the case of Communication Security we had to deal with some requirements that were out of scope of this assignment. For example V10.1 asked us to validate server certificates. In this case it was necessary to run it on a server with connections to external systems to come up with a verdict. For this reason, not all the requirements of Communication Security were that useful for this assignment.

Timeline

For us the line between the end of phase 1 and the start of phase 2 was a bit unclear. After using Fortify we wanted to check the warnings the tool produced, but this required some manual inspection of the code, which was part of phase 2. But because phase 1 didn’t take a lot of time, we decided to start looking at the other requirements. At the start of phase 2 we realized that we already did a lot of work for this phase in phase 1. It would have been better if a more extensive description for both phases would have been available at the start of the assignment. (Erik:Good point.)

Conclusions

As a reviewer of the code our perspective as a software developer has changed. Because of the assignments in this course we realized that a lot of time and work in fixing the code at the end of the developing process can be saved by looking at security issues during the design phase and when writing the code. But to be honest, both short assignments with PREfast and JML attributed more to this view. In this group assignment we missed some part about how to fix the code and solve some problems. We realize that the assignment would take a lot of time if we also had to think about this aspect, but it doesn’t make you think about practical ways to change the code and avoid certain errors from the beginning. For the other assignments a more active “attitude” was required and it was interesting to dive into the code and try to solve the warnings. In the group assignment we experienced a certain passive approach, compared with the other assignments. (Erik: I see your point. But an active approach to fix things is only doable on smaller examples, doing this on an existing web-application is hard to organise as a course project, and I think it's important to also iexperience issues on a more serious scale - at least once in your life :-).)