Software Security/Group 2/Code Scanning Reflection

SWAAT

A description of the SWAAT tool is given on the SWAAT site: "SWAAT is an open source web application source code analysis tool. SWAAT searches through source code and analyzes against the database of potentially dangerous strings given in the .xml files. Thus it does NOT positively identify the existence of a vulnerability - this generally requires application contextual knowledge. It identifies the usage of functions/strings/SQL that could lead to a finding. All potentially dangerous code references are included in the output report."

As you can read the SWAAT tool does not find any real issues. It only identifies php usages which can lead to a finding. As a result of this, you still have to inspect the code manually by looking at the results that SWAAT reports to you. Because of this SWAAT has some negative points with which you have to handle:

• Produces just a list with findings, you cannot change the layout of this
• Doesn't categorize its results

So, the results that are given by SWAAT are somewhat annoying to read. It is just a HTML page with a long list of findings which are categorized by the level of risk. Furthermore you cannot change anything about layout. You would have wanted the fold and unfold some findings and possibly some other categorization in order to see to which problem a finding can lead.

Of course the SWAAT tool is not completely bad. Some positive things about the tool are:

• Easy to install
• Produces its results really fast

The tool is installed really easy on a Windows system. It is just of matter of unzipping a file and than you can run the tool from the location to which you unzipped the file. After that it produces its results really fast. For the phpBB source code the tool needed less than 5 minutes to produce the results. This might be a result of the fact that the tool does not find real issues but only the usage of code which might lead to an issue. So in fact it might be that the tool only does a kind of search for code usage that might be unsafe. It also gives a warning for variables which have a name that indicates that the variables might be used for safety critical functions of the applications. Another positive thing about the tool is that it tells you how its finding might get dangerous although this description is often a bit too general.

For our purpose the tool seems to be a bit useless because it looks like there is nothing with SQL output encoding. Maybe for other groups the tool might be useful because, for example, it can find the usage of hidden fields. So we do not want to say that SWAAT is completely useless but for our purpose it seems to be less useful.

Fortify

We performed a static source code analysis using Fortify 360. The Fortify Software company is founded in 2003. In September 2010 the company was acquired by Hewlett-Packard Company (HP). Fortify Software creates products that identify and remove security vulnerabilities from software applications. The program we used was "Fortify 360 v3.0.0 Analyzers and Apps".

The tool was really easy to install. We just had to follow the instructions that were given in the mail which we received after we had asked for a fortify login. The tool uses an automatic installer, the only thing you have to do is tell the tool where it can find the license file and where to install. Afterwards it installs completely automatically.

When we ran the analyzer it took quite some time (about three hours) to run. In our case, the tool used a lot of memory. The default settings of 600 megabytes was not sufficient to fully scan the phpBB source code. After giving the tool the availability over 800 megabytes of memory it was able to complete the analysis successfully. For the phpBB source code, the tool finds 6094 vulnerabilities including 3543 critical, 2488 high, 1 medium and 62 low issues.

The issues are clearly divided into categories. In our case the issues where grouped into 17 categories including Cross Site Scripting, Dangerous File Inclusion and SQL Injection. The tool also gives to possible to group the results based on other characteristics such as file name, sink (calling function) and OWASP top ten lists. We think most of the grouping options are not really useful, but some of them may turn out to be useful.

When you select an issue in the menu, you will directly jump to the related line of the source code for the issue.

For each issue you also can see a short description of the vulnerability. For example: "The file admin_styles.php passes an unvalidated filename to a dynamic include statement on line 76. Allowing unvalidated user input to to control files that are included dynamically in PHP can lead to malicious code execution."

There is also a more extensive explanation of the vulnerability, even with some examples. Also recommendations are available to learn how to deal with the issue. So the tool does not only learn you that there are issues, you also learn more about the specific issues by reading the explanation and the recommendations.

Another nice feature about the tool is that you add comments to an issue, so a next time you can still have your thoughts of an issue available. This is made possible because you can save the results of a scan in a fpr file. This is more than saving a list, it also save the references to the code and so on.

Our overall impression of Fortify is that it is a very nice tool to use with a lot of functionality. Unfortunately it is a very expensive tool so not everybody can afford buying this tool. Nevertheless it was by far the best tool we have tried.

CodeSecure

It took quite a while for us to get a valid trial license for CodeSecure. At first we had a working copy of CodeSecure 4.0.0 since Geert works at Forus-P (a security company that has a license for it), but unfortunately Geert's laptop died around that time. Since the CodeSecure trial is linked to your computer's MAC address, we had to go through the entire process again. It took a while before this second license was approved, but we received and could start scanning after a delay of almost a month. The trial license allowed us to make full use of the software, to a limit of five different users and ten different applications. There was no limit on the number of scans we are allowed to perform on one and the same application.

Installing the tool was quite easy. The tool runs as a Windows service and uses Java to scan applications. Installing the tool only required us to unzip to any directory and run the CodeSecureServiceManager.exe application in order to install and run the service. After doing this the tool was accessible through any web browser on the local network, and even outside after forwarding the port specified in the configuration file.

Once we started the scanning process we noticed how smooth the tool runs on an average computer. Despite the recommended amount of 4 GB of RAM, there was no real impact on the system. We think this requirement could have been set somewhat lower.

The interface of CodeSecure is very clear and the various options are not at all hard to find. Scanning a new application is as simple as selecting the source files, via either ZIP, SVN, Windows Share, FTP or CVS, and giving it a name. CodeSecure will then look through the source files and detects the programming language used to write the application after which you are able to specify certain sanitizing functions for CodeSecure to account for during the scan. The following step is to specify what it should look for during the scan, followed by the actual scan.

The scanning process did not go as smoothly as initializing it did. The first scan we performed ran for about two hours, after which it looked like nothing was happening anymore. We then canceled the scan, because we thought something had gone wrong. The second scan ran for a whopping seventeen hours and revealed considerably more possible vulnerabilities. However, this scan also did not fully complete, since we canceled it too. The reason for canceling the second scan after seventeen hours is that in those fifteen extra hours of scanning it only progressed three percent further towards the end. The first scan stuck at 81% of the process and after fifteen more hours, the scan was still only 84% complete.

A week later we decided to perform another scan. This time we switched the laptop's power scheme to High Performance and did not allow it to sleep, hibernate or turn off any disks. This scan ran for almost two (!) entire days, namely 41 consecutive hours. The scan still was not yet finished after this period however, but since the laptop was needed for other purposes as well we decided to stop the scan and let it generate the report. In the end CodeSecure found over 2500 (possible) vulnerabilities. We will not go into detail here, but you can find them on their own page.

It should be noted that since the tool runs as a Windows service it is possible to put your computer to sleep, hibernate or even shut it down completely while the scan will still continue. As soon as the service is restarted, the tool will resume the current scan where it left off.

The results were very easy to read and filter. The tool features a trending function, which is essentially a means of checking whether more (or less) vulnerabilities arise over an extended period of repeated scanning. You can easily see how the application's security level progresses due to the graph it draws using the scans' results. As a developer you are also able to label each reported (possible) vulnerability as either Unvalidated, Validated, Needs Review or False Positive, all of which speak for themselves. Besides labeling them, a developer is also able to set the severity level of the vulnerability to High, Medium, Low or Info. When you want to review a possible vulnerability you are provided with a traceback, a file-path and the source code, the latter of which is almost a fully featured editor because of its syntax highlighting and line numbers. Clicking on the blocks in the stack of the traceback results in a jump to the corresponding line of the source code in the editor.

The reporting functionality is also very extensive. It is possible to only include specific types of findings in your report and there is also an option to include false positives. Before the latter is of any use however, it is required that a developer take a look into the output and label it a false positive. CodeSecure will not know this by itself, or else it obviously would not report it. Users are able to generate reports as a PDF file, an HTML file or an XML file, depending on their needs.

The tool is still very much being developed, since during the month it took to get a new license, CodeSecure was upgraded several times, from 4.0.0 beta 3 through 4.2.0. At first we suspected the performance drop after the 80% mark was a bug in the system and would be likely to be ironed out in the near future, but after Geert emailed with Armorize (CodeSecure's developer) he was told that the tool requires about 8 GB of RAM, contrary to the 4 GB the tool claims it needs when it is launched. This was probably the reason the scans we performed took so long, and since neither of us own a computer with 8 GB of RAM we will not be able to fully finish this scan.

All in all the tool is very easy to use. Installing required almost no effort, as did creating the application you want to scan. The only downside to it is that it takes considerable time to fully scan an application at the moment, but that is because our computers aren't powerful enough (i.e. they are not dedicated towards this type of task).