Onderzoeksmethoden 2/het werk/2010-11/Groep03

Uit Werkplaats
Ga naar: navigatie, zoeken

Group Members


Introduction

Nowadays it is more and more common for a company to have a web shop. In this research we will focus on the web shop from the company Exonet, which offers IT related services. In their order process you'll have to deal with complicated steps which are crucial for the order. There are a lot of potential customers that are lacking the knowledge of the whole system behind ordering a domain. This system contains some complicated options for reaching every persons satisfaction. Therefor the usability of a web shop could enhance the chance of people steering in the right direction; complete the order. In the screenshot below you find the configuration page of the Exonet ordering system.
Domainconfiguratie.png

Problem statement

There are countless companies who offer the same services as Exonet. And it is our task to ensure that we need to reach all those customers, beginners or experts in this area. Recently a new order system has been launched and to ensure it is understandable for all users, this research has started.

The online order system is mainly used to register new and/or transfer existing domain names. Customers are able to log in to this system and use their personal data to register new domains. Besides registering or transferring domains, it is also possible to order webhosting packages, virtual private servers and much more. Since the order process for domains is of most importance, we will focus on only this aspect of the order system.

So the new order system has recently replaced the old order system. This new system has more options and needs to fulfill the satisfaction of all existing and new customers. There are specific parts in the system which we think are too complicated and need to be improved. Some parts of the ordering proces involve configuring a domain name, creating contacts for this domain name, setting nameservers, creating redirects and forwarding mail. Of course these options aren't mandatory, but many customers requested these configuration options which saves them and the company a lot of time.

The problem is that many customers register multiple domains names at the same time, and most of the time each of these domain names requires a specific configuration. We tried to combine these configuration options in a single popup which might need improvement. Therefor the focus of this Think Aloud Protocol will be on this part of the system, specifically on the efficiency and the helpfulness of the system.


This leads to our main research question:
Is the system helpful and efficient enough for customers to configure and order domain names?


To answer this questions we have to answer the following subquestions:

  • To what extent does the users computer experience influences the successfulness of ordering and configuring domain names?
  • Which parts of the system cause problems or delays during the ordering and configuring of the domain names?

Method

The method used for our research to analyze a persons behavior is the Think Aloud Protocol.

We set up the following steps to complete our research:

Step 1: Create a conceptual model

First we brainstorm about which data we will need for our research and how we can tackle the problem. This brainstorm session will lead to a lot of information which will output a ORM diagram to describe our input, output and analysis. This ORM diagram will be our main tool to define how we analyze a person's behavior and what data we will save. The diagram is focused on helpfulness and efficiency.

Step 2: Find users to participate

For our setup we would like a total of four users to participate, two technical and two non-technical users. This way we can measure the results in difference of experience level. We hope to see a significant difference in speed between technical and non-technical users. Furthermore we hope to see an improvement in task completion over time with all the test persons.

Example scenario for non technical user

Step 3: Describe scenarios

Six different scenarios will be described for test persons to execute, each user will execute two or three scenarios (depending on the execution time). These scenarios will include very simple, advanced and hard tasks. For these scenarios we will take a look at the last 20 orders and see if our scenarios are too simple or too advanced. After comparing our scenarios, they are ready to be executed.

Step 4: Execute scenario and collect data

To keep track of the actions during the execution of a scenario, we will use the functionality of QuickTime on an iMac. QuickTime supports to record the screen and at the same time record the audio input. To make sure we also capture the facial expressions, we will start the QuickTime webcam record. After this we use the webcam video as picture in picture in the screen recording (see image right). Now we can translate the video recordings to text. For the location we use this equipment at home.

Step 5: Organize the data

After the execution of the scenarios we will organize the data according to the ORM diagram. We will save this data in a database for analyzing purposes later. To enter data into this database we've created a small web application with lots of forms. We've decided on how we will describe actions to make sure we all analyze the data the same way. This way the data will be consistent. We've added option fields for ratings, remarks and scenarios with selectable values to reduce the chance for human errors.

Step 6: Analyze the data

When all the data is added in the database we will use the same web application to analyze it. We will generate formulas to calculate how many actions are successful and how many relevant remarks there are throughout the sessions. Furthermore we can easily print out the failed actions and their remarks to see where it went wrong and how we can improve the system.

Step 7: Complete the research and write a report

When we have a clear vision from our analyzed data, we are able to complete the research and write the conclusions regarding the order system. These conclusions consist of tips to improve the system and shows which pages causes a lot of problems with the users. The data will support these tips and remarks.

Planning

Week # Action(s)
Week 42 Define subject and research question
Week 43 Brainstorm about the research plan
Week 44 Work out the research plan and ORM model
Week 45 Update the wiki and start with describing scenarios
Week 46 Recruit users and finalize scenarios
Week 47 Execute the scenarios with the users using TAP
Week 48 Process data
Week 49 Data analyzing
Week 50 Data analyzing and concept report
Week 51 Finalize report

Execution

Conceptual model

The following scheme is the initial ORM. During the execution of the think aloud protocols and processing of the data we made some additions. These can be found in the reflection.


Below you will find our latest ORM scheme.
ORM2 OZM2.png

Participants

We are going to test this on people which have experience in this sector and who don't. For the research we used 4 participants, 2 of each group. So there will be one session with 4 different users.

Group 1 - Technical users
Bard Duijs
Laurens Alers

Group 2 - Non technical users.
Ivy Raaijmakers
Martijn Stege

Scenarios

Execution of the scenarios are planned for 07/12/2010.

To keep track of the actions during the execution of a scenario, we will use the functionality of QuickTime on an iMac. QuickTime supports to record the screen and at the same time record the audio input. To make sure we also capture the facial expressions, we will start the QuickTime webcam record. After this we use the webcam video as picture in picture in the screen recording (see image right). Now we can translate the video recordings to text. For the location we use this equipment at home.

Martijn.png


The following scenarios are based on the exonet order page. We made three scenarios for non-technical users and three for technical users:

Non-technical users

Scenario Name File
1 NT-S1 Bestand:NT-S1.pdf
2 NT-S2 Bestand:NT-S2.pdf
3 NT-S3 Bestand:NT-S3.pdf

Technical users

Scenario Name File
1 T-S1 Bestand:T-S1.pdf
2 T-S2 Bestand:T-S2.pdf
3 T-S3 Bestand:T-S3.pdf

Notes

  • While executing the think aloud protocols, we made the the choice to use 4 participants instead of 6. Transcribing and analyzing the data seems like a lot more work then we expected. (30/12/2010)

Data structurering

We will save the data in xml format using the following structure (according to the ORM scheme). 


<?xml version="1.0" encoding="UTF-8" ?>
<session id="1">							id= unique session id
  <user exp="T">Users Name</user>					experience is T (technical) or NT (non technical)
  <scenarios>
    <scenario id="1" name="T-S1">					id = unique scenario id, name = document name which they executed
      <actions>
        <action id="1">							id = unique action id
          <description>description goes here</description>		description of the scenario
          <status>success</status>					Status is 'success' or 'failed'
          <duration>24</duration>					duration in seconds
          <page>domain-check</page>					the page on which the action is executed
          <remarks>
            <remark id="57" rating="2">Remark goes here</remark>	id = unique remark id, 'rating = 1 (irrelevant), 2 (medium), 3 (relevant)
            <remark id="58" rating="1">Some other remark</remark>
          </remarks>
        </action>
      </actions>
      <totalDuration>185</totalDuration>				Total duration of the scenario in seconds
    </scenario>
  </scenarios>
</session>

Notes

  • While structuring the data we decided it would come in handy to add a rating to the remarks of the user. (30/11/2010)
  • While analyzing the data we changed the structure into the a new, improved version which holds more data. The old one is still shown on the page (30/12/2010)
  • After our presentation on 13/01/2011 we've had a feedback comment about the rating terms we used, we decided on changing them into irrelevant, medium and relevant.

Data analysis

Method for Transcribing the data

To transcribe all the data we gathered during the Think Aloud Protocol, we created a custom tool (in PHP) instead of editing manual in the XML. This works more efficient and makes us not forget adding any information. Besides adding, modifying and deleting the tool comes in handy for generating statistics for our conclusion. We've taken three screenshots from our tool, which are displayed below (clickable).

A part of the transcribing our data is adding a rating to a remark of a user. This rating is fairly important because during a Think Aloud Session a lot of things are said. A lot of relevant things are said but a lot of nonsense is told as well. We decided on when a remark is 'Irrelevant', 'Medium' and 'Relevant' by checking if it relates to the system, and furthermore how it relates to the system. It could be something like "I don't understand this" which might be relevant, or something like "If you add x to this screen, that would come in handy!". After we rated all the remarks we went on and check all the remarks we rated relevant and decided with the three of us wether it was really relevant or not.

The images below from left to right:

  • An overview of all the Remarks within an Action
  • An overview of all the Actions within a Scenario
  • The page we used to add remarks to an Action

An overview of all the Remarks within an Action An overview of all the Actions within a Scenario The page we used to add remarks to an Action

XML Data

Notes/problems we encountered

  • During the analysis of the data we only transcribed en and defined actions that were relevant for the research.We excluded actions as : reading the scenario assignment.
  • During the transcribing of the data, we forgot to mention the page on the website the Action was referring to. This was necessary to get a good overview of which page on the ordering site is giving problems and needs improvement. Thats why we added this option to the database and the transcribing Tool.

Conclusions

After executing the Think Aloud Protocol we analyzed the gathered data and generated statistics For we can answer our main research question we need to find answers to the following subquestions:

To what extent does the users computer experience influences the successfulness of ordering and configuring domain names?

To support the answer to this question we refer to the images below. These images are screenshots from our tool.

Action statistics for technical users Action statistics for non-technical users

Looking at the statistics shown above we find that the results are not representative for the real findings. In the reflection we will explain why these results are not representative for the real findings. The success rate of technical users (76,92%) is smaller than non-technical users (78,26%). The cause of these results is that the technical users only did one scenario instead of the originally planned three. The non-technical users executed all three scenarios. This shows (as expected) that the technical users have less problems using the system compared with the non-technical users. From the total of thirteen actions performed, ten where successful. Looking at the statistics from the non-technical users we see a higher rate of successfulness, this is because of the fact that they executed all three scenarios and therefor have experienced a learning process. The first scenario which they executed was less successful as now shown in the results.

TNTscenarios.png

In the image above the scenarios T-S1 and T-S2 are executed only once, and NT-S1 is executed twice. The scenarios T-S1 and T-S2 was the first run for the technical users, so summing these results up we get a successful rate of 78,57% and an average duration per page of 88,13s. This duration is clearly more than the NT-S1 duration, but the T-S2 scenario is more complicated than the first. So the real results are as following:

  • Non-technical users executed their first round with a total successful rate of 61,54%.
  • Technical users executed their first round with a total successful rate of 78,57%.


So the computer experience has definitely influence on the successfulness of ordering and configuring domain names. Because looking only at the first scenario, the technical users did a lot better than the non-techincal users. On the other hand this confirms our expectation that the system is easy to learn and once executed users/customers know how to order and configure domains. This also means that some pages need improvement, to get the same results as technical users.

Which parts of the system cause problems or delays during the ordering and configuring of the domain names?

We found out that certain parts of the system cause problems and delays. This is derivable from the statistics shown in the figure 'General findings' below. As you can see there are a couple pages that generate a high percentage of failed actions. The remarks for each page with a high failure percentage are shown below.

Is the system helpful and efficient enough for customers to configure and order domain names?

In short, no. While executing our scenarios we came to the conclusion that there are parts of the system which need quite some improvement. The following points need some extra attention.

  • Domain check
    • Users are not aware of the multiple domain selection tool
    • Users try to change the time of registration (thus how many years) in this screen
    • Some non technical users have no clue on what a transfer or a registration is.
  • Shopcart
    • Some users ask if they can change contacts for domains here, we think that the previous page should make clear where along the ordering process they can change these contacts.
  • Debtor information
    • There was a particular case where a user thought that he was editing contacts for domains here. After reviewing the system he can't be blamed. The page states nowhere that the form is about debtor details instead of domain details.
  • Domain config
    • To some users it was unclear what a token was. The page should tell this.
    • One users noted that the token field is useless when there are only registrations involved, he's right.
  • Domain config (handle)
    • The page should state somewhere that handles are automatically created from the debtor details
    • The difference between owner and admin contact is unclear.
    • We found that roughly half of our test users have problems with selecting the right contacts for the right domains.
    • Sometimes it's even unclear for them wether they are working with debtor details or contact details. The system should point out where, how and when they can configure these contacts and it simply does not.
  • Domain config (nameservers)
    • Some labels on buttons are a bit unclear, adding a new nameserver group which states 'save changes' instead of 'add group'
  • Checkout
    • No problems here

To improve the helpfulness and efficiency of the system Exonet should take a look at these remarks that the users made during the execution of the scenarios. We also advise Exonet to rerun the research and protocol but then based on another product like Virtual Server or Hostingpackages. This way more glitches in the system will show.



General Findings

Technical and non-technical global statistics

Technical users

Technical global statistics

Non-Technical users

Non-technical global statistics

Reflection

We have learned quite a bit from this research and research method. We have made some mistakes which had a negative influence on our results. These mistakes are listed below.

  • This research did not have any control group, which would have been essential for better and more valid results. Nevertheless it is a very small research, it should have contained two control groups. One representing the technical users and one representing the non-technical users. We divided and selected our test users on computer experience and created for those groups different scenarios. Each group had up to three scenarios to execute. As soon as we were analyzing our data and trying to make conclusions out of it, we found out that we missed two control groups. These groups would have executed the scenarios from the opposite experience level.
  • Unfortunately there has been a slight miscommunication when executing our scenarios. The technical users only executed one scenario and the non-technical users have executed all three scenarios (which was the original plan). Therefor we are missing a lot of data for our technical users. This was a mistake, but it does not effect the research in many ways. Our conclusion exists of tips and comments for the development-team from Exonet. Still, for more tips and more accurate results it will be helpful for Exonet to conduct some extra sessions with more users of both groups.
  • When organizing the data we found out that we missed a critical point in our data collection. While adding actions we decided it would come in handy if we could define the page of the action. This way we get better results because now we can see per page what goes ok and what goes wrong.

Besides these problems and mistakes we find the Think Aloud Protocol a very useful research method to gather data. The next time when executing this protocol we need to tell the users more specific to say more than they did in this research. We've gathered enough data for this small research, but more comments would have been useful.

Also we found out that the recording software which is integrated within QuickTime on a Mac (probably also on PC) works pretty good. It uses the built-in microphone and camera for recording. These pieces of hardware were exactly what we needed for this experiment.

Other Notes and remarks can be found in every paragraph of the chapter 'Execution'

Sources

http://www.useit.com/papers/heuristic/
http://ozm2.robbinjanssen.nl

Overgenomen van "https://lab.cs.ru.nl/algemeen/index.php?title=Onderzoeksmethoden_2/het_werk/2010-11/Groep03&oldid=129759"