Onderzoeksmethoden 2/het werk/2011-12/Group 2
Inhoud
Problem statement
Nowadays there are several of navigation systems which can be used for navigating while walking. Such a system should be straightforward and efficient for a user. If irregularities occur, blocked roads for example, the navigation system should be able to assist the user in finding another route to reach his/her destination. Of course this should not be a difficult task for the user. This leads to the following research question:
To what degree of usability does pedestrian navigation software support navigating around obstacles that occur en route?
Theoretical framework
Think aloud protocols
To answer this question a Think Aloud Protocol (from now on called TAP) will be used. By letting people use the software and asking them to state what they are thinking while using the software. From a usability designer's point of view this is especially useful because it's a non-interfering method of analyzing his system. The basic approach to TAP is to set a goal for the user and to stimulate the user to talk about the thoughts he has the second they pop into his head. Because analysis will be done after the experiment, it's vital to record the session. This recording can then be used to transcribe and code(?) it. This, in turn, can be be used for for advice on how to improve the system.
Software
Because there are several different navigation systems that support walking a choice has been made to use Google Maps (running on a smartphone) for this research. This choice has been made after trying out several different systems (Google Maps, TomTom Mobile 7, iGo). The choice for Google Maps has been made because of the following reasons:
- Supports pedestrian navigation;
- Runs on most current mobile (smart)phones (which means it is widely used);
- The software is free under license (which means it is available for everyone with a smartphone).
Target group
The target group for this research will be males and females between the ages of 18 and 25 years. This age group has been chosen under the assumption that these people will be able to use a smartphone. People without too much knowledge of navigation software are preferred, to ensure that (1) all test subject have about the same knowledge of the software, and (2) the usability of the software can be tested for people on a "beginner" level. To ensure this, we will ask our potential test subjects to rate their experience with navigation devices on a scale of 1-10. We consider any score under 6 to be a suitable score. The male/female distribution will be 50/50. At least 3 test subjects will be used. The reason this number is chosen is for practical reasons and due to time constraints.
Usability
The research question states that the usability of the navigation software will be evaluated. In order to do this there must be some usability criteria to which the software can be held. Therefore Nielsen's Ten Heuristics for Usability[1] will be used. These heuristics are as follows:
- Visibility of system status;
- Match between system and the real world;
- User control and freedom;
- Consistency and standards;
- Error prevention;
- Recognition rather than recall;
- Flexibility and efficiency of use;
- Aesthetic and minimalist design;
- Help users recognize, diagnose, and recover from errors;
- Help and documentation. [1]
It is not expected that all of these heuristics will be relevant for this research but they will in any case be useful for categorizing user comments.
Conceptual model
Concepts:
- Person
- Obstacle
- Comment
- Heuristic
- Value
Method
With the use of Think Aloud Protocols it is analyzed how test subjects deal with Google Maps navigation software when they need to make changes en route. The following step by step list is an example of how an experiment-session will go.
- Subject is taken to a fixed starting location.
- Subject gets a smartphone with Google Maps navigation (Version 5.11.0).
- Subject gets a specific destination and needs to plot a route towards it using the navigation device.
- Observer will double check the destination and pedestrian mode of google maps. When a mistake has been made, the test subject is asked to retry.
- Subject starts walking towards the destination.
- Once in a while, about 5 minutes apart depending on the route, the observer will tell the subject that a particular turn cannot be taken.
- The observer will track the places marked blocked to make sure the test subject does'nt take that same turn via a detour.
- Subject must find a way around the obstacle using the google maps navigation.
- Subject continues walking to the destination.
During the complete session the test subjects' voice is recorded. The observer will also be recorded in the same recording when he/she declares a part of the way inaccessible. The navigation device is programmed to track the subjects route.
A set of rules is declared for the duration of the experiment. The goal of these rules is to force the test subject to interact with the navigation device.
- The test subject is not allowed the deviate from the direction the navigation suggests. He/she is however allowed to alter the route to find a more fitting way.
- Turns, crossings can be declared inaccessible by the observer. Streets can not, to minimize the chance of "locking up" the test subject.
- The observer needs to declare a crossing inaccesible every five minutes or so, but only when the test subject is not already taking a detour.
- The observer may only intervene when the time table for the experiment is in danger (for example, when the test subject is really messing up).
- The goal is to reach the destination in 30-45 minutes.
- Recording starts before and ends after the experiment.
After these experiments the recordings are transcribed. The transcription specifies who was saying what, and globally at which time in the recording the sentence was spoken. The transcriptions in turn are labelled. The labels specify which remarks re This gives an impression how the users experienced the system, in other words, how well the system performs from a usability perspective. Lastly, we will choose the most discussed heuristics, and will deduct the problems most encountered. One or more relevant text fragments will be added for argumentation. For each problem an advice will be given.
The conclusion is based on the serverity of the problems encountered for the heuristics.
Results
The results of the two experiments:
- Map 2 Note: this route started at the Tichelaar 5 like the first one, only gps tracking failed for the first 3-4 minutes.
|
Category
|
Total
|
Positive
|
Negative
|
Negative %
|
Note
|
|
Visiblity of system status
|
5
|
3
|
1
|
20%
|
Generally neutral to slightly positive remarks. Ofcourse the phone status bar is always visible, and is also referred to. This category will be dealt with further in the conclusion.
|
|
Match between system and real world
|
11
|
0
|
11
|
100%
|
This category will be dealt with further in the conclusion.
|
|
User control and freedom
|
8
|
3
|
5
|
62,5%
|
This category will be dealt with further in the conclusion. |
|
Consistency and standards
|
3
|
0
|
3
|
100%
|
|
|
Recognition rather than recall
|
4
|
0
|
4
|
100%
|
This category will be dealt with further in the conclusion. |
|
Flexibility and effiency of use
|
10
|
1
|
7
|
70%
|
This category will be dealt with further in the conclusion.
|
|
Aesthetic and minimalist design
|
2
|
1
|
1
|
50%
|
|
|
Help users recognize, diagnose, and recover from errors
|
2
|
1
|
1
|
50%
|
|
Conclusion
Visibility of system status
First the partial conclusions are listed:
|
Problem
|
Description of problem
|
Example
|
Advice
|
|
No overview
|
Although users were mildly positive about status indicators, it seems the major complaint about this heuristic is the fact that there is no clear overview. Users saw that there were status indicators, and what they meant. But they got confused as there was no one place to find all of them. |
Ik kan nog wel tegelijkertijd zien dat de batterij van de telefoon "rapide" achteruit gaat. Het is wel fijn dat ik in ieder geval de basisfuncties van de telefoon kan zien. Op mijn scherm is nog steeds geen enkele straat te zien behalve de route die we moeten nemen. En daar lopen wij op het moment naast.
|
The advice is to create one central button to ask the vital system indicators. Things like battery status, GPS and mobile internet reception all have to be listed. Also an indicator when the system is busy should be added, as sometimes the application briefly `hangs` only to continue shortly after. |
Match between system and real world
|
Problem
|
Description of problem
|
Example
|
Advice
|
|
Lack of roads
|
One of the bigger problems with the Google maps navigation is that it lacks many pedestrian-only roads.
|
Op het moment, ehhh... baal ik weer enorm. Want ik zie op de kaart dat er geen mogelijke route is tussen de Zuil en de plek waar ik nu ben. Maar als ik naar links kijk dan zie ik een voetpad lopen, waar ik vrolijk tussendoor had kunnen lopen. Maar google maps geeft mij deze niet aan. Dus loop ik om.
|
The advice is fairly obvious, although mostly a licensing problem – the solution is for Google to buy a license for map(s) that do include pedestrian roads. (Note: google maps produces it's own maps from satellite photo's.)
|
|
Lack of detail in existing roads
|
For the user to recognize how the shown model corresponds with the real world, it’s important that the model is as accurate as possible.
|
Maar eeh, er zit inderdaad weer een knik in de weg zoals het op de kaart staat. Het is wel raar dat knik in realiteit anders is dan op de weg. Dus als je de werkelijkheid ehh.. van het scherm vertrouwt...dan kan je wel eens raar staan te kijken als je omhoog kijkt en ziet hoe het werkelijk is.
|
Improve the accuracy of the maps
|
|
Inaccurate position
|
Sometimes the location on the map does not correctly correspond to the user’s actual position.
|
Alleen de pijl zegt niet dat we bij de kruising zijn. Ik denk dat de euh...de pijltjes...op het moment nog niet veel de weg volgen. (note: GPS reception was very good at the moment) |
|
User control and freedom
|
Problem
|
Description of problem
|
Example
|
Advice
| |
|
User is not always in control, system forces the user to trial-and-error
|
The user knows what he wants to do and mostly where to go but the systems is more in control than the user is. It provides the user with all kind of ‘useful’ options which confuse the user to do and select what he wants. Because of the confusion the user starts to try (trail-and-error) to accomplish his goals (accidently).
|
Ah, ik heb nou op een knop geduwd waardoor ik toevallig het eindpunt aan kan passen. Ehmm het zou handiger zijn als ik een tussendoorpunt aan kan geven, maar dat is er niet. Dan ehhh... Eh....dan gaan we zo naar de Leigraaf. Dan gaan we dat intoetsen. Het touchsscreen heeft er geen zin in. Oh, ik duw op een verkeerde knop. Een punt op de kaart. Heb ik geduwd. Het was eigenlijk meer omdat er geen intoetsmogelijkheid voor was. Maar ik hoop dat als ik nou aangeef...Jaaa. We kunnen een punt aangeven. Ik heb 'm geselecteerd. Google maps ziet het niet, maar via een andere route komen we ... toch op dat punt. Maar dat is niet de kortste route. Hmmm... Dit is klote. Maar het moet maar, want google maps geeft mij geen andere optie, terwijl ik toch met mijn hoofd duidelijk zie dat het beter kan. Dus ik klik op navigeer. En... zie het spijtig genoeg...moeten we weer een stukje terug. En hij geeft weer aan dat we rechtsaf moeten. Nou wordt het gezellig, want ik zie volgensmij een doodlopend stuk. Maarja. Google maps zegt dat dit kan. Hoe moet ik nou dan terug? Pijltje terug, netjes. Jij weet het zelf ook niet joh.. Routeinfo, misschien is dat wel iets. Ik druk even op die, alternatieve routes ha! |
The solution is to put the user back in control by providing settings whether or not providing the user with these ‘useful’ options. Implementing a possibility for the user to save his own preferences which are logically for him could also be helpful.
|
|
Recognition rather than recall
|
Problem
|
Description of problem
|
Example
|
Advice
|
|
Difficult to remember workflow
|
The users had difficulty remembering what exactly they had previously done. This made it difficult for them to re-plan the route, as they had to figure out how to navigate again and again. It was also found difficult to recognize the function of buttons by their icons and symbols. |
Ik duw weer op de verkeerde knop, ik vergeet nog steeds ik linksom mijn eindbestemming aan kan passen. We hebben als het goed is de kruising omzeild, dus als eindpunt stel ik weer de Zuil in. En...ja. We hebben weer een andere route. Dus ik duw op navigeren. En... hij geeft me weer een route.
|
We advise a revision of navigation workflow and the used icons and buttons. Then the users will learn quickly how to perform at least the basic functions of navigating somewhere. By using intuitively chosen buttons, icons and symbols user will immediatly recognize what kind of functionality they can use. When they have used it before they will remember well chosen icons more easily. |
Flexibility and efficiency of use
|
Problem
|
Description of problem
|
Example
|
Advice
|
|
Advanced options are difficult to use while normal options give unwanted results.
|
The problem is that the user knows what he wants the system to do, but finds it difficult/impossible to use that option. This forces the user to use a less advanced option that is easy to use, but causes the user (in this case) to walk a route that is not optimal.
|
Nou, ja, ik zie zelf al een route. Ik hoop dat google maps die ook vindt. Anders dan ehh. Is het wel heel erg kut. We gaan terug. Naar...navigeren. Ik krijg een mededeling tussendoor dat ik ook WiFi kan inschakelen om beter geloceerd te worden, maar....nee...dankje. Ik snap niet wat daaraan relevant is. Ja...op google maps zie ik de weg ophouden, maar het liefst zou ik nu een via punt aangeven, zoals ik al eerder probeerde. Of ik heb deze optie niet gevonden, of ik ben...of het is gewoon echt niet mogelijk. Maar...ik ga gewoon weer via een punt op de plaats een nieuwe eindlocatie.. ik hoop eerlijk gezegd dat google maps ziet dat als ik zeg "rechtdoor naar de grote weg ... Ik heb 'm geselecteerd. Google maps ziet het niet, maar via een andere route komen we ... toch op dat punt. Maar dat is niet de kortste route. Hmmm... Dit is klote. Maar het moet maar, want google maps geeft mij geen andere optie, terwijl ik toch met mijn hoofd duidelijk zie dat het beter kan. Dus ik klik op navigeer. En... zie het spijtig genoeg...moeten we weer een stukje terug. En hij geeft weer aan dat we rechtsaf moeten. Nou wordt het gezellig, want ik zie volgensmij een doodlopend stuk. Maarja. Google maps zegt dat dit kan.
|
The advice in this case is to make it easier for users to determine their own alternative route when they do not want to use the route that is offered to them. For example, when the system suggests an alternative route it could also have an option that says "create own alternative". |
Final words
Let's start with the conclusion: google maps is pretty bad when it comes to the usability aspects of navigating around obstacles encountered en route.
Why is this? For about 5 of the 10 heuristics that describe "good" usability aspects, we found concerning problems. Workflows in the software are a mess. It is difficult to recognize functions of all icons. Basic navigating functionality like navigating around a certain point is absent. Almost to the point that is seems Google intended the software for one route planning only, without expecting changes in anything but the destination. When diverting from the original route, the software does not even change the path one has to follow to get back on track.
The potential of the software is there, but many things have to be changed before google maps navigation satisfies the 10 heuristics adequately.
Future research
We recommend future research in the same area. Our main shortcoming was the amount of test subjects. Also, we skipped normal navigating by going straight for navigating around obstacles. It is possible to check for differences between the usability of normal navigation and obstactle navigating. Lastly, we did not have enough test data to research problem solving. This is a large area of interest for navigation, so an iteration of our experiment with the focus on problem solving will definately deliver interesting results.
References
Planning
October
- Define a destination √
- Make one or more test runs √
- Refine research question √
- Choose between simulation and real world conditions √
November
- Execute at least 2 experiments √
- Transcribe data √
- Label data √
December
- Complete wiki √
- Publish results √
- Reach conclusion √
