Hulp
Boekenmaker (uitschakelen)

Onderzoeksmethoden 2/het werk/2010-11/Groep06

Uit Werkplaats
Ga naar: navigatie, zoeken

Group Members

Introduction

Due to technical developments, more and more devices are being sold that can be connected to your television. Where in the past people only had a video recorder, they nowadays buy entire home entertainment systems, including components such as DVD recorders and Blu-ray players, digiboxes, dolby surround systems and game consoles. At the same time, the possibilities to connect all those components with each other have taken flight. There are different connections for different types of quality. Furthermore there is, in most cases, made a distinction between the connections for images and sound. All in all those home entertainment systems are much more complex than the old television with its video recorder.

Problem statement

The complexity of home entertainment systems turns out to be so high, that companies selling the previously named components are providing installers. That are people who install the entire home entertainment system for the consumer, mostly as a service. This arises some questions, whereas most components are provided with an extensive user manual, including an installation guide, which should take away the need for such installers. Are people too lazy to go into installing a home entertainment system, are they afraid to damage the expensive components or are the user manuals incomplete? Our research will be facing one of these questions for one component only, because of the limited amount of time. The research question therefore is:

To what extend provides the user manual of a Philips DVDR3575H/31 help by its installation within a high-definition home entertainment system consisting of a television and a DVD player?

Conceptual model

To conduct our research we will use the following way of reasoning:

  • The manual, which will be judged on its helpfulness, consists of sections.
  • When a participant reads a section, he / she may pass on to a certain action.
  • Actions based on a read section will be given a rating, which is positive, neutral or negative.
  • During a session a participant will give comments, which are rated positive, neutral or negative.


Our way of reasoning is further illustrated in the image below:

Conceptual model - Group 6 (13-01-2011).png

Method

Think aloud protocols

Approach

In this section we describe how we intend to perform the think aloud protocols in order to answer our research question. This means that the approach as described in this section is provisional and aspects may change in the course of our research. Those changes are elaborated on in the reflection.

Goal and tasks

We want to know which difficulties a participant experiences with the user manual of a HDD recorder, when that participant has to install the HDD recorder within a pre-existing high-definition home entertainment system consisting of a television and a DVD player. To discover those difficulties we tell the participants that they have just bought a new HDD recorder and ask them to install it within the existing home entertainment system in such a way that a DVD from the DVD player can be recorded on the HDD recorder. To guide the participants towards this goal, we give them four tasks which have to be performed in order:

  • Connect the HDD recorder in such a way that recording form the DVD player is possible.
  • Walk through the initialization process of the HDD recorder and configure it according to the manual.
  • Record a small part of material from the DVD which is inserted into the DVD player.
  • Playback the recorded material on the television.


When a participant enters the room, a short introduction on our research is given. Next the goal and the tasks as described above are explained. Besides this verbal explanation, the participants are provided with a description of both the goal and the tasks on paper which may be consulted at any time during the session. The session is considered complete in the following cases:

  • The particpant has exceeded the maximum time to perform the task. The maximum time for performing the task will be 30 minutes.
  • The particpant has completed all steps tasks.


When the participant doesn't understand a part of the installation manual, the attenant will perform the actions instead or give the participant extra information until the participant can continue again. If an action is performed by the attendant will get a neutral rating.

Facilities

Our existing configuration will be a common configuration found in most livingrooms. The configuration will be a properly installed and wired combination of a DVD player and a TV.

There will be multiple ways to connect the DVD recorder with the existing configuration, depending on the cabels and connections used by the participant. Because we don't want this to be an obstacle for the user, we will provide each installation configuration with all possible cables for audio and video.

There will only be a installation manual available of the DVD recorder, this will be in the native language of the participant. Only the relevant pages will be made avaible to the subject. The sections with pages contained are:

  • Remote control (216-217)
  • Device (219)
  • Standard connection DVD recorder (221-222)
  • Installation and settings (228-229)
  • Optional connections (224)
  • Using the main menu (238-239)
  • Recorder from an external device (252)
  • Playback from the HDD (259)

The full version of the manual can be found here:http://download.p4c.philips.com/files/d/dvdr3575h_31/dvdr3575h_31_dfu_nld.pdf

Location

We will use a location that is appropriate for 4 persons, that is 2 times the amount of persons needed in the room (1 participant + 1 attendant). To simulate a real as possible situation, the existing configuration mentioned will be positioned against the wall in a TV standing, which is common in most living rooms.

Selection of participants

Because we want to know where issues arise when using the manual, we will need users that need the manual as much as possible. We will select participants based upon their level of experience and technical skills, which should both be low. For practical reasons, like time and availabilties of rooms at the university, we will use 3 participants.

Data recording

For data recording, we will use a video camera, which will record the actions as well as the voice of the user.

Performing the TAP

Each TAP will be guided be one attendant. The following steps will be executed:

  • Introduction

The attendant introduces himself and explains the goal of the TAP.

  • General instruction about how to perform TAP

The attendant explains how the TAP will be performed

  • Instruction of the task

The attendant will explain the task to be performed by the participant and hand over the manual and a document which contains the goal and task that need to be performed during the task

  • Execution of the task

The participant will execute the task while the attendant records the actions and reaction of the participant.

  • Outro

The attendant thanks the participant for his participation.

Planning

Week nr Omschrijving Werktijd (per persoon)
Week 42 Onderwerp en onderzoeksvraag bedenken 1 uur
Week 43 Opzet onderzoeksplan opstellen 2 uur
Week 44 Opzet onderzoeksplan opstellen 2 uur
Week 45 Feedback in onderzoeksplan verwerken, Werven van proefpersonen 3 uur
Week 46 Conceptueel model definitief maken 3 uur
Week 47 Voorbereiding TAP 2 uur
Week 48 Uitvoeren van TAPS 3 uur
Week 49 Opnamen uitwerken 5 uur
Week 50 Data analyseren 3 uur
Week 51 Conclusies opnemen in rapport 1 uur

Transcription

Method

The recorded video material of the TAP sessions will be transcribed to XML documents. Each TAP session of a participant will be transcribed in one XML document. One XML document will contain a bundle of nodes that can be linked to a action or comment by the user, or a interception by the attendant.

The corresponding XML schema: 

<participant nr=""> 
	<action>
		<timestamp><!--Timestamp of when the action is performed.--></timestamp> 
		<id><!-- Only fill this in when the action is related to a section that has been read --></id>  
		<description><!-- The description of the action performed --></description>
		<rating><!-- Only if the id is filled in, actions that aren't documented do not need to be rated, the rating of the action: 0 = negative about the manual,	1 = positive about the manual--></rating> 
	</action>
	<comment>
		<timestamp><!--Timestamp of when the comment is performed.--></timestamp>
		<remark><!-- The remark expressed by the participant --></remark>
		<rating><!-- The rating of the comment: -1 = negative about the manual,	0 = neutral or not about the manual,
				1 = positive about the manual --></rating>
    </comment>
	<interception>
		<timestamp><!--Timestamp of when the action is performed.--></timestamp>
		<information><!-- The information given to the participant or the action performed by the attendant--></information>
	</interception>
</participant>

Sections & actions in manual

Remote control

  • Section 1: "Afstandsbediening", page 216-217 (C1S1)

Device

  • Section 1:"Apparaat", page 219 (C2S1)

Standard connection DVD recorder

  • Section 1: "De video-/audiokabels aansluiten", page 221 (C3S1)
  • Section 2: "Optie 1: via de scartaansluiting", page 221 (C3S2)
Action 1: Connect one end of the SCART cable to the EXT1 TO TV connection of the recorder (C3S2A1)
Action 2: Connect the other end of the SCART cable to SCART-IN connection of the TV (C3S2A2)
  • Section 3: "Optie 2: via de S-Video-aansluiting", page 221 (C3S3)
Action 1: Connect one end of the S-Video cable to the S-VIDEO OUT connection of the recorder (C3S3A1)
Action 2: Connect the other end of the S-Video cable to the S-VIDEO IN connection of the TV (C3S3A2)
Action 3: Connect one end of the audio cables to the AUDIO OUT connections of the recorder (C3S3A3)
Action 4: Connect the other end of the audio cables to the AUDIO IN connections of the TV (C3S3A4)
  • Section 4: "Optie 3: via de video-aansluiting (CVBS)", page 221 (C3S4)
Action 1: Connect one end of the composite video cable to the CVBS-OUT connection of the recorder (C3S4A1)
Action 2: Connect the other end of the composite video cable to the COMPOSITE IN connection of the TV (C3S3A2)
Action 3: Connect one end of the audio cables to the AUDIO OUT connections of the recorder (C3S3A3)
Action 4: Connect the other end of the audio cables to the AUDIO IN connections of the TV (C3S3A4)
  • Section 5: "Optie 4: aansluiten op een HDMI compatibele TV", page 222 (C3S5)
Action 1: Connect one end of the HDMI cable to the HDMI-OUT connection of the recorder (C3S5A1)
Action 2: Connect the other end of the HDMI cable to the HDMI IN connection of the TV (C3S5A2)
Action 3: Make a extra connection with your TV while using a SCART, S-Video, composite connection. (C3S5A3)
Action 4: Select the TV channel that corresponds with the choosen connection. (C3S5A4)
Action 5: Finish installation and settings (C3S5A5)
Action 6: Press OPTIONS (C3S5A6)
Action 7: Goto active video output settings and select the HDMI-output as the active video output.(C3S5A7)
Action 8: Select the TV channel that corresponds with the HDMI connection. (C3S5A8)


Optionele aansluitingen

  • Section 1: "Aansluiten op een videorecorder of soortgelijk apparaat" (C4S1)
Action 1: Connect existing attena cable to ANTENNA-IN connection of the recorder. (C4S1A1)
Action 2: Connect one end of the RF-attena cable to the TV-OUT connection of the recorder. (C4S1A2)
Action 3: Connect the other end of the RF-attena cable to the ANTENNA-IN or RF-IN connection of the TV. (C4S1A3)
Action 4: Connect one end of the SCART cable to the EXT1 TO TV connection of the recorder (C4S1A4)
Action 5: Connect the other end of the SCART cable to the SCART-IN connection of the TV (C4S1A5)
Action 6: Connect one end of a another SCART cable to the EXT2-TO VCR/SAT connection of the recorder (C4S1A6)
Action 7: Connect the other end of that SCART cable to the TV OUT of TO TV connection of videorecorder or similar device. (C4S1A7)


Installation and settings

  • Section 1: "Voordat u begint..." (C5S1)
Action 1: Power on TV (C5S1A1)
Action 2: Press STANDBY-ON to power on the recorder (C5S1A2)
Action 3: Choose menu language and press OK (C5S1A3)
Action 4: Select country and press OK (C5S1A4)
  • Section 2: "Groene knop als u door wilt gaan met de volgende installatiestap." (C5S2)
Action 1: Choose screen format (C5S2A1)
Action 2: Start searching chanel (C5S2A2)
Action 3: Press continue (C5S2A3)
Action 4: Set the time settings (C5S2A4)
Action 5: Set the date settings (C5S2A5)
Action 6: Skip guide+ settings by pressing green button (C5S2A6)
Action 7: Setup complete. Close the menu (C5S2A7)


Using the main menu

  • Section 1: "Hoofdmenu", page 238 (C6S1)
Action 1: Press the HOME button on remotecontrol (C6S1A1)
Action 2: Select medium and press OK or NEXT (C6S1A2)
Action 3: Select media (C6S1A3)
Action 4: Pres PLAY to begin (C6S1A4)
  • Section 2: "Mediumgegevens bekijken", page 239 (C6S2)
Action 1: Press the HOME button on remotecontrol (C6S2A1)
Action 2: Select medium (C6S2A2)
Action 3: Press BLUE button (C6S2A3)
Action 3: Press BLUE button again to return (C6S2A3)

Recorder from an external device

  • Section 1: "Opnemen vanaf een extern apparaat (camcorder/videorecorder/DVD-speler)" (C7S1)
Action 1: Set the TV on the channel that corresponds with the recorder. (C7S1A1)
Action 2: Connect the external device to the recorder (C7S1A2)
Action 3: Press CAM button on the remotecontrol (C7S1A3)
Action 4: Select the destination for the recording and press OK (C7S1A4)
Action 5: Select the input channel for the connection of the external device (C7S1A5)
Action 6: Power on the external device, search for the beginning of the intentional recording and put the device in PAUSE mode (C7S1A6)
Action 7: Press RECORD on the recorder to start recording (C7S1A7)
Action 8: Press STOP to stop the recording (C7S1A8)
Action 10: Press LIVE TV to close (C7S1A9)


Playback from the HDD

  • Section 1: "Navigeren op de harde schijf" (C8S1)
Action 1: Press the HOME button (C8S1A1)
Action 2: Select type of content (C8S1A2)
Action 3: Select the sorting option and press PLAY to display a title (C8S1A3)
Action 4: Press the HOME button to close (C8S1A4)

Result

Analysis

Method

With the analysis of the data, only some things are important.

  • the value a subject gives while using the manual in speech, the comments or remarks.
  • the outcome of the actions the user takes after reading the manual

Both of these determine the usability of the section. We will try to make sure the number of interceptions is not too high and interception will mostly be used for things that do not concern the manual, like for example, putting the right channel on the TV. If however we do have to make an interception concerning actions that are stated in the manual, these will normally come after a number of negative actions and negative comments, therefore we will just treat those as extra information.
The same reasoning can be used for the data about the result of the experiment. Did the participant finish to do the assignment or not. If not, this will possibly be due to misreading the manual, leading to negative comments as well as wrong actions.

Variables

  • Value (-1, 0, 1)
  • Outcome (-1, 0, 1)
  • UsabilityOfSection (N)

Formulas

Since the answer on the research question is one variable, we can answer this by one formula. For the rest we will look at seperate figure to see any striking results.

First we take the average of the relevant ratings for the actions and the comments (the ones that are not neutral)
Since the actions and comments are equally important in our view, the next step is to average these two values.

This leads us to the following formula:

UsabilityOfManualForParticipant =
0,5* (SUM(outcome)/(COUNT(outcome)-COUNT(outcome=0)) +
0,5*(SUM(Value)/(COUNT(Value)-COUNT(Value=0)))

If we take the average for all the persons, we will get an idea of the quality of the manual. We could also do this for the entire spectrum, but than more talkative participants will have a bigger weigth in the conclusion.
Please not that since this formula essentially only takes averages from values between -1 and 1, also the result will be exactly between that interval.

Result

The result can be displayed in the following table:


TAP result table
Participant 1 2 3 Total
Comments 186 176 86 450
positive 0 15 0 15
negative 13 25 19 57
neutral 173 138 67 378
Actions 22 29 40 91
positive 11 20 20 51
negative 4 9 17 30
neutral 7 0 3 10
Interceptions 21 20 11 52
Finished no yes no


This results in the following scores for the participants:

TAP result table
Participant 1 2 3 Average
Relevant comments 13 40 19 24
Sum of ratings -13 -10 -19 -14
Comment score -1 -0,25 -1 -0,75
Relevant actions 15 29 37 27
Sum of ratings 7 11 3 7
Action score 0,466667 0,37391 0,081081 0,30919
Average of scores -0,26667 0,0644655 -0,45946 -0,22049

Conclusion

From the analysis performed in the previous chapters, we can derive some interesting conclusions. First to answer our main question. The usability of the manual is scored with a -0,22049 on a scale from -1 to 1. This is very low. The manual therefore is not suitable for a-technical people. This is actually what we expected a bit. However since the TAP was small and the TAP-sessions were quite messy this result can of course be disputed.
Some interesting findings arose from here though. If you compare the action scores with the comment scores. It shows that the first is always graded positive (until 0,46 even for participant 1) while the comment score is always negative. This can have different causes. One is them is the human tendency to prefer complaining above praising. If something goes right, the instruction can be read out loud, deeming it as irrelevant, and an actio can be performed. However as soon as this is not the case, the participant will let it probably know.
Another striking thing is the difference in amount of comments and actions. As can be seen, while participant 3 has not even half the amount of comments the other participants have, the amount of action is almost double. This shows that this participant was not following the think-aloud instruction so well, and was more focused on the task than on the reading. Also in the transcription it shows that the participant had a tendency to call the manual "difficult", "hard to understand" and that she was just not good a these things. This is an interesting side conclusion, because it shows that this reluctant is probably a good cause for the fact that people need an "expert" to install this thing without looking in the manual.

Reflection

This exercise was merely to get some hands-on experience in doing a research with the TAP method. That is why this can not be called an official research. In this chapter we will tell what we learned along the way, problems we encountered and how we would olsve them the next time we will do a TAP.


Conceptual

The fact that usability of a manual can be assessed with comments and action could be true but will need some more investigation. It is hard to tell whether good actions are a result of the manual or of intuitive design or prior knowledge of the subjects tested. Therefore make sure the validity of the TAP is valid before you proceed. In this TAP there was a constant danger of assessing the participants instead of assessing the actual manual.

In the conceptual phase the main issue is to build in a certain amount of flexibility for the TAP. This is hard to estimate. In our research we hoped to directly map sections of the manual to all the things said and done by the participants. This however was a false assumption. People tend to do action being guided by prior knowledge and user interface. Also when something goed wrong, all the actions and comments concerned with fizing the problem are not directly relatedable to the manual. This needed some more flexibility from our side, which we did not foresee.

Method

The method was the messiest part of our research. With TAP you are not suppose to aid the subjects while they are doing the experiment. This requires a lot of preparation beforehand. We will list some issues we encountered and come with a solution for them.

  • users are unable to think aloud and read at the same time. This normally results in silence or reading out loud. Which is great to determine where they are in the manual and the fact that they read the manual, but it does not give a lot of extra information. Besides through reading participants tend to forget that afterwards they should still keep on thinking out loud.
  • It was very hard to make sure the participants would follow the manual and not just go about and try to put cables into holes that seemed to look like a fit. This is something that should be told explicitely.
  • most of the things we told them in preparation was forgotten very fast afterwards. We told them which remote control to use for the TV and which one for the DVD-recorder. However 2 out of 3 participants took the wrong remote control, resulting in an interception. Also they forgot what they should do in the end. Of course our TAP was quite intensive. A lot of neew experiences for the participants, which contributed to this. A good tip is to put everything on paper and tell them to use the paper everytime they encounter something which is not in the manual.


  • Also during the TAP, the facilitators should make sure they are still testing what should be tested. Since it was very hard to convince the participants to actively use the manual, our experiment went in to the direction of testing the usage of manual instead of usability. Or even the ability of the participants. Next time this will be made even clearer to the participants.

Sources

User manual Philips DVDR3575H/31

Summary

Overgenomen van "https://lab.cs.ru.nl/algemeen/index.php?title=Onderzoeksmethoden_2/het_werk/2010-11/Groep06&oldid=129735"