Onderzoeksmethoden 2/het werk/2007-8/Groep08

Assessing commercial website usability

      Jörg Mutter, s0457027, Information Science
      Ilona Wilmont, s0412600, Information Science 
      Eddy Klomp, s0413348, Information Science

---

Research Log

Introduction

In our modern society online activities are becoming increasingly prominent. Conversations are moving to chatrooms, banking activities are all available through the Internet and even for shopping we no longer have to set foot outside.

The latter phenomenon forms the focus of this project. E-commerce, or the sale of products through the internet, happens on a large scale. Websites facilitating sales, auctions and advertisements are widespread and widely used.

E-commerce is typically aimed at a wide and variable user group. This makes that a website facilitating such buying behaviour must be of an extremely flexible nature so that all users can easily work with it.

This project aims to find out whether large-scale commercial websites have succeeded in achieving this goal.

Problem statement

A highly variable user group places specific requirements on the usability of websites developed for e-commerce, since they should be highly intuitive and pleasant to use if they are to appeal to such a large group.

In our research we aim to assess usability of the commercial website eBay by comparing the behaviour two different user groups will show while interacting with eBay. Several factors are of importance for measuring and judging usability. These are, as formulated by De Haan et al. :

• Easy-to-use
• Easy-to-learn
• Efficient
• Fun-to-use

These conditions must be satisfied for an application to be considered as usable. The main aim is therefore to find out to what extend users think eBay satisfies these conditions. This project is thus of an evaluative nature.

This leads to the following research question:

What differences in efficiency, learnability, ease of use and positive affect can be observed when adults and younger people use the commercial website eBay for the first time?

Theoretical background

Elements

Within our domain the following research elements can be defined:

• Adults: For this experiment, we define this user group as people who are 40 years of age or older.
• Younger people: This user group constitutes of people who are between 18 and 25 years old.
• Commercial website: Website used for the purpose of selling products and making a profit.

Variables

• Differences: Differences in efficiency, learnability, ease of use and positive affect.
• Efficiency: Is the application time-efficient, is the effort required for operation in proportion to the service it provides and does it provide the desired results?
• Learnability: How easily can application be used and understood, either without prior instructions or with simple clues. This corresponds to the usability criterium of ’easy-to-learn’.
• Ease-of-use: Being able to utilize the application without any problems, be it visibly or implicitly.
• Positive affect: Users having positive associations and experiences after using the application. A vital factor for having them willingly re-use the website.

Method

The research question can be answered in several steps. These steps, outlined in this chapter, are made before the actual research started. Later on we will provide you with details about what we really did in these steps. These are the initial steps globally outlined:

1. Building a conceptual model: An ORM domain model describes the Universe of Discourse related to this research question. It describes the input, analysis and output.
2. Recruitment of volunteers: If one wants to make a valid research a minimum of 5 young people and 5 adults are needed, which do have a basic knowledge about using the internet. These people must be recruited by the researchers.
3. Set tasks: Some tasks must be performed by the volunteers.
4. Collect data: We ask the volunteers to comment on their thoughts during the execution of some task one by one. This must be done on in a non-disturbing environment. The computers must have a workable connection with the internet. The result is a video of each person interacting with the computer.
5. Record data: We will record the data via a video tape. After this we transcribe the text.
6. Structure data: We will link bits of text to task related categories via text analyses. A database is set up where the results will be stored. This database must be conform the ORM domain model, presented in chapter 2.
7. Analyse data: To derive conclusions, one needs a close look to the population of the ”Expression shows Differences” fact type.
8. Write a report: The outcome of the research is presented in a report. The main goal of this report is to answer the research question.

Indicators

1. Easy to use:
   1. nr. of errors during task completion
   2. verbal mentions of explicit frustrations during task completion
   3. verbal mentions of ease during task completion
   4. total time to task completion
   5. facial/bodily expressions of positive affect
   6. facial/bodily expressions of negative affect
   7. behavioral expressions of difficulty (lots of pauses to search, inability to find options, use of loud voice)
   8. behavioral expressions of ease (smooth task completion)
2. Easy to learn:
   1. nr. of explicit instructions given
   2. kind of instructions given
   3. nr. of clues given
   4. kind of clues given
   5. nr. of questions asked
   6. kind of questions asked
3. Efficient
   1. total time to task completion
   2. verbal mentions of ease during task completion
   3. behavioral expression of ease (smooth task completion)
4. Fun to use
   1. verbal mentions of positive affect during task completion
   2. facial expressions of positive affect

Domain Model

A verbalization of the ORM domain model is as follows:

• User groups perform Tasks.
• User groups consist of Adults and Adolescents.
• Adults is a User group with defining property Age of 40 years or older.
• Adolescents is a User group with defining property Age between 18 and 25 years.
• Performance is measured by Usability criterium
• Usability criterium measures Indicator (ease of use, learnability, efficiency, positive affect)
• User groups have Differences in Indicators.

Actual steps

Recruitment volunteers

Initially we hoped to recruit 10 volunteers. Actually six volunteers were recruited, because of time constraints. We didn't contact them by e-mail but we asked them personally. Personal recruitment improves people's willingness to co-operate. We succeeded in recruiting 4 volunteers. This, because of time constrainst. Regretfully this does reduce the reliability of our research.

Task setting

The tasks are all in Dutch, so it will be understandable for the volunteers. The volunteers will be asked to perform the following tasks:

1. creating an Ebay-account: A password must be created as well. Therefore we must tell the volounteer if he has any objection being filmed.
2. Searching for a certain article via product groups: The article is the Samsung Yepp YP-MT6 MP3-speler.
3. Searching for the same article via the search facility: The article is the Samsung Yepp YP-MT6 MP3-speler.
4. Placing a bid for that article: The article is the Samsung Yepp YP-MT6 MP3-speler.
5. Selling an article:
   1. Title: Samsung Yepp YP-MT6 mp3-speler
   2. Memory: 1 GB
   3. Kind of memory: Flash
   4. Brand: Samsung
   5. Colours: 1 kleur
   6. Extra functions: Radio, stemopname
   7. Pc-interface: USB 2.0
   8. New/second-hand: 2e hands
   9. Text: dit is nep
   10. No picture, no visitors counting
   11. Price: 50 euro
   12. Time: 3 dagen
   13. Accept Pay pas, give email
   14. Banktransit: mark the first option
   15. Shipping costs: 6,20 euro
   16. Push button doorgaan, after this stop this assignment.
6. Searching a certain topic in the Help-section of the site: What are the shipping cost in Holland, for a package less then 10 kilo (basispakket).

Collecting data

We asked the volunteers to comment on their thoughts during the execution of the task described in the previous section. This is done in a non-disturbing environment. In most cases it was in their own environment, where volunteers should be comfortable enough to complete the tasks. In some cases a computer was provided, because of compatibility issues with their own computer. Also a video camara and a webcam was provided. The computers had a working connection with the internet.

Recording data

Data was collected via video and webcam. A video was used to record computer-related tasks (for example: number of errors) and a webcam was used to record the human-related tasks (for example: facial expressions).

Transcribe data

As all data must be assigned to an indicator, one can transcribe the data and catagorize pieces of text to an indicator. We will do the transcription in only one case, because it also takes a lot of work to transcribe both verbal- and non-verbal-expressions, like facial expression. These indicators are necessary for answering our research question. We also think our indicators are not associated directly with text, but are associated with counting pieces of verbal- and non-verbal-expressions. This can also be done by analysing the video and the webcam, without transcribing the text. Another problem is that a video / webcam record can't be displayed on this wiki, because of its size. We acknowledge the fact that not transcribing and having no video record on this wiki makes our research less provable. Should someone ask for this record we always can provide this on a memory stick.

Data transcription

Notes:

• The transcription is in Dutch since all our participants were Dutch
• INDx.x refers to an indicator with corresponding number
• All other indicator results are presented in an Excel sheet, which can't be uploaded. Here is however a screen example: Media:TaskIndicatorCaseRelation.jpg

Ehmm... Even kijken, ik moet een account aanmaken. Dan moet ik me dus ergens in gaan schrijven. Even zoeken... [scrolt door site] [IND1.6] Oh, ik zie hierboven iets van 'Mijn eBay'. Dan kan ik daar eens op klikken. [klikt link Mijn eBay] [IND1.5][IND1.1] Even zien, ik kan hier inloggen... Nee... [leest tekst op site][IND1.7] Hieronder... Oh, 'Nu inschrijven'. [IND1.5] Dan zal ik dat eens doen. [klikt knop Nu inschrijven][IND1.8]. Oh nee, moet ik dat allemaal gaan invullen? [IND1.6][IND1.2][IND2.5][IND2.6:ask confirmation] ... Naam... Achternaam... Waarom doet hij dat nu niet? (cursor naar volgende veld laten springen) [IND1.2][IND1.6] Hè, dit gaat niet... ik klik toch?...[IND1.2] [IND1.7] Nou, e-mailadres, wat was dat ook alweer?... en dan nog een keer... Klaar, hoe ga ik nu verder? [drukt op Enter] Hoe kan dat nou, ik krijg precies hetzelfde scherm... [krijgt hint dat lijst nog langer is en dus nog niet volledig ingevuld is] [IND1.1][IND2.1][IND2.2:hulp basishandeling] Dus dan moet ik alles weer opnieuw gaan invullen? [IND1.2][IND1.6][IND1.7][IND2.5][IND2.6:ask confirmation] Verzin een gebruikersnaam, wat zal ik doen? Gewoon Hans... Oh nee, die bestaat al. [krijgt hint om er cijfercombinatie achter te plakken voor uniqueness] [IND2.1][IND2.2:hulp basishandeling] Wachtwoord... Ehmmm [typt wachtwoord in] En dan nog een keer... Geheime vraag... Nou, die met dat huisdier dan maar... [typt naam huisdier in] [IND1.8] Geboortedatum... Moet ik nu nog verder naar beneden? Nee, wat stom, hij doet het echt niet. [IND1.2][IND1.6] Hij wil niet naar het volgende veld... [IND1.2][IND1.7] Kan ik hier veilig mee akkoord gaan? [IND2.5][IND2.6:ask confirmation] Nou, inschrijven dan... Dus nu ben ik ingeschreven, nou handig zeg, dat was al een hele klus. Als dit gebruikersvriendelijk moet zijn... [IND1.2][IND1.6]

Totals:

• IND1.1 = 2
• IND1.2 = 7
• IND1.5 = 2
• IND1.6 = 6
• IND1.7 = 4
• IND1.8 = 2
• IND2.1 = 2
• IND2.2:hulp basishandeling = 2
• IND2.5 = 3
• IND2.6:ask confirmation = 3

Nouja, opdracht 2, wat moet ik nu doen? Een artikel zoeken. Samsung Yepp YP-MT6... wat is dat uberhaupt? [krijgt hint mp3speler] [IND2.1][IND2.2:verduidelijking opdracht][IND2.5][IND2.6:vraag om verduidelijking opdracht] En ik moet nu zoeken via Rubrieken... Auto's, Baby, Boeken, nee.. Oh Consumentenelektronica. [IND1.5] Dat zal hem dan wel zijn. [klikt link Consumentenelektronica][IND1.3][IND1.8] Oh, en ehmmm... subrubrieken... MP3-spelers en portables. Dat lijkt me de goede....[IND1.5][IND1.8] Oh jee, krijg ik nu een hele lijst... Dan ben ik nog wel een paar dagen bezig...[IND1.6][IND1.7] En dan... Sorteren, dat kan ook... [klikt op drop-down list van sorteren][IND1.1] Nee, daar schiet ik niets mee op. [IND1.6] Het staat ook allemaal niet op alfabetische volgorde ofzo... Dit is echt heel onhandig, als onervaren persoon vind ik dit heel ingewikkeld.[IND1.2][IND1.6][IND1.7] Misschien kan ik vergelijken... [klikt op knop Vergelijken][IND1.1] Nee, hoe ga ik nou weer terug? [krijgt hint druk op Backspace] [IND1.6][IND2.1][IND2.2:hulp basishandeling][IND2.5][IND2.6:vraag basishandeling] Is dit alles? Hij staat er niet bij.... [IND1.7] Hee wacht, hiernaast hebben we een zoekvenster... Land Nederland... Straal... 15 km... zoeken! [vult velden in en klikt op knop Objecten weergeven][IND1.1] Wat hebben we hier... Dit zijn allemaal... Wat zijn dit?... [IND1.7] Nee, nou krijg ik hem nog niet. Ik wil gewoon op merk kunnen sorteren... [IND1.2][IND1.6] Ik pak gewoon dat zoekbalkje hierboven... Samsung Yepp YP-MT6... [typt in en drukt op Enter] Ja, nu krijg ik hem... [krijgt commentaar dat dat niet helemaal de bedoeling was van zoeken via Rubrieken] [IND1.5][IND2.1][IND2.2:correctie achteraf] Ja, maar anders is het een onmogelijke opgave, ik vond de structuur op de pagina niet duidelijk, er zou op z'n minst alfabetisch gesorteerd moeten zijn.[IND1.2]

Totals assignment 2:

• IND1.1 = 3
• IND1.2 = 3
• IND1.3 = 1
• IND1.5 = 3
• IND1.6 = 5
• IND1.7 = 4
• IND1.8 = 2
• IND2.1 = 3
• IND2.2:verduidelijking opdracht = 1
• IND2.2:hulp basishandeling = 1
• IND2.2:correctie achteraf = 1
• IND2.5 = 2
• IND2.6:vraag om verduidelijking opdracht = 1
• IND2.6:vraag basishandeling = 1

Wat is de volgende opdracht? Zoeken via zoekfunctie.. ow, dat heb ik dus net gedaan... [IND1.5] gaan we weer... Nou, dan ga ik zoeken op.. mp3-speler... M-P-3-D.. nee... S-P-W .. [vult zoekveld in]... jaja, ik typ heel snel, zo snel dat klikken sneller gaat... [maakt typefouten][IND1.2][IND1.6] ... E-L-E-R.. nouja, en nu zoeken. [klikt op knop Zoeken] Nu krijg ik weer zo'n hele lijst... [krijgt hint om op merknaam te zoeken][IND1.2][IND1.6][IND1.7][IND2.3][IND2.4:verder helpen] Owja, natuurlijk... Samsung Yepp YP-MT6 [vult veld opnieuw in] en zoeken.... [klikt weer op Zoeken]. Ja, dit is veel makkelijker... Nu heb ik hem gelijk... [IND1.3][IND1.5][IND1.8] Volgende!

Totals opdracht 3:

• IND1.2 = 2
• IND1.3 = 1
• IND1.5 = 2
• IND1.6 = 2
• IND1.7 = 1
• IND1.8 = 1
• IND2.3 = 1
• IND2.4:verder helpen = 1

Een bod plaatsen... ehmmm... Nou ik heb hem hier al voor me... [het overzicht waar alle mp3spelers op een rij staan] Waar moet ik nu weer op klikken.... [scrolt op en neer op pagina][IND1.6][IND1.7]... Geen idee...[IND1.2] Vergelijken misschien weer... [IND1.1][IND1.6] Nee, ik kan nergens bieden... [krijgt hint om object aan te klikken][IND1.6][IND1.7][IND2.3][IND2.4:help with type of action] Owja, sorry ik kijk altijd maar half...[klikt op object] Hmm... nou dat zal dan wel deze zijn... Bod plaatsen? [klikt op knop Bod plaatsen][IND1.5] [IND2.5][IND2.6:ask confirmation] Ja... maximumbod... en dan doorgaan...[IND1.3][IND1.5][IND1.8]

Totals opdracht 4:

• IND1.1 = 1
• IND1.2 = 1
• IND1.3 = 1
• IND1.5 = 2
• IND1.6 = 3
• IND1.7 = 2
• IND1.8 = 1
• IND2.3 = 1
• IND2.4:help with type of action = 1
• IND2.5 = 1
• IND2.6:ask confirmation = 1

Opdracht 5, we komen ergens... hmmm... het artikel verkopen. dus dan zal ik het wel weer op moeten zoeken. nou ik weet nu hoe dat moet, dus... [typt artikelnaam in in zoekbalk]. ja, daar hebben we hem weer... [IND1.5] ehmm... even zien.. [krijgt hint dat dit fout is en terug naar homepagina moet gaan][IND1.1][IND2.1][IND2.2:ingreep] oh... [scrolt heen en weer][IND1.7] oh, kopen, verkopen... daar moet ik heen... [klikt op knop Verkopen][IND1.8] iets invullen over het object.. weet ik veel... nou mp3speler Samsung dan maar... en dan Beginnen met verkopen [klikt op knop Beginnen met verkopen][IND1.8] wat is dit nou weer? [IND1.6] Rubriek kiezen... maar hier staat helemaal geen geschikte rubriek bij... [krijgt hint om tabblad Bladeren door rubrieken te raadplegen] [IND1.2][IND1.7] oja, ik kijk maar half... oh, hier kan ik iets selecteren... consumentenelektronica... en nu? oh, met de scrollbar opzij... ja... geluid... weer opzij, wat is dat nou [schermresolutie niet helemaal goed afgesteld][IND1.2][IND1.6] MP3-spelers en portables... ik denk dan maar Overige MP3-spelers... oh, een berichtje.. ik heb de rubriek geselecteerd... mooi... [klikt op Doorgaan][IND1.3][IND1.8] oh nee, moet ik nu al die gegevens invullen... [IND1.2] en dan moet ik hier... [begint velden in te vullen, onverstaanbaar gemompel]...[IND1.6] volgende... je moet zo goed kijken op welke knoppen je drukt... [IND1.6] en dan heb ik hier maar een half scherm... nou eindelijk... [IND1.2] en dan moet ik op Doorgaan klikken? [IND1.7] Kan dat zomaar? Ik heb hier van alles ingevuld [krijgt benadrukt dat het niet echt doorgezet wordt][IND2.1][IND2.2:reassurance][IND2.5][IND2.6:ask confirmation] nou goed... op jouw verantwoording... [klikt op Doorgaan] en dan nu stoppen? hoe dan? [krijgt hint om ergens op willekeurige link te klikken om naar andere pagina te gaan][IND1.6][IND2.1][IND2.2:ingreep]

Totals opdracht 5:

• IND1.1 = 1
• IND1.2 = 4
• IND1.3 = 1
• IND1.5 = 1
• IND1.6 = 5
• IND1.7 = 3
• IND1.8 = 3
• IND2.1 = 3
• IND2.2:ingreep = 2
• IND2.2:reassurance = 1
• IND2.5 = 1
• IND2.6:ask confirmation = 1

De laatste dan... [slaakt zucht][IND1.6][IND1.7] verzendkosten Nederland voor pakketten onder de 10 euro... [scrolt langzaam naar beneden] hee... Verzendcentrum... [IND1.5] ik denk dat die het is... [klikt op logo Verzendcentrum][IND1.3][IND1.8] Ja, zie je wel... [IND1.5] nou, dan zal het TNT Post wel zijn... [klikt op link TNT Post][IND1.8] even zien, basispakket he? [klikt op link Basispakket][IND1.8] Ja hier... 0-10 kg 6,20 euro...[IND1.3][IND1.5] Was dat alles? ... [IND1.3][IND2.5][IND2.6:ask confirmation]

Totals opdracht 6:

• IND1.3 = 3
• IND1.5 = 3
• IND1.6 = 1
• IND1.7 = 1
• IND1.8 = 3
• IND2.5 = 1
• IND2.6:ask confirmation = 1

[wordt gevraagd om wat samenvattend, evaluerend commentaar] Nou, ik vond het voor de eerste keer wel een moeilijke taak, maar ik denk als ik het nu nog eens zou doen dat het dan een heel stuk makkelijker zou gaan. Op zicht is het een prima site, en ik denk ook wel gebruikersvriendelijk. Alleen dat zoeken via rubrieken vond ik helemaal niets, maar ik denk dat je dat niet vaak zult doen als je een bepaald iets in je hoofd hebt.

Structure data

All data have been structured in a database, which was set up according to the above ORM-model. An SQL-dump of the database can be found here.

Analyse data

To analyse the data, a certain weight is assigned to each indicator. Each positive indicator is multiplied by 1, each negative indicator by -1, except for the number of errors and the number of explicit instructions. These should weigh slightly heavier in our opinion.

The resulting table of weights:

Indicator	Weight
Ease of use:
nr. of errors during task completion	-2
verbal mentions of explicit frustrations during task completion	-1
verbal mentions of ease during task completion	1
total time to task completion
facial/bodily expressions of positive affect	1
facial/bodily expressions of negative affect	-1
behavioral expressions of difficulty	-1
behavioral expressions of ease	1
Ease of learn
nr. of explicit instructions given	-2
kind of instructions given
nr. of clues given	-1
kind of clues given
nr. of questions asked	-1
kind of questions asked
Efficiency
total time to task completion
verbal mentions of ease during task completion	1
behavioral expression of ease	1
Fun of use
verbal mentions of positive affect during task completion	1
facial expressions of positive affect	1

As becomes clear from the above table, we have left some indicators out of the equation. In the case of the kinds of questions, clues and instructions, this is due to the fact that we did not gather sufficient data to ojectively quantify these results. In the case of the total time to task completion, we did feel that the differences in internet speed (remember, the experimens were performed at different locations) might have had too much influence on the results.

These are the average results per task per indicator per user group type:

Task	Indicator	Adolescants	Adults
create_account	ease_of_use	-5	-14
place_bid	ease_of_use	0	-12.5
search_groups	ease_of_use	-1	-12.5
search_help	ease_of_use	-47	-5
search_search	ease_of_use	1	-0.5
sell_article	ease_of_use	4	-13
Totals ease of use		-48	-57.5
create_account	ease_of_learn	-3.5	-6
place_bid	ease_of_learn	0	-4.5
search_groups	ease_of_learn	0	-13.5
search_help	ease_of_learn	-2	-1.5
search_search	ease_of_learn	0	-2
sell_article	ease_of_learn	-2	-12.5
Totals ease of learn		-7.5	-40
create_account	efficiency	1.5	1
place_bid	efficiency	0	0.5
search_groups	efficiency	0.5	1.5
search_help	efficiency	0.5	0
search_search	efficiency	0	0.5
sell_article	efficiency	1	0.5
Totals efficiency		3	4
create_account	fun_of_use	0	0
place_bid	fun_of_use	0	0
search_groups	fun_of_use	1	0
search_help	fun_of_use	2	0
search_search	fun_of_use	0.5	0
sell_article	fun_of_use	1.5	0
Totals fun of use		4	0

Chronological results per indicator

Ease of use

Ease of learn

Efficiency

Fun of use

Conclusions

What differences in efficiency, learnability, ease of use and positive affect can be observed when adults and younger people use the commercial website eBay for the first time?

Differences in ease of use

At a first glance, there seems to be little difference in the ease of use of eBay between adults and adolescants, since their total scores differ little. However, the individual scores and the graph tell us that the result for searching a help topic is extraordinarily high for the adolescants. They score better in all other departments, but their scores plummet during the last task. Because of the small number of test results, we cannot give a good explanation for this.

One reason may be that the adolescants focused too much on the time factor in the end and wanted to end the experiment hastily. Or maybe they felt like they knew the site well enough and became idle.

If we had to base a conclusion solely on our test results, we would say that adolescants found eBay easier to use than adults.

Differences in learnability

According to our results, younger people found it easier to learn the use of eBay. There are no really abnormal results in this department, although it is noteworthy that adults struggled a lot with selling an article and searching via product groups. These tasks were perceived as being the most difficult by all of our test subjects.

Had these results been based on more test results, it would have been safe to conclude that younger people find it easier to learn how to use eBay than adults.

Differences in efficiency

In the department of efficiency, there are only minute differences between adults and younger people. There are no extraordinary results here either. Based only on our test results, we conclude that there is little to no difference in the efficiency with which adults and younger people use eBay.

Differences in positive affect

What becomes immediately clear from these results, is that there was not a single mention of positive affect by adult people. Younger people seem to enjoy using a new site on the internet just a bit more. So looking at our test results we conclude that there is a difference in positive affect between adults and younger users of eBay.