Onderzoeksmethoden 2/het werk/2010-11/Groep08
Group Members
Inhoud
- 1 Introduction
- 2 Problem statement
- 3 Conceptual model
- 4 Method
- 5 Approach
- 6 Planning
- 7 Data structurering
- 8 Data analysis
- 9 Conclusion
- 10 Reflection
- 11 Summary
- 12 References
Introduction
Blackboard Learn™ [1] is a web-based course-management system that enables students and faculties to participate in classes delivered online or to use online materials and activities to complement face-to-face teaching. With Blackboard, instructors or lecturers can provide students with course materials, discussion boards, virtual chat, online quizzes, an academic resource center, and more. The degree to which Blackboard is used in a course varies from course to course and from instructor to instructor. For example, instructors may supplement an on-campus course by putting the syllabus and handouts on Blackboard.
Nowadays, Blackboard Learn has become the mainstream of educational systems by which thousand of institutions and millions of people use Blackboard Learn each day. This also applies to Radboud University of Nijmegen where the Blackboard Learn has become a prominent tool used for teaching activities.
From this point forward, we will use the term Blackboard as a substitute for Blackboard Learn.
Problem statement
Blackboard is introduced and has as main function to provide lecturers the ability (options) to teach students in different way if they want to (communications, provide accessibility to information/material, etc.)Only there is no real good idea what lecturers think about blackboard.
The following research question underlies this research:
- What is the opinion of teachers about Blackboard and its usability?
Conceptual model
Method
Within this research we will focus on semi-structured interviews.
Research question:
Research question:
- What is the opinion of teachers about Blackboard and its usability?
Sub questions
- Is Blackboard meeting the demands of the users, in this regards, teachers?
- How hard is it to learn the basic functionality of the Blackboard?
- Is the Blackboard efficient enough for the basic tasks?
Research Elements:
The elements of our research are the users of the Blackboard system. With users we mean the people in the Netherlands within the Radboud University who uses the Blackboard system. There are 2 primary users of Blackboard, the students and the lecturers. The student have a lot of influence/wishes about what they want of the education (the material), but the person that provides the material is the lecturer. The lecturer determines how he publish his course material/information (considering the needs of his students). The lecturers are more working with the system and they have the most influence on the system, so we will focus in this research to the opinions of the lecturers.
Variables:
Usability is the main variable. In order to investigate usability we have find the following attributes [2]:
- Learnability: The system should be easy to learn so that the user can interact with the system.
- How hard was it to find your way in the new update of BB.(Interface)
- How much time was spend considering the learning curve?
- Efficiency: The system should be efficient to use so a high level of productivity should be possible to provide the desired results.
- Compared to the previous version of BB, did you notice anything positive/negative in your productivity?
- Memorability: The system should be (even after some period of no use) easy to remember
- Is the user interface, easy to remember?
- Even after two weeks of not using BB?
- Satisfaction: The system should be pleasant to use, so that users are satisfied when using it.
- Are you satisfied with the new BB?
- Are there any services that you would like to see?
Selection of participants:
Our key contact person, central coordinator/supporter of Blackboard, has indicated that the lecturers of the Social Science faculty are more working with Blackboard and he is interested in opinions of active users compared with less active users of Blackboard. The participants should be appointed by the facultaire support person of the Social Science faculty, but we have still no response from them. Because lack of time Jos have agreed that we select lecturers for the interview from the Faculty of Science (FNWI).
In order to limit the domain we will interview 2 selected active users and 2 selected less active users of the Blackboard system. The selected participants will be invited individually for the interview.
Collect Data:
In order to increase the reliability of the interviews it will be taken by two persons in a non-disturbing environment. The first person (the interviewer)
- ask the questions
- keeps eye on the goal
- monitor non-verbal signals
The second person is available for the inspection
- completes the necessary questions
- keeps eye on the goal
- keeps eye on the technology (recording equipment)
The interview (audio) will be recorded with a telephone and it will be used for transcription for later on in the research. To prevent any concentration problems by the interviewee, there will be a telephone used that produces no sound and without a sim card.
Structure Data:
The transcription of all the interviews will be divided in different groups and it will be labeled (e.g. text analysis.) With this labels, it's possible to structure the information and use this to get an answer on our research questions
Analyse Data:
To derive conclusions there will be a certain weight assigned to each indicator. From the score there can be drawn a conclusion.
Interview:
In order to be able to answer the sub questions (see above) we have formulated for each variable (learnability, efficiency, memorability, satisfaction and complain/errors) interview questions.
Before the interview , the interviewee will be informed about the interview structure. The interviewer will
- give an introduction on the subject and a explanation about the reason of the interview
- give information about the process (time line, presence of the observer (second interviewer), existence of recording equipment, etc.)
Interview Question:
General questions
- For how long are you known with blackboard? (years of usage)
- Do you use blackboard frequently for you courses ?
- What courses do you give?
- For how many students per course ?( +/- avg)
- Did you noticed that there was a new version of blackboard introduced this school year?
BB Experience/ Satisfaction / Functionality
- Does blackboard support you in given your courses? (ask why)
- What material do you put on blackboard?
- Do you share documents?
- Do you put grades on blackboard?
- Do you use any other functionality?( like a wiki)
- What is your opinion about the Usability of Blackboard?( Is it complicated ? Or easy to use?)
- Can you explain that ? Why is this ?
- Do you miss any functionality within Blackboard?( Like direct linking to You Tube or online document editing)
- What functions do you miss?(follow up)
Learnability
- How did you feel when you learn and/or operate Blackboard for the first time? (If the answer is short, try to ask them to elaborate their answer)
- How long did it take for you to get familiar with Blackboard? Week, days, hours? (Again, try to ask them to elaborate their answer here)
Efficiency
- What are difficult and time-consuming tasks in Blackboard?
- Can tasks be performed in a straight-forward manner? (So the task is less time consuming. )
Memorability
- Do you think it’s hard to use Blackboard again after sometimes, for instance after the Summer’s break? (If they answer ‘yes’, what part of the task that they find it’s hard).
- Do you always know where to look for a specific task?
Complains/ Errors
- Did you ever have any difficulties or problems with blackboard ?
- Was this problem occurring this school year?(new version of BB)
- What was the problem ? (maybe answer with the first question)
- Is that problem solved? (if there was a problem)
Approach
Summary
We start off with an interview with Jos Groenewegen(coordinator of Black Board)to get more information about Black Board and it's main users. With this information we can define our research question better and tightening up scope our research field. The interview will be recorded and it will be used to determine if the recording hardware is good enough for transcription for later on in the research. After the interview is being taken we can use the information to draw a Conceptual Model,from this model we can extract the questions for our next interviews. The second phase of the interviews are being held with the lecturers from the Faculty of Science (FNWI), we will interview 4 lecturers. Before we invite our interview candidates for the second phase of the interviews, we will set up a new interview with more structure to improve the quality of the questions. So after the interview on paper is finished we will review the questions and invite our candidates. The next step is to make an appointment for the interview, after the date and time for the interviews are scheduled we can interview the teachers. The interviews will be recorded, so afterwards it can be transcripted. The transcriptions will be spread among the researchers of this group and all these transcriptions shall be brought to one main transcription. This transcription of all the interviews will be divided in different groups and labeled. With this labels, it's possible to structure the information and use this to get an answer on our research questions.
Select the research question
After we formed the research group, we make some research questions and select three from it. Then we discuss which question is the best for us.
Set up the interview
First make an appointment with Jos Groenewegen(coordinator of Black Board). After the interview with Jos we can refine our scope and set up the interview.
Select and Invite the interviewee
The interviewee will be the lecturers from the Faculty of Science (FNWI). From these lecturers we make an selection between the more experienced users of Black Board and the one who are new to the system.
Take the interview
The four selected lecturers will be interviewed. For each interview a group of two researchers will be formed.
Make a transcript of the interviews
The two groups will make their own transcript of the taking interview. Each research group will transcript two interviews.
Data structuring of the transcripts
All the data that is gathered from the interviews will be labeled during a meeting. All the members of the research group will attend the meeting. For this event it is for own interest to make a reservation for a room.
Data analysis
After the transcriptions are labeled, the data will be analyzed. Probably during the data structuring meeting.
Conclusion
If all the data is analyzed, we have to look if we can fit the data to the research question.
Reflection
The last step is to reflect all the taking actions.
Planning
- Before 7 October: Making an appointment with Jos. Tolga
- Before 13 October: Getting access to Wiki. Deri.
- Before 13 October: Setting up interview with Jos E.G questions Ashwien/Menno
- Before 11 November:Problem statement. Tolga
- Before 11 November:ORM and later will be revised by Deri. Menno
- Before 11 November:Introduction. Deri
- Before 11 November: Approach. Ashwien
- Before 18 November: Sending e-mail for getting users. Tolga
- Before 18 November: Data Structuring. Ashwien
- Before 18 November: Methods. Tolga
- Before 18 November: Setup question BB Interface, Usage. Revise the ORM model. Deri
- Before 18 November: Setup question Complians, BB Experience, Functionality. Menno
- Before 18 November: E-mail Stijn about feedback. Deri
- Before 24 November: work on feedback. All
- Before 24 November: setup q Learnabilty, Memorability Deri
- Before 24 November: setup q Effiency,Errors,Satisfaction Menno
- Before 22 November: Selecting persons. All
- Before 24 November: Expand approach, look at language Ashwien
- Before 24 November: Methods. completed Tolga
- Before 25 November: Wait for the mail of Key contact person, else choose self contact persons for the interview
- Before 25 November: Ashwien, find indicators for the variables.
- Before 16 December: ORM variables. Deri
- Before 16 December: Change the Problem statement, Selection of participants. Tolga
- Before 16 December: Template for the transcription, make new Deadlines. Menno
- Before 16 December: Update the wiki, add the Formula. Ashwien
- Before 20 December: All interviews taken
- Before 6 januari: All interviews transcript, analysed, labelled and having a conclusion.
Data structurering
Will follow after taking the interviews. The transcriptions will be tagged and shown here below. Every member will tag his own transcription.
Data analysis
Variables
After the interviews are taken, the transcriptions will be made from these. The text of the transcriptions will be analyzed, the variables and their indicators are shown below:
- Learnability [Positive, Neutral, Negative]
- Efficiency [Positive, Neutral, Negative]
- Memorability [Positive, Neutral, Negative]
- Satisfaction [Positive, Neutral, Negative]
- Complain [Negative]
For each transcription we made a table with the variables and the indicators. The transcriptions were commented with the variables we found and got labeled with a color corresponding with green for positive, blue for neutral and red for negative.
The outcome of these results are saved in the document Recap_totaal.pdf Media:Recap_totaal.pdf
Formulas
Learnability = (X * indicator) gives [Positive, Neutral, Negative]
Efficiency = (X * indicator) gives [Positive, Neutral, Negative]
Memorability = (X * indicator) gives [Positive, Neutral, Negative]
Satisfaction = (X * indicator) gives [Positive, Neutral, Negative]
Complain = (X * indicator) gives [Negative]
Positive = SUM(Learnability.Positive, Efficiency.Positive, Memorability.Positive, Satisfaction.Positive)
Neutral = SUM(Learnability.Neutral, Efficiency.Neutral, Memorability.Neutral, Satisfaction.Neutral)
Negative = SUM(Learnability.Negative, Efficiency.Negative, Memorability.Negative, Satisfaction.Negative, Complain)
Conclusion
For all conclusions see:Media:Total_conclusions.pdf
Total Conclusion:
In general the interviewed where quiet negative about blackboard. The interviewed all menaced that blackboard is inefficient and takes too much time to fully understand.
Looking at the research questions the following conclusion can be taken:
Is the Blackboard meeting the demands of users, in this regards, teachers? Its provides in some functionality the users wants. Like sharing files and providing e-mail list. However blackboard is mostly to extended and there are too many functions that aren’t used by the lectures. Also the tasks like sharing files and e-mailing a group of students are too inefficient
How hard is it to learn the basic functionality of the Blackboard? The basic functions like sharing files , e-mail students and sharing grades aren’t that hard to learn. Only if the lecturer wants to provide more functions that blackboard provides it can be a time consuming task to setup the required functions.
Is the Blackboard efficient enough for the basic tasks? All interviewed sad something about the inefficiency of Blackboard. Some even use own made websites because they don’t trust blackboard or don’t agree upon the way Blackboard works. One of the Lecturers menaced that he could not make a back-up form the files on blackboard and that’s why he uses an own site too share files with his students. So overall there is a negative feeling about Blackboard. Some feel it is too inefficient to use others use it but only the very basic set of functionalities.
Reflection
Conceptual
For the first phase we realized an ORM scheme, within this diagram we fitted our research subjects and the variables. Making this concept helped us alot for structuring our ideas and made the startup of the research more concrete. It allowed us to discuss about the topic, with the support of the diagram, so everyone was talking about the same universe.
Method
When the first ORM diagram was made, we contacted the coordinator of Blackboard who is responsible for the maintenance and implementations within the Radboud University. After our first contact by email, we decided to make an appointment with the coordinator for an interview. The interview had the intention to give us more information about the use of Blackboard and its users. When the interview was finished, the coordinator gave us the contact information of the persons who could raise interest within the Social Science Faculty. But unfortunately no one responded, even after some email contact. So we decided to select the 4 lecturers from the FNWI. From these lecturers we made an selection between the more experienced users of Black Board and the one who are new to the system. So we invited the selected lectures, gladly all the lecturers responded rather quickly so we could arrange the interviews. In the meantime we already made the interview questions, after some reviews the final interview was ready. The next phase was to interview the users, each researcher was responsible for his own interview.
After the first interview was taken, we scheduled a meeting for setting up a method for making the transcription and how to label the variables. So after the interview everyone made his own transcription and labeled it using the template. The transcription and labeling of the interview took most of the time. When this was finished, we arranged the last meeting to discuss the results of the data. From these meeting we could review the results and look what the last tasks were that were still open. This was our last meeting and within a week of this meeting all the tasks were finished.
Summary
In the beginning, our key contact person sent us the right path with good usable information and focus points about Blackboard usage.
We have started well and the considering the fact that this is a 3ec course, we have the tasks proportionally well divided. When selecting participants for the interviews, we thought that we come running out of time, because we had after some time still no response from the facultaire support person of the Social Science faculty (they would appoint participants based on Blackboard usage). We have solved this (with the approval of our key contact person) through selecting participants from the Faculty of Science (FNWI).
As a group we have limited or no interviewing skills, but all the interviews proceeded well. The preparatory work was good, so we could through asking targeted questions obtain the required information.
We can conclude that it was an educative research.
References
- ↑ http://www.blackboard.com/Teaching-Learning/Learn-Platform.aspx
- ↑ J. Nielsen (1993), Usability Engineering, Morgan Kaufmann Publishers Inc., San Francisco – CA