Onderzoeksmethoden 2/het werk/2011-12/Group 1

Introduction

All people are part of one or several communities or organizations. Each organization and community is unique as the people they consist of, but they also have some similarities. No one is really trying to understand how they are organized, what processes happen inside. This is particularly for companies in the IT businesses or IT departments of large companies which employ people with technical background. Therefore it is extremely important to have a better understanding about what is going on, because people can perform their tasks much more efficient if they understand the whole domain.

During our literature study we found it to be very difficult to find any information about this subject. There was only a minimum amount of relevant scientifically literature about how people with technical background would see reality in their mind.

We will try to extract this information from their minds using interview technics. IT designers will be encouraged to talk about organizations, communication inside the organization, structure of the organizations, what are the goals and tasks performed in the organization. By analyzing the gathered data and assuming that people talk about similar thing using same concepts with different words and phrases we will be able to design a model of these concepts and create a vocabulary with witch IT people describe organizations.

Question

What words and phrases does the group use to describe organizations?
What model would be a reasonable representation of the way the group talks about organizations?

Brainstorming for the research question

Conceptual Model

Interviews

Brainstorming on the interviews

Questions

We want the subject to talk about organizations or maybe better about his organization. Interview should take about 30 so more than 30 questions would be too much.

Introduction

- Please tell us about yourself in few words
- What is your background regarding your education and job history in the IT industry?
- What is your current job?
- Could you please introduce the company for which you work?
  - Name
  - Industry
  - What does it do?
  - Goal
- What are the responsibilities that come with your job?

Communication
- Would you be able to do your work without communicating with your colleagues?
  - What would go wrong? (if no)
  - Do you communicate with anyone in this organization about your work? (if yes?)
- What are the most important means for communicating with colleagues?
  - What goals do you achieve when communicating with colleagues?
  - Which ones are preferred and why?
- Does the organization provide a suitable environment for communicating? (good and bad)

Structure (hierarchy & tasks and processes)
- What are the main parts of your organization?
  - Is there a clear distinction between parts?
  - Does this structure influence how you work?
  - Which structural parts play the most important role in day to day affairs?
- How would you describe the role of the company hierarchy in day to day affairs?
- What is your position in the companies hierarchy?
- Could you describe for us what your tasks are?
- Do you ever have to wait till someone finishes something that you need?
  - What kind of things?
- Do people ever have to wait for you to finish something they need?
  - What kind of things?

Interview subjects

Jan Broos - Dots&Digits
Harko Cuppens - Radboud University Nijmegen
Ronny Wichers Scheur - Radboud University Nijmegen
Jan Michels (pilot)

Some things about the interviews

not more than 30 minutes
Record audio
Paper and pen for drawing diagrams (if would be needed)
Record video for drawing diagrams? (test in pilot)

Interview instructions

Start the interview with an introductory statement in which you state your name, the name of the interviewee and the topic.

For example:

Today is [DATE]. I am [NAME] and I am interviewing [NAME]. We will be talking about the organization [NAME] is working in.

Finish the interview by thanking the interviewee and give an opportunity to add something.

For example:

This was the last question I had. Maybe you have a feeling that we missed any important aspect? Thank you for your time!

During interview

We want to generate as much relevant text as possible so let the subject talk as long as he is describing his organization.
Be careful with explaining questions in examples. We have a problem with the introduction of concepts in questions and examples may increase these problems.

Preparation

Make sure you know what you are going to ask so that this will not take to much attention.

Transcriptions

Transcription is a very important part of the research. It is a material for content analysis, so nothing important should be left out. In this particular case it is not important o transcribe everything up a sound or emotions. Timeline should also be kept simple.

Coding

We will use two main rounds of coding.

Coding schema

First round

In the first round we identify text that discusses and organisation. There are two types of flags. We will cast the net wide. Rather have a false positive that a false negative.

Description of an organization
- Pieces of text in which the interviewee describes his or any other organisation or part or aspect of these explicitly
- Pieces of text in which the interviewee explains, defines, or otherwise clarifies the meaning of a concept that he uses to describe organisations.
Not a description of an organization
- Everything else

Second round

Second round of coding will be done with the following coding scheme. The relation to our conceptual model may not be immediately clear but will be discussed later

Goal of the organization
Structure of the organization
Hierarchy of the organization
Tasks in the organization
Communication in the organization
Processes in the organization
Assets of the organization

The coding will just be done on the text that was coded relevant in the first round of coding. We will mark pieces of text with a code when we are of the opinion that a piece of text is about the code concept. Hierarchy is a form of structure and so are tasks and processes. The structure code will just be used for expressions about structure that are not about the other two.

First coding round

Second coding round

Results

This listing gives us a base for determining:

A vocabulary
The meaning of the coded concepts
Some parts of the mental model the interviewee is using to think about his organization.

Coding results

Discussion

Evaluation of Interview as a means to generate data for generating a vocabulary

Interviews, workshops and Think Aloud Protocol are an efficient ways to gather data by extracting people thoughts and understanding how they see particular things.

For the purpose of generating a vocabulary that IT people use for describing organizations semi-structured interviews were used. Because we use semi structured interviews we had a lot of control over the topics that were discussed. We wanted this control to make sure that most of the generated text would be about the organisation. We selected some topics that we considered to be aspects of organisation that a lot of people would include in the description of an organisation even when not prompted to include these elements. Other topics could have been included. There is an aspect of interviewing that worried us and still does. Introducing topics is a way of controlling the interview which is what we wanted. It is also a potential problem since we are introducing our own schema into the data by doing so. It is also very hard not to influence the choice of words when you are having a conversation. Other data sources that were just created by domain experts and not by researchers would be preferable in this respect. Those sources would most likely have the disadvantage of having a larger part irrelevant text.

We think that, by being clear about the elements we introduced and by generating vocabularies per topic, we were able to keep the distinction clear what was found in the text and what were the parts we introduced. Also by making the whole transcript available and the interview soundfile to those who ask people can check if the conclusion we reach, the vocabularies we generated, were generated from text we introduced.

As discussed in the next section the question of which language to use for the interviews is very important and influences the data that is generated. This is especially true when either interviewee or interviewer is using a language he is not a native speaker of. We expect that this has influenced our data but since we only used one language we don't know to what extent.

Evaluation of coding: quality and differences

We found that coding is another aspect where different approaches can have an impact on the results. When looking at the the coding done by different members of our group it became clear that we had different ideas of how to select the pieces of text that were to be labelled or how many labels a piece of text could have. We skipped making a coding manual and we saw the results in these differences. One of the differences was that some of us coded whole sentences or even paragraphs with one label. Others coded words or phrases that were ready for the vocabularies. The differences were solved when those that coded bigger pieces of text did another round of selecting and come up with the words for the vocabularies. Next time we could solve part of this problem by using a coding manual. We did a pilot and this helped us in performing the interviews but we didn't code the pilot interview. Next time a coding of the pilot could give us a base for writing the coding manual.

In some cases it is hard to determine the relevance of data or the meaning of it. What is meant by a word or phrase depend on the context in which it is used. This should be taken into account. There is no obvious right or wrong approach, as long as the results are met and gathered data did not lose it essential meaning. Because of this it is important to make transcripts and semifinished products available for critical review so others may also evaluate the context of the words and phrases.

Quality of the generated vocabularies

Creating a vocabulary is a very tricky task, especially if it comes to creating a vocabulary for the specific domain. There are several bottlenecks in this.

The fact that interviewees were people with technical background should be taken in consideration.
Most of the time, their answers were clear up to a point, but still had a room for interpretation.
The other obvious bottleneck was a fact that interviews were conducted in English and this was not the native language for any of interviewees or interviewers. This made and impact on the quality of generated vocabulary.

The vocabularies we generated are not just lists of words. We generated lists of words and phrased that are constituent parts of the meaning of the concept used for labeling. These lists give us an impression which words are used by the subjects to talk about that aspect of their organisation.

How to judge the quality of these lists?

One of the main points is to judge whether the words were used to talk about the concept/category. This is hard to decide at this moment. We would consider intercoder reliability as an indicator but it would not be enough. A possibility could be to let the subject validate this relation between word and category but it is not clear to us how the setup would have to be for doing this.
Another aspect of the quality would be how much of the vocabulary connected to the category (coded concept) we extracted. In other words, would there be a lot of new words and phrases if we talked to the subject again? We can say something about this once we do that and since we haven't we can't judge this aspect of quality

Usability of a research

Penetration and diversity of this research could not make its results very usable. The scale of the research was much to small to make any statistical significance. To obtain a serious impact in such a large domains as IT and organizations more then 3 IT people from 2 organization should be interviewed. But the research helped to develop and test in practice some useful skills and patterns of how things should be done, what are the bottlenecks and how to work with data.

Conclusions

Our work resulted in vocabularies. So in that sense our method was successful.

We also have to conclude that the method needs a lot of work and more testing before the resulting vocabularies could be considered as representations of the way someone or some group speaks about organisations in a more general sense, meaning not just on that occasion. Intercoder reliability for instance would be something that could be improved.

We also have to conclude that our method did not give us an impression of the mental model that the subjects were using. This, we think, is mainly due to a lack of time and a lack of coding experience.

References

Alan Bryman "Social Research Methods" 3rd Edition. Oxford University Pres Inc., New York, 2008