Experiences from conducting semi-structured interviews in empirical software engineering research

---

Find the fulltext

Summary

In 2005 a pair of researchers who worked at the Simula Research Laboratory wrote a paper that combined their own experiences conducting interviews for research along with a literature review of previous studies based on interviews. There research, and that of the studies, was focused on software engineering. The focus of this paper was on preparing and conducting the interviews. The authors felt that not enough attention is placed on the quality of interview’s. On top of that many studies based on interviews do not publish enough information about how the interviews are conduced and therefor the quality of the data could be suspect.

The meat of the paper is on the Interview Experience. Including the necessary activities for a good interview experience, the required qualifications of the interviewer, issues seen in interviews, and interview techniques.

They make some helpful conclusions

1. Take the time and effort to create a good interview experience
2. Interviewers should be subject mater experts
3. Audio recordings are highly valuable, If interviewers do the work to make subjects feel comfortable and safe.
4. Describe the interview process when you report on your interviews

There are other helpful highlights for folks who are interested in undertaking an interview process.

1. There is a high level of effort to create a good interview experience

The paper describes a bunch of necessary activities that you must do before, and after the interviews, some of which can take hours.

• Scheduling
• Collecting of background information
• Preparing interview guides
• Discussions/meetings
• Summary writing
• Transcribing

Also, these activities don’t include the fundamental work of reflecting and summarizing the results of the interviews.

Effort in relation to analysis must also be taken in to account. Analysis is time consuming in nature and will probably exceed the time spent on the activities listed above by several times.

2. They present a summary of skills useful to a successful interview

According to their experience:

• encouraging the interviewees to talk freely,
• asking relevant and insightful questions, and
• following up and exploring interesting topics.

From the literature:

• Use a tidy and well-organised process: schedule the appointments as early as possible and be flexible to changes.
• Gain the trust of the subject: ensure confidentiality, explain your research motives, describe how the data will be applied and in which papers they are going to be used.
• Be courteous at all times. Remember to thank the subjects for their contribution.
• Be well prepared. Bring slides to the interview with the interviewee’s name on, your research goals, a short presentation of the research institution you represent etc
• Take care with your appearance and make a good first impression. We have experience with two strategies; 1) dressing up to make a formal and serious impression, 2) dressing down to appear less threatening. A good rule of thumb is to dress at approximately the same level of smartness and formality as the interviewees. If the interview is conducted in the subjects’ work place, it is beneficial to have an idea of the dress code in the company and adapt to it.
• Allow the interviewees to view the questions in advance, so they can prepare for the interview.
• Talk informally with the interviewees before or after the interviews to facilitate a friendly and relaxed atmosphere.
• Use humour. This can contribute positively to the interview. Laughing together can create a more relaxed and open climate. However, humour and bonhomie must not be used excessively or in inappropriate situations, because it is important to project an image of gravitas and seriousness.
• Be active and show interest by nodding, paying attention and asking follow-up questions.
• Be careful not to argue or question the answers you get. The interviewee may become defensive and lose respect for you.

3. They describe interview techniques, including types of questions

Six different kinds of questions from the literature

1. Behavior/experience questions, which elicit descriptions of experiences, behavior and actions
2. Opinion/value questions, which investigate what people think about certain issues
3. Feeling questions, which aim at understanding the emotional responses of people to their experiences and thoughts
4. Knowledge questions, which identify what factual information the respondent has
5. Sensory questions, which capture the experiences of the senses, and
6. Background/demographic questions, which identify the characteristics of the person being interviewed. All of these questions can be asked in the present, past or future tense.

In general, focus on what and how questions and avoid why questions. Also, try and stay way from questions that can be answered with a yes, or no.

They also talk about their experience with various question types:

• Questions where the interviewees must describe how they work often provide rich information, i.e., “Can you describe what you have been working with?” or “Can you explain how you tested the solution?” This corresponds to the behaviour/experience questions described in [21].
• Reflexive questions also provide useful information, such as “What could have been done differently in the project?” or “What was most challenging for you as a developer_manager in this project?” This corresponds to the opinions_value questions described in [21].
• Very detailed questions, such as “How many lines of code have you written in your career?” can be difficult to answer.
• Series of questions that presuppose that some activity has been completed successfully can be risky. If the activity has not been completed successfully the questions are without value. To illustrate, if it is presupposed that participants in the experiment used UML documentation and the interviewer prepares many questions in relation to this, those questions will be useless if the participants did not use such documentation at all.

They also talk about different question techniques.

• Probes are a device to get the interviewee to expand on a response
  • “Anything more?”
  • A period of silence
  • An enquiring glance
• Prompts suggest to the interviewer the set of possible answers that the interviewer expects
• Laddered question techniques: Allow interviewer to consider the invasiveness of questions
  • “By studying the subject closely and adjusting the level of intrusion by choosing the right types of question, quality interviews can be conducted.”

And they cover some additional techniques

• It is valuable to ask informal questions at the beginning, in order to loosen up and create a relaxed atmosphere.
• The interviewees’ ability to answer very general questions satisfactorily varies to a large extent. To approach this, it is possible to either
  • ask very specific questions at the beginning, such as “Can you describe how you worked with this task?”, and then ask more general follow-up questions, such as “Is this how you normally work?” or
  • start with general questions and follow up with more specific questions to ensure that the important issues are covered.

---

Notes on Sections

Introduction

Qualitative research methods originate from sociology and anthropology [7], and were designed mostly by educational researchers and social scientists to study human behaviour [27]. Interviews are a frequently used technique for data collection within qualitative research.

However, interviews are a resource-demanding data collection method; activities such as planning, conducting and analyzing are time-consuming by nature. In addition, interviewees have to spend time on a “non-productive” activity. Furthermore, the interviews should be carried out carefully, because the way in which the interview is conducted determines the quality of the data collected. It is also important to ensure that the interviewees feel comfortable, so they are willing to share their experiences with the interviewer.

In fields such as sociology and educational research, a large amount of knowledge already exists on how to conduct qualitative interviews. Interviews conducted within the field of software engineering seldom use this knowledge.

We present experiences from 280 interviews conducted in 12 software engineering studies in our research group at Simula Research Laboratory.

We have identified four areas that deserve special attention when planning and conducting interviews

(i) issues related to the effort necessary to accomplish the interviews, (ii) the skills of the interviewer, (iii) interaction between interviewer and interviewees, and (iv) tools and project artifacts.

2. Background

For this paper we have selected literature that either (1) deals with interviews in software engineering research, or (2) addresses one or several of the topics that we have identified as important for planning and conducting research interviews.

It’s common to differentiate between individual interviews, and group interviews (or focus groups)

Also important to differentiate between, structured, semi-structured, and structured interviews.

Despite the fact that interviews are used frequently in
software engineering, few papers address factors related to the planning and conducting of the interviews.

Tape recorder usage is frequently discussed. Many researchers use a tape recorder to avoid loss of information. The tape recorder also permits the interviewer to be more attentive to the subject, as outlined in

3. Experience Collection

Gives a table of studies the authors are using in their reviews

4. Interviewing Experiences

This section reports our experiences with planning and conducting interviews regarding effort, skills of the interviewer, interaction issues, and appropriate tools and project artifacts.

4.1 Necessary Effort

Understanding how much effort an interview study requires is important in the planning phase for allocating resources and scheduling the study.

4.1.1. Activities

Interview studies include several activities that require effort

• SchedulingIt is necessary to make appointments with interviewees. If it is necessary to recruit or select subjects carefully, this activity will probably require a lot of effort.

• Collecting of background informationIn some of the studies that we conducted, it was necessary for the researchers to read project documentation, to collect information from personal CVs, etc.

• Preparing interview guidesThe time required for preparing interview guides varied substantially from study to study. Some factors can increase the time and effort required, for example, several researchers making the guide together or needing to adapt the guides to each subject.

• Discussions/meetingsIf several researchers are involved in the study, it might be necessary to spend time on meetings and discussions before, after or between the interviews.

• Summary writingIn our experience, it takes approximately three to four hours to summarize one hour of audio taped interview.

• TranscribingIt normally takes about eight hours to transcribe one hour of audio taped material.

Effort in relation to analysis must also be taken in to account. Analysis is time consuming in nature and will probably exceed the time spent on the activities listed above by several times.

4.1.2. One or two interviewers.

A research interview is usually conducted by one interviewer. However, in some situations, it can be beneficial for two interviewers to conduct the interview together.

Based on these results from DES and experiences from other studies, we have identified the advantages and disadvantages of having two interviewers rather than one. We claim that the advantages of having two interviewers are as follows:

• In most situations, two interviewers will ask more questions than one interviewer. This will lead to the subjects talking more and thus, more information will be collected.
• It is often easier to use two interviewers than one because of the possibility of dividing the responsibilities. In semi-structured interviews, the interviewer must sometimes improvise, and it is challenging to listen closely to what is said and at the same time plan the next question. Thus, is beneficial to have a second interviewer who can focus on what is said, ask follow- up questions and aid the primary interviewer when necessary.
• Two interviewers have the opportunity to discuss and verify their interpretation of the interview. This increases the probability of understanding the subject correctly.

However, there are also disadvantages of being two interviewers. The workload associated with some of the interview activities will be doubled, and it requires more planning to conduct the interview with two interviewers than with one. The researchers must define their roles and agree upon the structure and content of the interview so they do not pull in different directions.

4.2 Qualifications

“It is obvious that the qualifications of the interviewer will have a great effect on the quality of the interview thereby, also the quality of the collected data. In software engineering interviews, it is important that the interviewer
has comprehensive knowledge of the software engineering etc.
area under investigation, in addition to good interviewing skills.”

4.2.1. Interviewing skills.

Good interviewing skills include the following:

• encouraging the interviewees to talk freely,
• asking relevant and insightful questions, and
• following up and exploring interesting topics.

The advice found in the literature is useful and in accordance with our experience. In the following, we provide some practical advice and a number of concrete examples that complement and re-emphasize the recommendations found in literature:

• Use a tidy and well-organised process: schedule the appointments as early as possible and be flexible to changes.
• Gain the trust of the subject: ensure confidentiality, explain your research motives, describe how the data will be applied and in which papers they are going to be used.
• Be courteous at all times. Remember to thank the subjects for their contribution.
• Be well prepared. Bring slides to the interview with the interviewee’s name on, your research goals, a short presentation of the research institution you represent etc
• Take care with your appearance and make a good first impression. We have experience with two strategies; 1) dressing up to make a formal and serious impression, 2) dressing down to appear less threatening. A good rule of thumb is to dress at approximately the same level of smartness and formality as the interviewees. If the interview is conducted in the subjects’ work place, it is beneficial to have an idea of the dress code in the company and adapt to it.
• Allow the interviewees to view the questions in advance, so they can prepare for the interview.
• Talk informally with the interviewees before or after the interviews to facilitate a friendly and relaxed atmosphere.
• Use humour. This can contribute positively to the interview. Laughing together can create a more relaxed and open climate. However, humour and bonhomie must not be used excessively or in inappropriate situations, because it is important to project an image of gravitas and seriousness.
• Be active and show interest by nodding, paying attention and asking follow-up questions.
• Be careful not to argue or question the answers you get. The interviewee may become defensive and lose respect for you.

4.2.2. Qualifications in software engineering.

Two views from the literatures

• Interviewer should be an expert in the field.
  • Helps to ask clarifying questions
  • helps steer questions to useful information
• Interviewer shouldn’t be an expert in the field.
  • Familiarity can cause them to overlook details
  • Subjects may assume an expert already knows things

Our experience is in accordance with [15]. In the field of software engineering it is very important for the interviewer to have extensive knowledge of the theme of the interview. Armed with such knowledge, the interviewer will be able to understand the information that the interviewees give, ask the relevant follow up questions, and be able to clarify ambiguities and to control the interview. It can be difficult for an interviewer with limited knowledge to understand what is important and to follow up interesting and relevant topics further. If the interviewer does not have appropriate knowledge, the quality of the interview depends to an overwhelming extent on the interviewee.

4.3 Interaction issues

This section discusses experiences related to interaction between the interviewer and the interviewees. Issues such as how to handle a subject that either says the barest minimum or strays from the question are outlined. Our experiences with interview questions and in particular sensitive questions are covered.

4.3.1. Silent interviewees.

Why

• Subjects may be unresponsive because they fear the consequences of talking too much
• see no potential benefit in participating in the interview and therefore no point in cooperation

Strategies

• Asking questions that cannot be answered only with “yes“ or “no”
• interviewer can feign ignorance and ask for details that are already well known to the interviewer
• it is important to ensure that the interviewees understand that there are no “right” answers
• software developers tend to believe that anyone coming to interview them is really there to evaluate them

Types of folks

• the uninterested ones, as described in [28]
• the shy ones that are often very technically orientated

In general, our experience is that it can be difficult to make these subjects more responsive.

4.3.2. Interviewees that talk too much

It is often easy to interview subjects that are very verbal and provide much information. Yet, there are subjects that talk too much about irrelevant topics.

It may therefore be better to let the interviewee ramble, since the rambling might make sense in hindsight. Steering a talkative subject back to the topic must be done gently.

When very talkative subjects go off track, we have positive experiences with letting them finish and then changing the topic.

Cutting the subjects off too soon might result in the loss of relevant information. However, if it is evident that the information is not relevant, or that there are time constraints, it will be necessary to interrupt them tactfully.

4.3.3. Interview questions.

This section addresses interview questions. Firstly, an overview of types of question described in the literature is provided. Secondly, our experience with different types of question in terms of how good we perceived them to be is detailed. The same is done with question techniques.

Types of Questions

Six different kinds of questions from the literature

1. Behavior/experience questions, which elicit descriptions of experiences, behavior and actions
2. Opinion/value questions, which investigate what people think about certain issues
3. Feeling questions, which aim at understanding the emotional responses of people to their experiences and thoughts
4. Knowledge questions, which identify what factual information the respondent has
5. Sensory questions, which capture the experiences of the senses, and
6. Background/demographic questions, which identify the characteristics of the person being interviewed. All of these questions can be asked in the present, past or future tense.

Similar types of questions are outlined in [15, 27]. Both [15] and [21] state that “what” and “how”-questions should be asked, but “why”-questions and questions to which only “yes” and “no” answers are possible should be avoided. The use of leading questions is discussed in [15] and it is argued that they might enhance the reliability of the interviews.

Our experiences with different types of question are as follows:

• Questions where the interviewees must describe how they work often provide rich information, i.e., “Can you describe what you have been working with?” or “Can you explain how you tested the solution?” This corresponds to the behaviour/experience questions described in [21].

• Reflexive questions also provide useful information, such as “What could have been done differently in the project?” or “What was most challenging for you as a developer_manager in this project?” This corresponds to the opinions_value questions described in [21].

• Very detailed questions, such as “How many lines of code have you written in your career?” can be difficult to answer.

• Series of questions that presuppose that some activity has been completed successfully can be risky. If the activity has not been completed successfully the questions are without value. To illustrate, if it is presupposed that participants in the experiment used UML documentation and the interviewer prepares many questions in relation to this, those questions will be useless if the participants did not use such documentation at all.

Question Techniques

• Probes: is a device to get the interviewee to expand on a response
  • “Anything more?”
  • A period of silence
  • An enquiring glance
• Prompts: suggest to the interviewer the set of possible answers that the interviewer expects
• Laddered question techniques: Allow interviewer to consider the invasiveness of questions
  • “By studying the subject closely and adjusting the level of intrusion by choosing the right types of question, quality interviews can be conducted.”

Additionnel techniques

• It is valuable to ask informal questions at the beginning, in order to loosen up and create a relaxed atmosphere.
• The interviewees’ ability to answer very general questions satisfactorily varies to a large extent. To approach this, it is possible to either
  • ask very specific questions at the beginning, such as “Can you describe how you worked with this task?”, and then ask more general follow-up questions, such as “Is this how you normally work?” or
  • start with general questions and follow up with more specific questions to ensure that the important issues are covered.

Sensitive Questions

Questions that touch upon sensitive issues require special attention. Individuals might be uncomfortable with certain questions and hence might be unwilling to discuss certain topics, might hold back information or might not be completely honest. It is important to avoid situations in which subjects feel that the questions are so intrusive that they are uncomfortable in the interview situation. It can be devastating for a study if some of the important subjects refuse to participate in the interview because they feel offended or that their privacy has been invaded. It is therefore important to handle such issues with care.

When approaching sensitive topics, it is important to create an atmosphere of trust and ensure confidentiality [15].

sensitive questions should be asked late in the interview, after the researcher has indicated that he or she understands the subject and is sympathetic to the interviewee [25]

“normalizing perceived deviance” [9], which entails that the interviewer should not express dismay or openly disagree with what the respondent says. Hence, the interviewees do not feel judged and an open atmosphere is created.

Our studies have led us to realise that questions related to the following areas are potentially sensitive in software engineering studies:

• issues related to the economy
• opinions about colleagues and customers,
• explanations of why things went wrong
• questions related to the interviewee’s own competence and mistakes.

When approaching potentially sensitive questions, we have been careful to follow many of the guidelines mentioned in the existing literature,

We have noted that when these precautions are taken, the interviewees are surprisingly open and informative on sensitive issues

4.4 Tools and artifacts

Audio Recorders

• Audio recorders help keep an accurate record
• Effort is involved to transcribe the audio though

Regarding taking notes instead of audio recording the interviews, our experiences vary to some degree. In the DES case study, we both took notes and audio recorded all the interviews. In this case it was evident that the notes were not sufficient to capture all the details of the interview.

They highly recommend using audio taps, as they haven’t seen it put them at a disadvantage at all.

Visual Artifacts

During interviews, it is common to use different elicitation techniques to encourage the informants to reveal what they know, feel, think or believe. Visual techniques can be more appropriate than verbal elicitation in certain situations.

They took artifacts from projects and used them to elicit information

We observed that using visual elicitation techniques had the following positive effects:

• It became easier for the interviewees to remember what they had been working with.
• It became easier for the interviewers to ask good follow-up questions related to the artifacts.
• The interviewees talked a great deal and provided rich and informative information.

5. Reporting from interviews

The quality of the data obtained from semi-structured interviews depends on the planning and conducting of the interviews.

They think that if you are creating research based on interviews you must include information about how the interviews were conducted. At a minimum.

• The interviewees should be described in terms of number of interviewees, how they were selected (e.g. their roles in the project or company under study) and how they were recruited.
• The interviews should be described in terms of number of interviews (possibly number of interviews with each interviewee) as well as duration and location of the interviews.
• The number of interviewers and their roles if several.
• The interview guide(s) that were used during the interviews, as well as other tools or project artifacts.

6. Conclusions and future work

In this paper, we have shown that semi-structured interviews are frequently used as a data collection technique within the field of software engineering. Semi- structured interviews involve high costs, and the quality of the collected data is related to how the interviews are conducted. In addition, it may be challenging to ensure that the interviewees experience the interview in a positive way.

Consequently, we believe that advice on planning and conducting interviews may be useful for many researchers planning to undertake research involving interviews. It is further important to collect, systematize and share experiences with interviews within the field of software engineering in order to increase the probability of collecting measures of high quality.