This article tackles the question whether it is a viable strategy to conduct online surveys among university students in developing countries. By documenting the methodology of the National Service Scheme Survey conducted in Ghana, we set out to answer three questions: (1) How can a sample of university students be obtained? (2) How can students be motivated to cooperate in online surveys? (3) What kind of devices do students use for completing an online survey? Our results indicate that online strategies can be very useful to reach this particular target group, if the necessary precautions are taken.
This article describes the results of an experiment designed to examine the impact of the use and amount of delayed unconditional incentives in a mixed mode (push to web) supplement on response rates, response mode, data quality, and sample bias. The supplement was administered to individuals who participate in the U.S. Panel Study of Income Dynamics, the longest running national household panel in the world. After 10 weeks of data collection, individuals who had not yet completed the interview were sent a final survey request and randomly assigned to one of three treatment conditions: no incentive, US$5, and US$10. The impact of the incentives on response rates and mode, effects on data quality, and sample bias are described. The implications for the use of incentives in mixed mode surveys and directions for future research are discussed.
Teaching has attracted growing research attention in studies of human and animal behavior as a crucial behavior that coevolved with human cultural capacities. However, the synthesis of data on teaching across species and across human populations has proven elusive because researchers use a variety of definitions and methods to approach the topic. I propose a novel method for the study of teaching behavior to be used across disciplines and populations toward such a synthesis: a teaching ethogram for animal and cross-cultural human research (TEACH). This article compares the results of the TEACH method with interview and time allocation data from the same study populations on Yasawa Island, Fiji. The TEACH method better matches the emic view of teaching as playing a role in children’s learning in Fiji, in contrast to the time allocation method. The TEACH method also produces quantitative data with greater behavioral detail than the other methods. This feature is particularly important for the usefulness of the TEACH method in making broad comparative data possible.
We describe an experiment to provide a time-limited incentive among a random sample of 594 hard-to-reach respondents, 200 of whom were offered the incentive to complete all survey components of a study during a three-week winter holiday period. Sample members were primary caregivers of children included in the 2014 Child Development Supplement to the U.S. Panel Study of Income Dynamics. The incentive provided US$50 to caregivers who completed a 75-minute telephone interview and whose eligible children each completed a 30-minute interview. Results indicate that the incentive was an effective and cost-efficient strategy to increase short-term response rates with hard-to-reach respondents and had no negative impact on final response rates.
Multi-actor studies are particularly suited for partner and family research, as they capture relationships beyond the conventional restraint of the household. Previous research on partner participation in the German Family Panel indicates higher participation of cohabiting and married partners compared to those living apart together. The present study evaluates whether this finding is due to unobserved relationship quality aspects associated with relationship status, differentially affecting the likelihood of partner response, or rather to field procedures favoring the participation of coresidential partnerships. Fixed effects models find a positive effect of moving in together on partner response, indicating that part of the relationship status effect found in previous research can, in fact, be attributed to coresidence. Analyzing the response process in detail reveals that the effect of moving in together goes back to main respondents’ consent to the partner survey, whereas no significant effect can be found on partner participation.
Social media are useful facilitators when recruiting hidden populations for research. In our research on youth and radicalization, we were able to find and contact young people with extreme ideals through Facebook. In this article, we discuss our experiences using Facebook as a tool for finding respondents who do not trust researchers. Facebook helped us recruit youths with extreme Islamic and extreme left-wing ideals. We conclude by discussing the benefits and limitations of using Facebook when searching for and approaching populations who are difficult to reach.
Previous research shows that interviewers to some extent fail to expend the effort that is needed to collect high-quality survey data. We extend the idea of interviewer satisficing to a related task, in which the interviewers themselves answer survey questions. We hypothesize that interviewers who self-administer the questionnaire in a careless manner will also not apply themselves fully to the task of administering survey interviews. Based on interviewer and respondent data from the sixth round of the European Social Survey in Belgium, we find support for some of the hypothesized associations between (suboptimal) response characteristics of interviewers in the "task as respondent" and the same (suboptimal) response characteristics recorded for their respondents, specifically with regard to interview speed, multiple response, and item nonresponse to the household income question.
Congregation-based health program evaluations often rely on surveys, but little documentation is available regarding specific methods and challenges. Here we describe methods used to achieve acceptable response rates (73–79%) in a survey of HIV-related attitudes and behaviors in two African American and three Latino churches in high HIV-prevalence communities in Los Angeles County. Survey participation was enhanced by conducting survey sessions at church-based meetings (e.g., women’s Bible study) and after worship services; employing diverse survey staff; providing participation incentives for pastors, church coordinators, and survey participants; and working collaboratively and respectfully with congregational leaders. Achieving broad participation in church-based surveys on sensitive health topics is feasible when done collaboratively with congregational leaders and with a flexible protocol, which permits tailoring survey approaches to cultural and organizational contexts and leverages available resources appropriately.
Experimental economic games reveal significant population variation in human social behavior. However, most protocols involve anonymous recipients, limiting their validity to fleeting interactions. Understanding human relationship dynamics will require methods with the virtues of economic games that also tap recipient identity-conditioned heuristics (RICHs). This article describes three RICH economic games—an allocation game, a taking game, and a costly reduction game—that involve monetary decisions across photos of one’s social network, integrating recipient identities while maintaining decision confidentiality. I demonstrate the ecological validity of these games in a study of male social relationships in a rural Fijian village. Deciders readily map these games onto daily life, and target earnings vary widely; consistent with ethnography, relative need is the primary rationale for decisions across the games, while both punitive and leveling motives drive reduction behavior. Consequently, altruism and spite are both elevated relative to anonymous target games in neighboring villages.
Cumulative evidence is mixed regarding the effect of lottery incentives on survey participation; little is known about why this strategy sometimes works and other times fails. We examined two factors that can influence the effectiveness of lottery incentives as suggested by leverage-salience theory: emphasis of survey attributes in invitations and characteristics of target populations. We conducted a web survey experiment where one condition highlighted lottery incentives in the e-mail invitations (incentive-centered condition) and the other highlighted the value of the survey with a brief mention of the lottery (survey-centered condition). We found that the incentive-centered condition had a significantly higher response rate than the survey-centered condition, especially among individuals with a relatively low income. Although invitation emphasis affected respondent compositions regarding motives for participation, the differences in response quality between the two experimental conditions were small and mostly not significant.
Prior studies suggest memories are potentially error prone. Proactive dependent interviewing (PDI) is a possible method to reduce errors in reports of change in longitudinal studies, reminding respondents of previous answers while asking if there has been any change since the last survey. However, little research has been conducted on the impact of PDI question wording. This study examines the impact of PDI wording on change reports and how these wordings interact with other survey features such as mode, question content, and prior change. Experimental results indicate that asking about change in an unbalanced fashion leads to more reports of change initially than other wordings, but only in a face-to-face survey. Follow-up questions led to final change reports that were similar across all wordings, but this necessitates asking additional questions. Findings suggest that asking PDI using change as the initial option should be avoided.
There is much debate over the number of interviews needed to reach data saturation for themes and metathemes in qualitative research. The primary purpose of this study is to determine the number of interviews needed to reach data saturation for metathemes in multisited and cross-cultural research. The analysis is based on a cross-cultural study on water issues conducted with 132 respondents in four different sites. Analysis of the data yielded 240 site-specific themes and nine cross-cultural metathemes. We found that 16 or fewer interviews were enough to identify common themes from sites with relatively homogeneous groups. Yet our research reveals that larger sample sizes—ranging from 20 to 40 interviews—were needed to reach data saturation for metathemes that cut across all sites. Our findings may be helpful in estimating sample sizes for each site in multisited or cross-cultural studies in which metathematic comparisons are part of the research design.
Few empirical studies exist to guide researchers in determining the number of focus groups necessary for a research study. The analyses described here provide foundational evidence to help researchers in this regard. We conducted a thematic analysis of 40 focus groups on health-seeking behaviors of African American men in Durham, North Carolina. Our analyses revealed that more than 80% of all themes were discoverable within two to three focus groups, and 90% were discoverable within three to six focus groups. Three focus groups were also enough to identify all of the most prevalent themes within the data set. These empirically based findings suggest focus group sample sizes that differ from many of the "rule of thumb" recommendations in the existing literature. We discuss the relative generalizability of our findings to other study contexts, and we highlight some methodological questions about adequate sample sizes for focus group research.
Although there is a large body of literature on interviewer gender effect, there are still two areas that merit further research. First, very limited attention has been paid to non-Western settings. Second, previous research has focused on responses to a limited number of survey questions and ignored other broader response behaviors. In this study, we use the Asian Barometer Study data to examine the gender effect on acquiescent response style for 53 agree/disagree questions asked in 11 Asian countries and societies. In general, we find that in six countries/societies, male interviewers elicited more acquiescent responses than female interviewers. We also discover that the interviewer gender effect is similar for male and female interviewees.
When recruiting respondents for cognitive interviews testing translated survey questionnaires, researchers often recommend interviewing monolingual non-English speakers because they are the likely users of the translations. However, these individuals are hard to recruit, and there is no standard definition of monolingual. Using cognitive interview data collected from pretesting of Chinese and Korean translations of the American Community Survey Language Assistance Guide, we investigated whether there were differences in respondents’ understanding of survey questions according to their level of English proficiency and if such differences remained after considering demographic characteristics. We found that the types of issues reported by monolingual speakers and partially bilingual speakers were similar and that differences seemed to be driven by different demographic characteristics and not necessarily by language proficiency. Our findings suggest the value of evaluating translated questionnaires with individuals having diverse demographic characteristics and recruiting both monolingual speakers and partially bilingual speakers as research participants.
We assess the consistency of measures of individual local ecological knowledge obtained through peer evaluation against three standard measures: identification tasks, structured questionnaires, and self-reported skills questionnaires. We collected ethnographic information among the Baka (Congo), the Punan (Borneo), and the Tsimane’ (Amazon) to design site-specific but comparable tasks to measure medicinal plant and hunting knowledge. Scores derived from peer ratings correlate with scores of identification tasks and self-reported skills questionnaires. The higher the number of people rating a subject, the larger the association. Associations were larger for the full sample than for subsamples with high and low rating scores. Peer evaluation can provide a more affordable method in terms of difficulty, time, and budget to study intracultural variation of knowledge, provided that researchers (1) do not aim to describe local knowledge; (2) select culturally recognized domains of knowledge; and (3) use a large and diverse (age, sex, and kinship) group of evaluators.
I analyze the effects of household sociodemography, interviewer performance in the current survey, and fieldwork characteristics on cooperation in a central telephone survey, where households with no publicly listed landline number receive face-to-face visits. Using the 2013 refreshment sample of the Swiss Household Panel, I employ household–interviewer cross-classified multilevel models and analyze first and later contacts separately. Some sociodemographic groups are less cooperative in the first contact only, others in both the first and later contacts, and still others in later contacts only. I offer recommendations about which households should be finalized at the first contact, which should be transferred to the face-to-face sample instead of being worked by telephone, and which interviewers should work which household groups.
This study compares the application of probing techniques in cognitive interviewing (CI) and online probing (OP). Even though the probing is similar, the methods differ regarding typical mode setting, sample size, level of interactivity, and goals. We analyzed probing answers to the International Social Survey Programme item battery on specific national pride. While probing answers in CI show indications for a higher response quality, OP can compensate through a larger sample size. Therefore, both methods have complementary strengths with regard to error detection and themes.
Although using random digit dialing (RDD) telephone samples was the preferred method for conducting surveys of households for many years, declining response and coverage rates have led researchers to explore alternative approaches. The use of address-based sampling (ABS) has been examined for sampling the general population and subgroups, most often using mail or invitations to web surveys. For surveying rare groups, these studies often involve multiple phases to identify eligible population members and then collect responses. We describe a new approach for surveying rare subgroups using ABS with a single-phase mail survey. The study incorporates a new method for using a list frame with the ABS frame to increase the yield of the subgroup of interest while avoiding a potential bias inherent in many dual-frame survey designs. The findings suggest that the new approach is both efficient and effective for identifying and measuring a rare population.
Self-reports of financial information in surveys, such as wealth, income, and assets, are particularly prone to inaccuracy. We sought to improve the quality of financial information captured in a survey conducted by phone and in person by encouraging respondents to check records when reporting on income and assets. We investigated whether suggestive prompts influenced unit response, compliance with the request to check records, precision of estimates, and accuracy. We conducted a split sample experiment in the Community Advantage Panel Survey in which half of telephone respondents and half of in-person household interview respondents were encouraged to check the records. We found a modest positive effect of prompts on compliance but no effect on unit response, precision, or accuracy.
This article explores questions of reflexivity, positionality, identity, and emotion within the process of ethnographic research. We reflect on our feelings of privilege and guilt in and through our ethnographic fieldwork and discuss the ways in which these experiences encouraged reflexive thinking and a crucial interrogation of the place of the self in the research process. Specifically, we construct individual vignettes centered on our differing research efforts to acknowledge that our work is unavoidably personal and to embrace the myriad ways in which our positionality impacted our findings. We conclude by offering insights into the ways in which our field experiences are important in the production of knowledge, both in our own work and more generally.
Respondent incentives are a popular instrument to achieve higher response rates in surveys. However, the use of incentives is still a controversial topic in the methodological literature with regard to the possible reduction or increase in response quality. We conducted an experiment in a large-scale German face-to-face study in which the treatment group was promised a modest monetary incentive. We used different indicators of response quality and compared the incentivized group with the control group. Our results indicate that in general there are no systematic differences between the incentivized and the control group concerning response quality. We found some hints that specific subgroups react differently to incentives in terms of response behavior. While response quality usually tends to be lower for older respondents, we found that in the incentivized group the response quality is higher for older respondents as compared to younger ones regarding the level of item nonresponse.
We discuss a recent development in the set theoretic analysis of data sets characterized by limited diversity. Ragin, in developing his Qualitative Comparative Analysis (QCA), developed a standard analysis that produces parsimonious, intermediate, and complex Boolean solutions of truth tables. Schneider and Wagemann argue this standard analysis procedure is problematic and develop an enhanced standard analysis (ESA). We show, by developing Schneider and Wagemann’s discussion of Stokke’s work on fisheries conservation and by discussing a second illustrative truth table, that ESA has problematic features. We consider how scholars might proceed in the face of these problems, considering the relations between limited diversity and the different methods of reducing truth tables instantiated in Ragin’s QCA and Baumgartner’s Coincidence Analysis.
The rise of social media websites (e.g., Facebook) and online services such as Google AdWords and Amazon Mechanical Turk (MTurk) offers new opportunities for researchers to recruit study participants. Although researchers have started to use these emerging methods, little is known about how they perform in terms of cost efficiency and, more importantly, the types of people that they ultimately recruit. Here, we report findings about the performance of four online sources for recruiting iPhone users to participate in a web survey. The findings reveal very different performances between two types of strategies: those that "pull in" online users actively looking for paid work (MTurk workers and Craigslist users) and those that "push out" a recruiting ad to online users engaged in other, unrelated online activities (Google AdWords and Facebook). The pull-method recruits were more cost efficient and committed to the survey task, while the push-method recruits were more demographically diverse.
This article presents a simple approach to making quick sample size estimates for basic hypothesis tests. Although there are many sources available for estimating sample sizes, methods are not often integrated across statistical tests, levels of measurement of variables, or effect sizes. A few parameters are required to estimate sample sizes and by holding the error probabilities constant (α = .05 and β = .20), an investigator can focus on effect size. The effect size can be thought of as a measure of association, such as the correlation coefficient. Here, effect size is linked across three of the most commonly used bivariate analyses (simple linear regression, the two-group analysis of variance [ANOVA] or t-test, and the comparison of proportions or 2 test) with a correlation coefficient or equivalent measure of association. Tabled values and examples are provided.
Traditional approaches to researching family planning in developing countries utilize surveys, focus group discussions, key informant interviews, or some combination thereof. However, these methods are limited in their ability to measure community-level influences on family planning behavior. In this article, we assess the benefits of applying community mapping, a participatory research technique, to understand social and cultural influences on family planning in rural Ethiopia and Kenya. In gender-segregated focus groups, participants collaboratively created a map of their district or zone, which they subsequently used to anchor discussion of local supports, barriers, and targets for change. Qualitative analysis of the transcripts revealed that community mapping is a powerful tool with which to understand community-level factors that influence family planning. By facilitating reflection on and discussion of local assets and issues, the mapping exercise identifies relevant factors from the perspective of community members and defines priority points for intervention.
Previous studies have found monolinguals and bilinguals use different areas of the brain when solving problems, while bilinguals manifest a cognitive system with the ability to attend to more relevant information while ignoring distractions (Bialystok 2007; Hernandez and Bates 2001). Since the mental process that monolinguals experience when answering survey questions differs from that of bilinguals, they likely encounter different problems in understanding the meaning of questions even in their native language. Therefore, for cognitive interviews aimed at improving the quality of translated questionnaires targeting monolingual non-English speakers, it is crucial to conduct interviews with such respondents to elicit relevant information. However, there has been no standardized tool for identifying these speakers. By using different definitions of monolingual speakers: (1) based on self-rated English reading, (2) based on self-rated speaking, and (3) based on both, we investigate how similar or different these groups are by comparing their demographic characteristics.
Growing rates of nonresponse to telephone surveys can contribute to nonresponse error, and interviewers contribute differentially to nonresponse. Why do some telephone interviewers have better response rates than others? This study uncovered a critical behavior of successful telephone interviewers over the course of introductions: responsive feedback. Using detailed coding of telephone introductions, I explored interviewers’ speech. Interviewers were most responsive to answerers in contacts that resulted in deferrals and least responsive in refusals. Practical applications for telephone interviewer training are discussed.
This article provides evidence-based guidance for practical survey work, namely choosing interviewers and their workload. Analyzing a survey of 3,568 households obtained through computer assisted personal interviews (CAPI), we find that interviewers learn considerably while the survey progresses. Time requirements for fieldwork increase concavely with sample size. This allows using larger samples with a given budget than would be expected in planning such projects with simplistic cost estimates. We find a decrease of interview duration of almost 50%, which translates into a significant increase of the average hourly wage the interviewers receive. These learning effects cease after around 20 interviews. Based on our results, we recommend targeting interviewer training by age and technology-affinity of interviewers for CAPI surveys.
Online purposive samples have unknown biases and may not strictly be used to make inferences about wider populations, yet such inferences continue to occur. We compared the demographic and drug use characteristics of Australian ecstasy users from a probability (National Drug Strategy Household Survey, n = 726) and purposive sample (online survey conducted as part of a mixed-methods study of online drug discussion, n = 753) using nonparametric (bootstrap) and meta-analysis techniques. We found significant differences in demographics and drug use prevalence. Ideally, online purposive samples of hidden populations should be interpreted in conjunction with probability samples and ethnographic fieldwork.
With nearly 50% of U.S. mobile phone subscribers using smartphones, survey researchers are beginning to explore their use as a data collection tool. The Got Healthy Apps Study (GHAS) conducted a randomized experiment to compare mode effects for a survey completed via iPhone mobile browser and online via desktop/laptop computer web browser. Mode effects were assessed for three types of outcomes: randomization/recruitment, survey process/completion, and survey items. In short, the distribution of survey completion times and the distribution of the number of apps owned were significantly different across survey mode after accounting for block group. Other key mode effects outcomes (including open-ended items, slider bar questions, and missing item rates) showed no significant differences across survey mode. Some interesting qualitative findings suggest that iPhone respondents enter more characters and omit fewer items than originally thought.
Surveys are an important tool for assessing physician and nursing professionals’ practice patterns and guideline adherence. Obtaining quality survey data consisting of low item and unit nonresponse remains a persistent challenge in these populations. We tested the relative impact of two envelope types (padded vs. priority mail) on unit and item nonresponse in a survey of Minnesota health care workers. Respondents were randomized to receive a survey in one of two envelope types: a padded 8.5'' x 11'' envelope or a similarly sized priority mail envelope. After the first mailing, the response rate was 53.9% and did not differ across envelope conditions. Females and RNs were more likely to respond to the priority envelope than the padded envelope, but this finding did not hold in multivariate analysis. There was no difference in item nonresponse across the two envelope conditions. It may be that our two approaches were not enough to permeate the semi-porous membrane of gatekeeping that has been posited as a driver of low physician survey response rates relative to those observed in the general population. Nonetheless, our findings suggest that packaging may matter for some populations and not others.
Two primary forms of qualitative data collection in the health and social sciences include self-report interviews and direct observations. This study compared these two methods in the context of weight management for people who had varying degrees of success with weight loss (n = 20). We asked general habits of eating as well as barriers to weight loss and maintaining a healthy diet. Video-recorded observations (20 minutes) followed audio-recorded interviews (45 minutes). Data were organized into four primary sections: (1) confirmatory data, where the interviews and observations held similar information; (2) discrepancies between what was reported in the interview and what was observed in the home; (3) new information that was unique to the observation and was not mentioned during the interview; and (4) clarification of data collected in the interview and observation. In general, the observations contained more confirmatory data for participants who had been successful at weight control than those who had not. The majority of observational data were emergent, which led to the discovery of new data of which we were unaware prior to the observations.
Mixed method designs are often used in scale development to generate potential items and enhance face validity. Less frequently, mixed method designs are used to examine other aspects of validity. As part of a larger evaluation in the Lao People’s Democratic Republic, we developed a scale to assess household self-reported changes in access to livelihood assets following a mine action program. In this article, we detail the process of assessing measurement validity using mixed methods. The qualitative method of evaluation was based on an analysis of in-depth interviews with key informants. The quantitative evaluation relied on data collected from the scale administration (N = 271) evaluated using Rasch measurement. The article offers an example to researchers interested in using mixed methods to assess validity in measurement development. It contributes to the growing literature on using mixed method designs to evaluate the properties of scales and to the methodological work on the measurement of access to livelihood assets.
In this article, we present a Bayesian inference framework for cultural consensus theory (CCT) models for dichotomous (True/False) response data and provide an associated, user-friendly software package along with a detailed user’s guide to carry out the inference. We believe that the time is ripe for Bayesian statistical inference to become the default choice in the field of CCT. Unfortunately, a lack of publications presenting a practical description of the Bayesian framework in the context of CCT models as well as a dearth of accessible software to apply Bayesian inference to CCT data has so far prevented this from happening. We introduce the Bayesian treatment of several CCT models, focusing on the various merits of Bayesian parameter estimation and interpretation of results, and also introduce the Bayesian Cultural Consensus Toolbox software package.
In this overview, we discuss the current state of survey methodology in a form that is useful and informative to a general social science audience. The article covers existing challenges, dilemmas, and opportunities for survey researchers and social scientists. We draw on the most current research to articulate our points; however, we also speculate on both technological and cultural changes that currently influence or may soon affect the efficacy of different methodological choices.
Mobile phone–assisted personal interviewing (MPAPI) is becoming a more widely used technique to collect survey data. Not having a hard copy paper document to return to when cleaning data raises the question of how data error rates compare with traditional paper surveys and at what cost. One hundred health research interviewers trained to use traditional pen-and-paper (PAP) survey methodology were recruited and randomly assigned to either a PAP or an MPAPI group. After receiving training on the survey instrument, each of the 100 interviewers conducted interviews with the same five interviewees, for a total of 500 interviews. Costs associated with the two survey methods were calculated. Very low error rates were achieved in both PAP and MPAPI, with a total of 381 data errors identified in 21,500 survey items. Findings suggest that experienced, well-trained interviewers using a short, well-constructed survey can produce very low error rates, independent of survey mode and that the benefits of MPAPI would be magnified as the size and complexity of the study increases.
There is a growing interest in examining network processes with a mix of qualitative and quantitative network data. Research has consistently shown that free recall name generators entail recall bias and result in missing data that affect the quality of social network data. This study describes a mixed methods approach for collecting social network data, combining a free recall name generator in the context of an online survey with network relations data coded from transcripts of semistructured qualitative interviews. The combined network provides substantially more information about the network space, both quantitatively and qualitatively. Although network density was relatively stable across networks generated from different data collection methodologies, there were noticeable differences in centrality and component structure across networks. The approach presented here involved limited participant burden and generated more complete data than either technique alone could provide. We make suggestions for further development of this method.
There is debate about the proper role of emotions in research; however, in the pursuit of certain research questions, an emotional impact on researchers is unavoidable and has the potential to engulf researchers and research processes. This article explores the emotional challenges involved in completing a large, mixed-methods research project focused on high-risk youth. We outline four dilemmas researchers confronted as they managed the emotional impact of the fieldwork: tentative disclosures, catharsis, hopelessness, and motivation. Understanding these dilemmas enhanced the analysis and the conceptual work of the research.
Asset ownership is frequently used to assess the welfare status of households in rural areas of developing countries. Researchers often want to know the prior status of households or how that status has changed over time. In a case study in the Brazilian Amazon, we compare recall data with contemporary reports on assets from a panel survey. We consider multiple dimensions of the consistency of retrospective and contemporary data and seek to identify characteristics that lead to more accurate recall. We find that although retrospective data provide some information on past assets owned by households, they do not provide a highly accurate measure of either individual asset ownership or counts of types of assets owned. Consistent with previous studies, we find that items with greater salience are recalled more accurately. We also find that wealthier households exhibit upward bias when recalling assets owned in a previous period.
Few studies have employed a controlled experimental design to test the effectiveness of unconditional cash incentives on the rates of participation in web surveys. Even fewer studies have looked at the effects of these incentives on nonresponse bias in web surveys. This article addresses these two underresearched areas by utilizing two separate sources of data on a random sample of college students. Specifically, we examine the impact of prepaid token incentives on response rates to a web survey and compare survey data on respondents to administrative records of all sampled persons. Results support the use of unconditional incentives in web surveys as an effective way to improve response. However, contrary to several studies on the relationship between token incentives and nonresponse bias, our findings suggest that prepaid cash incentives may actually produce data that are less representative of the target population.
This article focuses on a novel use of cognitive interviewing as a follow-up rather than as a pretesting methodology to explore mode effects. Respondent from a quantitative mixed-mode experiment took part in cognitive interviews where questions were administered face to face, by telephone, and by web, followed by a retrospective think-aloud. The focus of the think-aloud (used in combination with some prescripted probes) was not on what respondents understood by certain words or answer categories but on how they had answered questions in the different data collection modes. This article discusses the methods used in these mixed-mode cognitive interviews (which involved an element of mode mimicking) and how the interviews differed from standard cognitive interviewing for question testing. The benefits and limitations of our approach are discussed as well as lessons learned for using cognitive interviewing to explore patterns observed in survey data.
In 1984, Shin and Yu proposed that sampling Koreans by simply identifying those with the common surname Kim would yield a representative sample, as determined by geographic distribution. We extend the evidence that individuals with specific common surnames in Korea are representative of the whole population. We found that individuals with any of the five most common Korean surnames, not just Kim, were highly representative in Korea in terms of sociodemographic characteristics, attitudes, and health characteristics. In the United States, we found that Kim sampling produces a representative sample of Korean Americans among those who have a Korean surname. While Kim sampling generates a representative sample of Korean American men, it underidentifies Korean American women who grew up in the United States, many of whom do not have Korean surnames, thus potentially biasing samples of women.
In personal network studies, respondents (egos) are asked to report information about members of their personal network (alters); egos respond based on their perceptions. Previous studies of informant accuracy present a varied picture: Some find egos’ reporting on their relationships with alters to be accurate; others do not. In our study of people living with HIV/AIDS in Uganda, egos were asked to invite up to four alters named during their interview to answer questions about their relationship with ego. Using Gower’s general coefficient of similarity, we calculated a measure of accuracy both by variable and by alter. Our analysis by variable tends to confirm informant accuracy research, while our analysis by alter adds to the literature by identifying how accurate a particular ego is when discussing an alter and what characteristics might be associated with accuracy or inaccuracy.
Interest in a multimode approach to surveys has grown substantially in recent years, in part due to increased costs of face-to-face (FtF) interviewing and the emergence of the Internet as a survey mode. Yet, there is little systematic evidence of the impact of a multimode approach on survey costs and errors. This article reports the results of an experiment designed to evaluate whether a mixed-mode approach to a large screening survey would produce comparable response rates at a lower cost than an FtF screening effort. The experiment was carried out in the Health and Retirement Study (HRS), an ongoing panel study of Americans over age 50. In 2010, HRS conducted a household screening survey to recruit new sample members to supplement the existing sample. The experiment varied the sequence of modes with which the screening interview was delivered. One treatment offered mail first, followed by FtF interviewing; the other started with FtF and then mail. A control group was offered only FtF interviewing. Results suggest that the mixed-mode options reduced costs without reducing response rates to the screening interview. There is some evidence, however, that the sequence of modes offered may impact the response rate for a follow-up in-depth interview.
Dependent interviewing (DI) uses respondent data from earlier waves in panel surveys to improve the data quality of change estimates. Apart from a positive effect on data quality through reducing overestimations of change, DI could also affect data quality negatively when it leads to satisficing and an overestimation of stability between waves. In this article, we experimentally test two frequently used DI designs under different levels of measurement error. Our data consist of income reports from a four-wave panel survey conducted in the Netherlands. The effects of our experiment on data quality are modeled with a quasi-simplex structure to enable the decomposition of variances into measurement errors and true change. Our main conclusion is that there is some risk of a negative effect on data quality for proactive DI but not for reactive DI.
The article presents a method to elicit personal network data in Internet surveys, exploiting the renowned appeal of network visualizations to reduce respondent burden and risk of dropout. It is a participant-generated computer-based sociogram, an interactive graphical interface enabling participants to draw their own personal networks with simple and intuitive tools. In a study of users of websites on eating disorders, we have embedded the sociogram within a two-step approach aiming to first elicit the broad ego network of an individual and then to extract subsets of issue-specific support ties. We find this to be a promising tool to facilitate survey experience and adaptable to a wider range of network studies.
This article explores methodological shifts in longitudinal qualitative research and illustrates how researchers can maintain methodological continuity while staying open to necessary modifications. We describe methodological modifications and changes that occurred during our longitudinal research process. These change processes were initiated due to general advances in the field of qualitative research methodology, personal research experiences gained during the research process, and preliminary analysis findings. The change processes complicated our methodological decision making and simultaneously refined study purposes and analytical intentions. The investigation of change over time, time in context, and time and texture also play a significant role in our examples and reflections in this article. We argue that careful analysis and accounting of methodological continuity, modification, and changes can strengthen the trustworthiness of longitudinal studies, an important goal considering the methodological complexity often associated with longitudinal qualitative research.
Can existing longitudinal surveys profit from the (financial) advantages of web surveying by switching survey mode from face-to-face interviews to web surveys? Before such a radical change in data collection procedure can be undertaken, it needs to be established that mode effects cannot confound the responses to the survey items. To this end, the responses of the Dutch European Values Study of 2008 were compared to the responses of a time parallel web survey. The responses on 163 of the 256 items differed significantly across modes. To explain these response differences between modes, an exploratory crisp set qualitative comparative analysis approach was used. Five sufficient conditions—combinations of survey mode characteristics—but no necessary conditions for response differences between survey modes were found. Two survey characteristics were neither necessary nor sufficient to produce the outcome. Results suggest that switching modes may affect comparability between waves in a longitudinal survey.
We report sample composition discrepancies related to demographic and personality variables occurring in different stages of development of a probability-based online panel. The first stage—selecting eligible participants—produces differences between Internet users and nonusers in age, education, and gender distribution as well as in the personality traits of openness to experience, conscientiousness, and extraversion. The second and third stages of panel development—asking about willingness to participate in the panel and actual participation in online surveys—result in fewer and smaller discrepancies. The results suggest that among the three potential sources of sample composition bias considered, the largest impact comes from coverage differences with regard to Internet access.
Time diaries are a well-established method for providing population estimates of the amount of time and types of activities respondents carry out over the course of a full day. This article focuses on a computer-assisted telephone application developed to collect multiple, same-day 24-hour diaries from older couples who participated in the 2009 Panel Study of Income Dynamics (PSID). We present selected findings from developmental and field activities, highlighting methods for three diary enhancements: (1) implementation of a multiple, same-day diary design; (2) minimizing erroneous reporting of sequential activities as simultaneous; and (3) tailoring activity descriptors (or "follow-up" questions) that depend on a precoded activity value. A final section discusses limitations and implications for future time diary efforts.
Mode effects on responses to survey items may introduce bias to data collected using multiple modes of administration. The present study examines data from 704 surveys conducted as part of a longitudinal study in which parents and their children had been surveyed at multiple prior time points. Parents of 22-year-old study participants were randomly assigned to one of two mixed-mode conditions: (1) web mode first followed by the offer of an interviewer-administered telephone mode; or (2) telephone mode first followed by the offer of the web mode. Comparison of responses by assigned condition on 12 measures showed one statistically significant difference. Analyses that modeled differences by completed mode and the interaction between assigned condition and completed mode found significant differences on six measures related to completed mode. None of the differences indicated that more socially desirable responses were given in interviewer-administered surveys.