SURVEYS design

You will probably have to conduct a poll or survey (and certainly interpret one!) in your career.

We're going to talk today about how the PRACTICAL (non-sampling) issues of how to conduct a survey.

Sampling is the scientific part of survey research, but the practicalities of question design, etc., call for the knowledge of the world and the cunning (art) that I spoke of earlier.

>> People answer regardless of ignorance (i.e., they guess), indifference (i.e., they give an answer regardless of their commitment to it), or unwillingness (i.e., they simply lie).

>> Lots of other types, like the "feeling thermometer", the "semantic differential", fancy questions like flipping a coin to confess a bad answer: "Flip a coin. If it comes up heads, tell me the truth. If it comes up tails, tell me you have smoked marijuana."

We need to decide if a question is better "open-ended" or "closed-ended". See Howard Schuman and Stanley Presser "The Open and Closed Question" American Sociological Review 1979, Vol 44 (October):692-712. Their abstract is as follows: "Two quite different reasons for employing open as opposed to closed attitude questions can be distinguished. One is to discover the responses that individuals give SPONTANEOUSLY (e.g., “What do you think of President Bush’s performance in office?”); the other is to avoid the bias that may result from SUGGESTING RESPONSES to individuals (e.g., “What singer do you like the best: Elvis Presley, Enrico Caruso, Sting, Jay-Z, or someone else?” The first goal can be satisfied through careful pretesting, whereas the second requires that open questions be used in the final questionnaire. We examine both goals by means of experiments within large-scale sample surveys. A widely used closed question on Work Values is first compared with a parallel open question, and then the responses to the latter are used to reformulate the closed alternatives in new comparisons. More limited experiments on two other items also are discussed. In all cases there are large and reliable differences between question forms in univariate distributions, and in most cases important differences in bivariate relations also occur. An attempt is made to explain and reconcile both kinds of differences. The evidence suggest that if closed alternatives initially are constructed on the basis of sufficient open responses, then remaining open/closed differences may be due mainly to interviewing and coding problems with open questions, rather than to bias from closed questions."

>> Doesn't force R into a set of answers, i.e., into your theoretical box; doesn't categorize R

>> R enjoys open-ended questions more--they're less boring--, which leads to a better response rate

>> Open-ended questions are more personal and thus promote rapport with R (i.e., they say, "We care about what you think)

>> Open-ended questions can be used to get an exhaustive list when not all possible responses can be given (e.g., "Who is your favorite musician?")

>> Open-ended questions don't give away the answer (e.g., "Who is your State Representative?") / don't allow for guessing / avoid "central bias"

>> Answers are guaranteed to fit inside one's theoretical framework, making them suitable for confirmatory research. (Note that this is a different advantage from the previous one.)

>> Don't require much thought or writing skills on the part of R, which yields a better response rate

>> Not having to explain your response can be an advantage (e.g., it is less intimidating); in particular, closed-ended questions may be better for sensitive questions

>> We must be sure to include all the variables of interest (Y), the causal variable[s] (X), and possible third variables (Z) that could be hypothesized to explain away the association between X and Y.

>> Many concepts (e.g., conservatism), we will have to use many questions to get one overall measure.

>> Clarity: Will the respondents all understand the question in the same way? By contrast, "What do you think of President Bush?" can be understood in any of a variety of contexts.

>> Don't ask double-barreled questions. Here's one such question: "Are you in favor of SDI, or do you favor lower defense expenditures?" Here's another one from the Duluth News-Tribune of 2/5/94 (their "This week's question" on the editorial page): "Do you believe there is a health care crisis in America, or should the status quo be maintained?"

G. ARE THE QUESTIONS PHRASED POSITIVELY? ["Positively" is not the opposite of neutral.] Avoid double negatives: "How wrong do you think President Bush's opponents are?" "Does it seem possible or does it seem impossible to you that the Nazi extermination of the Jews never happened?"

Babbie’s example (8^th edition:152): Should communists be prohibited from teaching in the schools? Yes / No” Many people answered yes to mean they should be allowed to teach. When the options were changed to “Permitted / Prohibited”, the research got clearer.

>> on the one hand, not over-elaborate, stilted, difficult, jargon-filled, or ponderous ("Are you currently matriculated in an institution of higher education?")

>> on the other hand, not too slangy or informal ("What musician are you down with?")

>> "Do you favor retaining loud-mouthed, pro-terrorist, racial agitator Andrew Young as a U.S. ambassador to the United Nations?" [From Kahane, Ch. 4, p.90.]

>> "Are you in favor of the Equal Rights Amendment, which will forbid discrimination on the basis of sex?" [From the New Jersey Poll, approx 1975. "Equal rights" is a loaded term, so the question was a bad one, even though that was indeed the name of the amendment. Furthermore, and worse, "discrimination" prejudges whether the differentiation in law of women and men is legitimate.]

• “Now that my [Grams’s] $500 per-child tax credit has been enacted, do you want me to continue seeking more tax reductions along with simplifying our tax code and overhauling the I.R.S.?” [From a “Rod Grams for U.S. Senate” request for donations, 1997.]

• “Do you want me to continue promoting the abolishment [he means “abolition” or "abolishing"] of wasteful government bureaucracies, such as the Department of Energy, which alone costs taxpayers nearly $20 billion annually?”

>> But even beyond avoiding loaded words, try to stay away from the social desirability bias by means such as the following: "Probably everyone has broken some law in their life. In the last month, have you broken any law?"

>> Associated with social desirability is the consistency problem (which I believe is also called the "halo effect"). If you ask two questions close together, R feels impelled to show consistency. So if you ask R's political affiliation just before you ask h/her rating of Bush, you force h/her into h/her "party line".

A. ARE THE OPTIONS EXHAUSTIVE? It is frustrating to R to be asked a question with no provision for h/her answer (e.g., "Do you generally vote Republican or Democrat, or do you generally split your ticket?"). [What about a member of the Prohibition Party?]

B. ARE THE OPTIONS MUTUALLY EXCLUSIVE (NON-OVERLAPPING)? Overlaps confuse R or make him think you're a fool (e.g., "How old are you: 18-20; 20-30; 30-40; or 40 or older?").

C. ARE THE OPTIONS APPROPRIATELY PRECISE? Measure income level to the nearest $10,000, for example.

D. THE OPTIONS SHOULD AVOID CENTRAL BIAS (e.g., "Do you believe the defense budget should be increased less than 35%, 35-39%, 40-44%, 45-50% or more than 50%?").

1. IDENTIFY SEVERAL SOURCES OF BIAS IN WRITING SURVEY QUESTIONS. WHAT ARE SOME WAYS OF AVOIDING BIAS?

HAVE STUDENTS CONSTRUCT A SURVEY QUESTION(S) DESIGNED TO DETERMINE ONE OF THE FOLLOWING:

Nonresponse is a bigger problem than sample size; can screw up a sample instantly

• Another way: check characteristics [as Uslaner & Weber did, arguing that the nonresponse was the same for both Republican & Democratic party chairs]

• Another way: graph responses/characteristics over time & extrapolate to nonrespondents

A. Give some easy questions first. How long have they lived in New Jersey? How would they rate the president?

C. Sensitive questions (e.g., income) should go last, for fear of nonresponse. But in general, questions calling for factual content are not sensitive to question wording, emphasis, sequence. Opinion questions are sensitive to these.

D. Funnel structure: from general to specific; focus from broad to narrow. Avoid cuing--but we still have to worry about early, broad commitments (e.g., "I'm a liberal"). Sometimes [xx when?] a structure from narrow to broad will be appropriate.

F. Minimize boredom by breaking up modes of questions and options; reverse questions from time to time. Shift type of question to keep up interest:

- Shift response/judgment format (e.g. from comparisons or rankings [choice of candidate] to absolute ratings [positives and negatives])

G. Avoid "response set". Many questions, all answered yes, yes, yes or no, no, no.

>> Most important: model INTERESTED CURIOSITY: be nondirective & nonjudgmental; be pleasant & relaxed; take the attitude, "Aren't I privileged to be allowed to hear what you think!"

>> Appearance (if face to face): dress and act like the people you're interviewing, but don't look uncomfortable doing so. (I wouldn’t wear a dashiki in Harlem.)

>> Don't talk down or talk up to respondent. "Yo, dude, can we rap for a while?” or “Let’s get down with this survey."

>>the self-administered, written questionnaire. Mail it out and hope we get it back. (This is usually done by mail, but group administration and home delivery and home administration and home pick-up are also possible.)

>>telephone interview, usually at R's residence. (We exclude businesses.) Usually done from a phone bank, but it can be done from the interviewers' homes.

We evaluate specific designs according to (19) factors, as given on your handout: [HAND OUT] These aren't intellectually difficult, but I'll go through them, trying to mention some of the less obvious practicalities. I've tried to list them in order of decreasing priority.

>> Cost (including time [we need quick results], organizational effort [we may not be set up to do this kind of work, e.g. just getting the phones or hiring the interviewers all over the country], and monetary cost [which means a decreased sample size])

>> Response rate, including (especially) the bias due to nonresponse / refusal / noncompletion. A nonresponse rate of 50% is said to be adequate, 60% good, and 70% very good. But demonstrating the absence of any nonresponse bias is much more important than higher response rate [Babbie].

>> Response rate for individual questions (i.e., the incidence of "don't know" and "no answer" responses)

>> The length of the survey, meaning the number of questions it is possible to ask

>> The flexibility of the interview: the ability to explain, follow up, clarify, or change the order of questions

>> The ability to obtain data even if R refuses to participate (e.g., information about race, sex, house, neighborhood)

>> The risk of the interviewer influencing R, including the so-called "social desirability" factor

>> The ability of the researcher to randomize, and in particular to work without a list frame

>> Ability of the researcher/interviewer to control accurately whether R is in the universe (e.g., to make sure that the survey isn't answered by a child, or by two people discussing the answers with each other)

>> The ability of the interviewer to enter answers directly into a computer (as opposed to using separate data entry operators)

>> The ability to prevent or control "curbstoning" and other interviewer problems

• The ability to disguise the sponsor of the research (until the end of the survey, anyway)

Let me note that "the ability of the interviewer to wear whatever clothes s/he likes" is not an important issue in distinguishing these three types of surveys.

Pretest, pretest, pretest, including on each other! Do this even if you aren't really the target of the research. A little effort here can avoid major problems later. Pretest the whole survey: remember the story of the person who, after all the return envelopes were addressed and stamped, discovered that the questionnaire couldn't fit into them.

Pretest closed-ended questions with open-ended, to be sure the options you provide are faithful to people's responses.

* Evaluate the pros & cons of doing an "automated" telephone survey of Duluth residents, where the computer dials the random numbers, gives the questions, and records the data entered by phone.

* Evaluate the pros and cons of doing a survey of UMD students where one goes through the dorms at 4 a.m. and puts copies of two surveys under each door, with a self-addressed envelope returnable through campus mail to Professor Chilton.

* Evaluate the pros and cons of using interviewers of the same ethnicity and gender as the respondent to be interviewed in a face-to-face interview.

* Professor Eileen Zeitz Hudelson is [October 22, 1997] running for one of two at-large seats on the Duluth School Board. The election is on Tuesday, November 4, 1997 (only two weeks away). Design a simple survey for finding out how many votes she and her opponents will get, and the major reasons for Rs’ choice. Since you are on a tight budget, be sure to gather only the most necessary background information.

• How will you approach Rs?—face-to-face interviews, telephone interviews, mail questionnaires, other self-administered?

• What background variables do you want to gather that will be of use in your study? [Here are some possibilities: gender; precinct; age; kids in school, either now or previously; school district; likelihood of voting; income; education; knowledge of candidates & general political information; being registered.]

From Sheldon R. Gawiser & G. Evans Witt "Twenty Questions a Journalist Should Ask About Poll Results", Appendix I, pp. A129-A134 in Earl Babbie The Practice of Social Research 7th edition

1. WHO DID THE POLL? A reputable, independent survey research org? Don't trust anything done by, say, a candidate's organization.

2. WHO PAID FOR THE POLL, AND WHY WAS IT DONE? This will affect what results are given and how they are presented, and it may affect how everything about the survey is slanted.

5. WHAT AREA (NATION, STATE, OR REGION) OR WHAT GROUP (TEACHERS, LAWYERS, DEMOCRATIC VOTERS, ETC.) WERE THESE PEOPLE CHOSEN FROM? (And is the Universe being discussed or implied? And we should also ask about the sampling frame used!) This is one way to bias the results.

6. ARE THE RESULTS BASED ON THE ANSWERS OF ALL THE PEOPLE INTERVIEWED? It is legitimate to report subsets. It is common to show the results broken down by race or sex or income or party i.d.; we may want to restrict our answers to likely voters. But we have to say what we did, and we have to report the sample size (and the appropriate SEE).

7. WHO SHOULD HAVE BEEN INTERVIEWED AND WAS NOT? This is a Universe-Frame question again. Wellstone is supposed to be 34 points ahead of Boschwitz "on the range", according to a 10/20/96 Star Tribune poll, but the actual survey really only covered four counties.

8. WHEN WAS THE POLL DONE? Events can change the results, and in any case we don't want outdated information.

10. IS THIS EITHER A DIAL-IN POLL (e.g. a 900 number), A MAIL-IN POLL, OR A SUBSCRIBER COUPON POLL? ... in which case one should just dismiss the results entirely.

11. WHAT IS THE SAMPLING ERROR FOR THE POLL RESULTS? It should be reported. Remember to distinguish between the 95% and 68% confidence levels. And remember: this assumes that the survey was based on a probability sample. If it isn't, no sampling error can be computed; or anyway, the SEE is at least as large as that reported.

12. WHAT OTHER KINDS OF MISTAKES CAN SKEW POLL RESULTS? In particular, what was the nonresponse rate?

13. WHAT QUESTIONS WERE ASKED? We need the exact wording, because it makes a difference.

15. WHAT OTHER POLLS HAVE BEEN DONE ON THIS TOPIC? DO THEY SAY THE SAME THING? IF THEY ARE DIFFERENT, WHY ARE THEY DIFFERENT? You'll often see a history given: Wellstone was leading Boschwitz by 14 points two weeks ago, 4 points a week ago, and 12 points now.

16. SO, THE POLL SAYS THE RACE IS ALL OVER. WHAT NOW? It ain't over till it's over. A reputable poll won't "call" elections, particularly since it can affect the election and get blamed for the result. ("Discouraged the putative loser's supporters." "Made the putative victor's supporters complacent, so they didn't turn out.")

18. SO I'VE ASKED ALL THE QUESTIONS. THE ANSWERS SOUND GOOD. THE POLL IS CORRECT, RIGHT? We'll never know if a poll is correct unless we do a census.

19. WITH ALL THESE POTENTIAL PROBLEMS, SHOULD WE EVER REPORT POLL RESULTS? Sure, if the answers are good.

20. IS THIS POLL WORTH REPORTING? This concerns the substantive value of the poll, which you'll have to judge for yourself.

When you read the next report of a survey, note how few of these questions are answered (or can be answered with the information given).

* Mention "constraint" (Converse & Stokes The American Voter, which seemed to show no intellectual coherence) vs. Feldman & Zaller's concern with principled / ideological orientation, which only emerges when you let people speak their own coherence.

* [Get lecture from E07.10: Stanley Feldman & John Zaller "The Political Culture of Ambivalence: Ideological Responses to the Welfare State"]

III. STUDYING CHANGE OVER TIME: CROSS-SECTIONAL RETROSPECTION, TREND, COHORT, AND PANEL ANALYSIS

* Usual survey is cross-sectional: a snapshot of a population at a given time. (It DOESN'T mean a survey of the entire U.S. population; it could describe a cross-section of any particular population, e.g., Yale graduates.)

* This provides no sense of change over time, but we can get a sense of how things have changed over time by asking people to provide retrospective information. But this information is notoriously unreliable.

* Trend analysis just looks at how the overall pattern changes over time. For example, Ethridge, p.217, Figure 1, shows "strength of party identification" in Britain over a twenty-year period. [QUIZ: What are the different bold dots along the angular line?

What is the vertical dimension, and how is it measured?

What is the meaning, in words, of the decline from 2.19 in 1964 to 1.74 in 1983?

]

* Trend analysis is good at showing change. The problem comes when we try to understand how/why this change is occurring. Two every-present, polar-opposite hypotheses about change: the "replacement" hypothesis and the "overall change" hypothesis.

>> Replacement: no one ever changes their minds; the change is entirely due to the continuing replacement of one generation by the next. The change is entirely BETWEEN generations and not at all WITHIN generations.

>> Overall change: everyone changes their minds at the same rate; the change is entirely due to this mass change. The change is entirely WITHIN generations and not at all BETWEEN generations.

>> There can be a mixture of these forces, of course, and there can be other forces as well: for example, the change is due to people shifting from Labour and Conservative parties to the Alliance party, which inherently doesn't command as much allegiance (or which commands the allegiance appropriate to new converts).

* Trend data offer no way to know which hypothesis (or which mixture of these hypotheses) is true. For example, look at Figure 1 again: what is going on? We can't tell.

* Ideally, we can break it down person by person, so that we have data on each person from 1964 to 1983. This is called a "panel study" or "longitudinal study". Panel studies follow individual people, surveying them repeatedly over a period of years or even decades.

>> Hard to do: it's hard to follow people in this nation of movers, and hard to get their cooperation repeatedly, and they might die anyway.

● Experimental mortality: Each reason for losing a person is potentially confounded with variables of interest. Movers vs. nonmovers. Cooperators vs. noncooperators. People who die vs. people who live. Etc. Each of these might be associated with (e.g.) the strength of people's political identification.

● Sensitization: having studied people once will make them differ in their subsequent behavior (e.g., they might pay attention to politics more closely) and/or make them differ in their subsequent responses to the questions, even if their intermediate behavior hasn't changed (e.g., they might say they are more interested in parties, now that they know what your orientation is).

* An alternative is to do a "cohort analysis", where we create fictitious aggregate people called "cohorts". Most of the time (as in the Clarke & Stewart reading) the cohorts are age cohorts, but they could be "dog owners" vs. "cat owners" vs. "other or no pets", or "Labourites" vs. "Tories" vs. "Alliance" vs. "others or no party".

* Let's look at how these issues play out in the Ethridge reading example. [Get lecture from E07.11: Harold D. Clarke & Marianne C. Stewart "Dealignment of Degree: Partisan Change in Britain, 1974-83"]

1. STATE A RESEARCH QUESTION OF INTEREST TO YOU THAT REQUIRES AN ANALYSIS OF CHANGE OVER TIME, I.E., THAT CANNOT BE ANSWERED BY A SINGLE, CROSS-SECTIONAL SURVEY.

3. STATE THE "REPLACEMENT" HYPOTHESIS AND THE "OVERALL CHANGE" HYPOTHESIS IN THIS CONTEXT. WHICH HYPOTHESIS (OR WHICH MIXTURE OF THEM) DO YOU BELIEVE IS CORRECT?

1. DEFINE AND DIFFERENTIATE CROSS-SECTIONAL, TREND, COHORT, AND PANEL STUDIES, AND EXPLAIN THE ADVANTAGES/DISADVANTAGES OF EACH.

Often we want to measure a concept but don't have any single (valid) question: "How prejudiced are you?" won't work! We measure by means of multiple questions.

One way to approach this is to use several questions and combine them. (There are other ways to measure; e.g. physiological reactions [skin reactivity when a person of another race walks in the room] and experiments [e.g., having someone pretend to be a KKK recruiter and talk to them].)

There are two ways of combining questions: indexes and scales. Scales classify people in terms of the pattern of responses. Indices only look at the total responses (sometimes weighted).

The best-known and most often used form of scale is called the "Guttman Scale", after the famous methodologist Louis Guttman. Guttman scales measure intensity by "cutting" the dimension of interest at different "cutting points". For example, we can measure prejudice through the Bogardus Social Distance Scale (p.407). (etc.) [Write this into these notes.]

Indexes (indices) measure intensity by looking at the variety of agreement/experience (e.g., a party identification index based on six items):

Index means counting; scale means points along a line (or sometimes other categorization). Index means breadth; scale means depth/intensity.

"GET INTO GROUPS OF THREE OR FOUR AND SUGGEST QUESTIONS THAT COULD BE USED IN A SCALE OR INDEX TO MEASURE POLITICAL PARTICIPATION AMONG THE STUDENTS IN THIS CLASS." [Note that the target population is important.] "EACH GROUP THINK OF TWO ITEMS TO MEASURE IT. CONSIDER THE POSSIBILITY OF WEIGHTING YOUR ITEMS."

"WHAT TWO POSSIBLE ITEMS HAVE YOU THOUGHT OF THAT COULD GO INTO A SCALE OR INDEX?"

[Go through the suggested items, and critique them in terms of the various forms of validity.]

Note also that there may be more than one form of participation, in which case we may want to use a scale instead of an index approach. For example, we may find that participation can be separated into participation at the national, state, or local levels. We might also find that some people are issues-oriented (and so would participate through educating themselves, attending political discussion forums, reading opinion pieces, writing letters to the editor, etc.), while others are candidate group-oriented (and so would participate through campaigns, campaign contributions, etc.).

* [For indices:] Intermediate proportion: we can't have people all at one end of the scale.

* unidimensionality/concurrent validity: Do they all measure the same dimension? Is there only one factor in a factor analysis? When we control for the overall dimension, is there any residual correlation among the items?

* Correlational validity: Does it correlate with parents' party i.d. or with the respondent's ideology?

* Reliability: This is a subset of validity, but one that can be more directly assessed:

>> by split-half validity [Do we get the same answer if we use the first two questions as the last two?].

* The "Twenty Questions a Journalist Should Ask About Poll Results" (by Sheldon R. Gawiser & G. Evans Witt; Appendix I, pp.A129-A134 in Babbie)

5. What area (nation, state, or region) or what group (teachers, lawyers, Democratic voters, etc.) were these people chosen from?

15. What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?

There were other polls. (I did one several years ago.) Their results are not given. No comparison given with polls from other college campuses.

18. So I've asked all the questions. The answers sound good. The poll is correct, right?

The three different types of surveys (face-to-face, telephone, and self-administered) differ according to different dimensions:

* Cost (including time, organizational effort, and geographic "reach": all of these affect the possible sample size)

* Response rate, including (especially) the bias due to nonresponse / refusal / noncompletion

* Response rate for individual questions (i.e., the incidence of "don't know" and "no answer" responses), which affects the possible bias of the survey

* The length of the survey, meaning the number of questions it is possible to ask

* The flexibility of the interview: the ability to explain, follow up, clarify, or change the order of questions

* The ability to obtain data even if R refuses to participate (e.g., information about race, sex, house, neighborhood). Such data is helpful in judging the bias due to nonresponse.

* Interviewer bias arising from the interviewer influencing R, including the so-called "social desirability" factor

* Sampling bias arising from any choices left to the interviewer, i.e., a free choice of who to interview in a household (but this should never happen in a well-designed survey)

* Ability of R to respond on h/her own time, at h/her own pace, at h/her leisure

* The ability of the researcher to randomize, and in particular to work without a list frame

* Ability of the researcher/interviewer to control accurately whether R is in the universe

* The ability to prevent or control "curbstoning" and other interviewer problems

"Twenty Questions a Journalist Should Ask About Poll Results" (by Sheldon R. Gawiser & G. Evans Witt; Appendix I, pp.A129-A134 in Earl Babbie The Practice of Social Research (7th Edition)

2. Who paid for the poll, and why was it done? [This will affect what results are given and how they are presented]

5. What area (nation, state, or region) or what group (teachers, lawyers, Democratic voters, etc.) were these people chosen from?

10. Is this either a dial-in poll (e.g. a 900 number), a mail-in poll, or a subscriber coupon poll?

11. What is the sampling error for the poll results? [This assumes that the survey was based on a probability sample, of course.]

12. What other kinds of mistakes can skew poll results? [In particular, what was the nonresponse rate?]

15. What other polls have been done on this topic? Do they say the same thing? If they are different, why are they different?

18. So I've asked all the questions. The answers sound good. The poll is correct, right?

When you read the next report of a survey, note how few of these questions are answered (or can be answered with the information given).

* A good methodologist could find a flaw in any survey--so what's the purpose of claiming something based on a survey?

Quite right! I intend that this course will make you careful but not a nihilist.

* I don't think the method of using a coin flip to determine R's answer would be valid.

Then you're willing to give me 2:1 odds if I bet on tails? Come see me in my office, and bring a lot of money!

* What was the type of question relating to the coin flip and cocaine use? Does that type of question have a name?

* Wouldn't it be un-neutral if in a question you used the word "favor"--implying that they should?

I'm not sure I understand the question. But you're right that the word isn't a neutral one.

* I didn't understand the variable part (with X, Y and Z). What exactly is X? What is the background stuff?

X is my shorthand for all the variables which we are interested in looking at as causes of the variable (or variables) Y. Among those X variables are almost certainly the variables that relate to a person's background--i.e., h/her life experience--gender, age, race, SES, and so on.

It really depends on your purpose. Open-ended questions are harder to code (i.e., harder to reduce to numeric data), but in my opinion they yield much more interesting data if the researcher is willing to spend the time digging for it.

These are frequently used in "exploratory" research, where we are interested in finding out in what terms the Rs think and speak. It is important (and too-little appreciated) that good research flow from our fidelity to the way Rs think. Good research doesn't flow from elaborate statistical manipulations of data that we get by forcing Rs into our frame of reference.

That can work. Tape recorders will put off some Rs, but the new, tiny recorders are a lot less threatening than the huge reel-to-reel recorders I used to use. So it's a tradeoff, as usual; you have to judge what will work, based on your knowledge of the world (and in particular, your knowledge of how your Rs will react).

* is there a sort of instructor's manual that tells one how to do a survey, or will there be problems all the time?

There are a number of good books on doing surveys, including one by Babbie, as I recall. But you will always be running into new problems (and interesting new techniques!) in surveys.

* I'm aware we would be learning statistics in our analysis. How will they relate to a survey population?

* How do you approach people to answer a questionnaire which looks to be annoying?

I take it you're talking about the homework assignment. Just approach them as usual, and then explain to them afterward what you're doing.

* Is there any quick way to devise and conduct a survey without sacrificing its utility?

In most cases, yes--but there is no general answer. You have to use your cunning and your knowledge of the world.

One way to save time is to use questions already developed in other surveys. Another way to save time is to do telephone interviews. These techniques are not always a good idea, of course.

"Social desirability" means that one answer is more acceptable than another in our culture, so people will tend to give that answer even if it isn't the truth. From a survey researcher's point of view, therefore, social desirability is a nuisance.

* How can you be sure you are getting an accurate number when you do the coin flip?

You can't. Your analyses have to take into account that you will get varying numbers of heads. This is, of course, a form of sampling variance.

* I thought it was better to ask sensitive questions at the beginning or middle to keep the respondent interested.

Like all things in life, it's a tradeoff. You have to be the judge, based on your knowledge of the world.

I don't have time to discuss everything, and the other things I could say on the topic involve more detailed knowledge of survey research.

Probably not. But your question seems to imply that there are no degrees of bias. There are; our object is to keep the bias as small as possible.

* I learned a lot that I'll need to do in order to make my poll for Public Opinion and Propaganda more effective.

* I'd like to see a GOOD example of a recent, short survey to see how a GOOD survey should look.

You can pretest them, ask other professionals their opinion of them, and so on. But ultimately, you can't be sure.

CONSIDERATION ("R" = "Respondent"; "I" = "Interviewer")	INTERVIEW	PHONE	MAIL
Cost (incl. time, organizational effort, & decreased ability to get a large enough sample)
Response rate, incl. nonresponse / refusal / noncompletion
Response rate for individual qs (avoiding "DK" & "NA")
Length of the survey (# of qs possible)
Intimacy or rapport w/ R
Flexibility of interview: ability to explain, follow up, clarify, or change order of qs
Ability to obtain data even if R refuses to participate (e.g., information about race, sex, house, neighborhood)
Ability to probe
Danger to I
Risk of I influencing R, incl. "social desirability"
Ability to randomize & work w/out a list frame
Anonymity/confidentiality
Ability to answer open-ended qs at length
Ability of I to pick up on nonverbal cues
Ability to control whether R is in the universe (e.g., to ensure that the survey isn't answered by a child, or by two people discussing the answers w/ each other)
Ability to enter answers directly into a computer
Ability to arrange for callbacks or revisits
Ability to prevent or control "curbstoning" & other I problems
Ability to get responses returned, analyzed, & reported quickly