text:
A Quick Guide to Audience Research
Dennis List
Original Books Wellington, New Zealand 2006
Original Books Head office: P O Box 6637 Marion Square Wellington New Zealand Publisher: Niel Wright
Quick Guide to Audience Research ISBN 1-86933-727-1 Version 1.6, May 2006 copyright Š Dennis List, 2006
International distributor: Audience Dialogue 1 East Tce Nailsworth Adelaide SA 5083 Australia www.audiencedialogue.org
Quick Guide to Audience Research
Introduction and scope of this guide ................................ 4
Beginning with secondary research ................................. 9
Situation analysis ................................................. 9
Media impact assessment ..................................... 11
The survey process...................................................... 16
Sampling............................................................ 17
Writing a questionnaire ........................................ 20
Gathering data ................................................... 22
Processing the data.............................................. 22
Analysis ............................................................. 25
Interviewer surveys .................................................... 28
Face-to-face surveys ........................................... 28
Telephone surveys .............................................. 30
Observation ........................................................ 32
Questionnaire surveys ................................................. 33
Mail surveys ....................................................... 34
In-publication questionnaires ................................ 34
Visitor surveys ................................................... 35
Audience workshops ............................................ 36
Internet surveys.................................................. 36
Qualitative research .................................................... 38
In-depth interviews ............................................. 38
Consensus groups ............................................... 39
Response cultivation ............................................ 43
How to choose a method ............................................. 44
Whether to do a survey ................................................ 44
Choosing a survey method............................................ 44
Choosing a qualitative method ...................................... 47
Conclusion: Use your findings! ...................................... 48
Further reading....................................................................... 52 Appendix 1: Simple 1-page questionnaire................................... 52 Appendix 2: Glossary of audience research terms ........................ 53
Quick Guide to Audience Research
Audience research is for any organization with an audience - whether that audience is called listeners, readers, viewers, visitors, customers or users. Reading this Quick Guide won't make you into a professional researcher, but it should give you a good understanding of the simpler audience research methods: which methods you could use, whether you should commission a research company to do the research, or do it yourself. If you decide to do it yourself, this Guide will show you how you might begin. If you want do to audience research for the first time, I recommend that you choose the most suitable example, and change it as necessary for your situation. Some people say "Why do we need fancy audience research? We already get plenty of feedback from our audience?" The answer: feedback is usually unsystematic, and can't be trusted. People who are dissatisfied with a service are less likely to contact the provider (unless there's a sudden change in the service), so unsought feedback often gives you an unrealistically favourable view of the service. Audience research is different because it is systematic, and it tries to cover the entire population. It does not view the world through rosycoloured glasses, and seeks objective findings.
Reasons for needing audience information
There are three common reasons why media outlets want information about their audiences: 1. Because they have in mind to take a decision, and want to know whether the audience will accept it. (For example, a radio station manager, noticing that all other stations broadcast news bulletins at the beginning of each hour, may wonder about having bulletins halfway through each hour.) Often there's a choice of going ahead with the decision, or making no change, or perhaps a compromise and partial change. 2. Because they simply want to understand their audience better, without necessarily making one particular decision.
Quick Guide to Audience Research
3. To inform others (such as potential advertisers) about their audience. For making a decision, or informing others, the usual research method is a quantitative method: generally a survey – or perhaps a situation analysis, or response monitoring. For understanding the audience, a qualitative method is best: such as a set of in-depth interviews or consensus groups. However, if you have no knowledge at all about the audience, a basic survey is a good starting point. (Consensus groups are halfway between qualitative and quantitative, so they can serve both purposes.)
Audience measurement research A question: what do radio and TV have in common, that no other industry has? Answer: every other industry can count its users. Newspapers know their circulation and their print run, factories know how many products they produce, and service industries can count their clients. But with radio and TV, the program goes out into the air, and there's no way of knowing whether everybody tunes in - or nobody. Not without audience research. So for every other industry, research is optional, but for radio and TV it's vital. If you want to convince potential advertisers that they should advertise on your station, or donor agencies to fund you, they may ask "How can I be sure that you have an audience?" That's why the commonest type of audience research for radio and TV simply measures audience sizes. A large international industry has evolved to serve these needs, served by multinational research companies such as A C Nielsen, TNS, and Arbitron. To measure TV audiences, for example, they use "peoplemeters" – devices attached to TV sets in sampled households, to automatically record the programs viewed and transmit that data to a central computer. For radio
distributed to households for completion, and collected a week or two later. As these research methods are very expensive, the reports are often syndicated: i.e. a group of broadcasters or print media owners shares the cost of the surveys.
Quick Guide to Audience Research
Who should do the research?
Consider doing the research yourself if most of these statements are true... You have studied social sciences at university level. You are able to remain objective - really able. (Not many people are.) If the research finds that people hate your special program, can you face the facts? Or will you quietly toss the findings into the rubbish bin? You can't afford to buy research from a commercial market research company. You have plenty of time. (If you do research yourself, it can be cheap, but it takes more time than most people first expect.) Commission research from a market research company if most of these are true.... You need to convince potential advertisers that your station has a large audience. (If you do the research yourself, no matter how well, they may not believe you.) You have more money than time. You need help with deciding exactly what you need to know. You are mainly interested in finding out the size of the audience, not their opinions. However, a market research company cannot tell you what you want to know, nor how to use the results. You will still need to spend a lot of time thinking about what to find out, and how to apply it. Apart from doing a whole survey yourself, and commissioning it from a research group, there are several other possibilities that may not occur to you at first - syndicated research, omnibus surveys, and shared surveys.
Syndicated research This is the international "peoplemeter� and radio diary research mentioned above. National readership surveys are also syndicated. In most countries with large populations, these surveys are already being done. A radio or TV station that wants to find out its audience size can often subscribe to syndicated reports from these sources. Note that these surveys are designed to be used mainly by advertisers, and tend to exaggerate audiences, compared with some other research methods. Often they are not very useful for decisions about programs. Because these surveys are very expensive to carry
Quick Guide to Audience Research
out, they are also expensive to buy information from, even when there are many subscribers.
Omnibus surveys If you need a numerical answer to only a few questions, and the adult public in our country or area is a suitable sample, you can often buy a few questions on a shared survey where many other organizations also have a few questions. This is a cheap solution, but occasionally answers to some questions may be distorted by answers to previous questions. If you use this option, try to ensure that no preceding question covers a similar topic to your questions.
Shared surveys These are surveys initiated by a group of local media, all sharing the costs. Effectively this is the same as a syndicated survey, but the local group is in control. A small warning: the organization and administration of such surveys often takes a lot more effort than you first expect. Fierce disputes about question wording can occur.
Deciding which type of research you need
As mentioned in section 1.1 above, there are three main approaches to research, depending on what you intend to do with the results: understanding the audience, making a decision, and informing others. If your purpose is to understand your audience, you should know that this never finishes. In that case, a survey is not the best value-formoney way to get the information – a set of consensus groups or indepth interviews will give you much richer information, though indepth interviews provide no quantitative data. The decision-making method is quicker: do a survey, get the answer, and make a decision. If you already have enough data, you may not need decision-led research - but the annoying thing about data you already have is that it's seldom precisely relevant. If your purpose is to inform others - such as potential advertisers it's best not to do a survey yourself, or even to commission it yourself. The data will have much greater credibility if they are known to come from an independent source. For advertisers, the most credible surveys are those funded equally by all members of an industry.
Quick Guide to Audience Research
Researching all the audiences
A media organization might think of "the audience," but in fact most organizations have many audiences. This is not merely a matter of dividing up "the audience" in different ways. It is a rethink of the whole idea of the audience. These audiences are not only the direct consumers of the organization's output - who may be labelled (depending on the organization) as listeners, viewers, readers, visitors, users, subscribers, members, or customers. The other audiences include all the groups of people that deal with the organization. These audiences may overlap: they consist of roles, not individuals. For a media organization, these other audiences include program suppliers, funding bodies, advertisers, shareholders, staff,
regulatory bodies, other government agencies, lobby groups, political parties, non-government organizations - in fact any group of people that the organization deals with, even if indirectly. The reason for researching all the audiences is that for an organization to work well, all its audiences need to be satisfied in some way. To ensure they are satisfied, it helps to know what they are thinking - which requires some form of audience research. The difficulty of researching each audience will vary, depending on its size and accessibility. A common argument against researching these "other" audiences is that "we talk to these people all the time." Perhaps that's true for some, but talk is not research. Thus it can be very informative to contact samples of all these audiences and systematically discover the
organization. Here's an interesting project. Make a list of all your (organization's) audiences. For each audience, consider: its size (now, in the past, and likely future trends); how homogeneous that audience is; what you expect of that audience; what that audience is believed to expect of you; frequency of contact between you and that audience; what new information about that audience would be helpful: this is the basis of researching that audience.
Quick Guide to Audience Research
Beginning with secondary research
If you don't already know what secondary research is, it may seem strange not to begin this Guide by discussing primary research. Secondary research is given that name because it has already been done by somebody else. When you then use it, you are a secondary user. But why re-invent the wheel? Primary research is expensive, and as governments and other organizations collect and publish a lot of data – particularly for media industries, which are often highly regulated, it is sensible to review existing data before going out and doing your own study. This section includes two forms of secondary research: situation analysis and impact assessment.
Situation analysis
sources, all the relevant information about your current situation also about the relevant past, and the near future. This information can be categorized as follows: 1. Population data. Most governments conduct a census every 5 or 10 years, and publish results in fine geographical detail. It's useful to collect
population in the area your publication serves. 2. Information about your stakeholders – all the groups of people and organizations that you deal with. In particular: a listing of your competitors. Also relevant are your suppliers, those you exert power over (your staff?), and those who exert power over you (government agencies). Above all, there are your customers, both direct and indirect. For a radio or TV station or a print publication, "customers" includes not only buyers, but also listeners/readers and advertisers. Two particular types of stakeholder deserve close focus: Your customers / audience / readers. Doing a situation analysis, even if it doesn't find information about them, will help in clarifying what information might be needed for primary research.
Quick Guide to Audience Research
Your competitors: it's useful to make a list of them, to better understand the alternatives available to your audience. These include not only competitors in the same industry, but all competitors for your audience's attention. Another aspect of stakeholders is the pressures they are exerting on you. Your audience probably wants more of everything, more quickly than now, and at a low price. Your suppliers probably want smooth management, with a minimum of fuss and time wasting. The government probably wants you to report favourably on their activities. Though all of these pressures are obvious to some people, it's helpful to list them (on a stakeholder by stakeholder basis) and question to what extent each pressure is important – and whether any have been omitted. 3. Information about your own organization: staff numbers and positions, the inputs you receive, the way you transform those inputs, and the outputs you produce. This is often neglected in a situation analysis, because everybody involved with the analysis works for the same organization, and assumes that everybody know all this information. In fact, this is often not true, and presenting it in summary form is very useful for making decisions. 4. Trend-related information. What social and industry trends are affecting your industry, and your publication? Rather than rely on subjective opinions, it's preferable to try to measure real trends, by comparing current population and industry data with equivalent data from a few years ago. Does all this seem too obvious to be useful? Do you think you know this already, and it's a waste of time to write it down? Well, perhaps you know it all, but probably the other people you work with don't know it all. It's more likely that everybody knows a little of it. And what you think you know may be wrong – perhaps your knowledge is out of date, or perhaps you were misinformed in the first place. The advantage of situation analysis is that it collects all the data in a concise format, so that everybody who goes past can read the information, and make suggestions or corrections. As much of the information collected by a situation analysis consists of lists, tables, and (potentially) graphs, it's a good idea to present a situation analysis as a wall chart, or a set of them. Reports aren't read much, but displaying the situation analysis on a wall that a lot of staff see is a good way of keeping everybody in touch with the
Quick Guide to Audience Research
current situation. When working on a project for Radio Republik Indonesia (RRI) a few years ago, I visited their office at Banjarmasin, in Kalimantan, and found a whole meeting room with whiteboards on three walls. On those whiteboards was a useful situation analysis.
Media impact assessment
Many organizations automatically collect information about their audiences, but don't make much use of it. The principle of media impact assessment is to collate information collected for other purposes. You need to collect two types of information, and relate them. These could be called "cause" and "effect". The cause is your main activity, and ways in which it varies. The effect is the audience response: the numbers of people and the reaction they give. An organization can measure three kinds of things: inputs, outputs, and impacts. Inputs are the resources you use, to do whatever you do: money, equipment, and time that people spend. Outputs are what you produce: newspapers produce issues and copies, radio and TV stations produce programs, schools produce classes, museums produce exhibitions - and so on. Impacts
newspapers: how many copies were read, and what effect did that reading have? For radio and TV programs: how many people received the programs, and what effect did the programs have on their audiences? For schools: how well did the students learn? For museums: how many people visited, and what did they get out of it? You might have noticed there are two kinds of impacts. The immediate one is sometimes called "outcomes": e.g. audience size. The longer-term impact is the result from being an audience member. For a school, the immediate outcome is how many pupils attended; a longer-term outcome is how much they learned. For a radio station, the immediate outcome is the audience size; the longer-term outcome is the effect of the program. If it was (for example) an antiAIDS campaign, how many people started practising safe sex as a result of it?
Quick Guide to Audience Research
The role of media impact assessment is to relate impacts to inputs: through what processes are inputs transformed into impacts? When the mechanisms are understood, the process can be improved. It works like this: Step 1: List the inputs, or causes Step 2: List the outcomes and impacts, or effects Step 3: Try to work out exactly how the inputs cause the outcomes. Impact assessment usually measures short-term impacts. Long-term impacts are harder to measure than short-term ones - but since the main purpose of many activities is to have long-term impacts, it's worthwhile to try to assess these – though (obviously) they take longer to assess, and there are often so many potential causes that it's hard to which out which of them led to the effects. For that reason, assessing long-term impacts is best done by setting up multiple measures. If all the measures end up pointing in the same direction, this adds evidence to the effectiveness of the process being evaluated.
Example 1: The impact of a social marketing campaign This is an ambitious type of assessment, because almost any largescale communication has multiple effects, and also because sought effects have multiple causes. Therefore, trying to trace a link from a single cause to a single effect can be almost impossible. A good example would be an anti-smoking campaign on TV: a series of (say) 70 commercials, broadcast on one channel, over a period of a month. The desired behaviour is: A cigarette smoker sees one (or more) of the commercials Learns: "Smoking is bad for me - I must stop" Never smokes a cigarette again. Of course, it hardly ever happens like that - otherwise everybody would have stopped smoking long ago. Realistically, people stop smoking because of pressures from all directions: the disapproval of others (both specific others and people in general), restriction of places where smoking is allowed, increasing prices of cigarettes, nasty medical photos on cigarette packets, and so on. The antismoking campaign - which has taken place, on and off, for around 30 years in developed countries, has been an excellent example of effective social change, but it hasn't happened easily or quickly. The
Quick Guide to Audience Research
addictiveness of tobacco has been part of the reason for this, but other social changes are equally slow: they involve changing not only individual behaviour, but also the social context. But the focus here is on impact assessment. A simple set of criteria for impact assessment for that set of anti-smoking TV commercials would be: For there to be any impact, people must see the commercials. So the first measures are of audience size: how many people saw the commercials each possible number of times, from 1 to 70. For the commercials to be effective, they probably need to be remembered. So the second impact measure is how many people remember seeing the commercials. How many people actually did something about the commercials they saw - and what was it they did? Options include mentioning the commercials to a smoker, mentioning to a non-smoker, smoking X fewer cigarettes before resuming the previous habit, giving up smoking entirely - and so on. There are always a few relevant actions that can't be anticipated easily. Depending on the budget available, the actual research could be more or less elaborate than the above. The source for all three types of information would be a survey of the general public in the service area of that TV station, carried out within a few weeks after the last commercial was shown. As long as the survey used a random sample, results could be projected to the entire population of that area. The obvious problem with that approach is that the impact measures would be based purely on statements by respondents. Some respondents would try to please the interviewer by exaggerating the extent to which they'd cut down on smoking. The "gold standard� solution would be to conduct an experiment, instead (or as well as) the survey. This could involve choosing around 30 geographical areas, surveying all the populations beforehand to estimate frequency of smoking, running the commercials in half of the areas (chosen at random), then measuring the frequency of smoking after the campaign. Though sounder in theory than an after-only survey, the experimental method can produce unexpected results, and is far more expensive. Nor is it helpful in working out what to do next: if the campaign was very successful or very unsuccessful, you'll never know exactly why. But if you use the survey method, you can collect a lot of information from respondents about how they reacted to the commercials.
Quick Guide to Audience Research
Instead of spending vast sums of money on an experiment, it's often possible to draw some conclusions from data collected for another purpose. For example, you might be able to get statistics on the numbers of cigarettes sold in the area in the month before and the month after the commercials. This information may not mean much by itself, but could be used to compare the survey data with. If the survey indicated a 10% drop in smoking, this should be reflected in cigarette sales. Another possibility for verifying survey statements would be to ask other people in the smoker's household if they had noticed the smoker had cut down since the campaign. Any form of verification will add substance to such survey data. A separate problem is that the effects might be delayed. It could be that, a year after seeing the commercials, some people might decide to give up smoking - purely due to those commercials, not for any other reason. In fact, it is very rare for this to happen as a single reason, but the drip-feed effect will probably have some impact in the end - though this will not be attributable to any single campaign. You can see from the above discussion how messy this type of impact assessment can be. Whole books have been written on this subject. But for a simple approach, the after-only survey, with some attempt at independent verification, is often adequate - given that the budget for impact assessment never seems to be enough to do it properly.
Example 2: Orchestral concerts I helped set up an impact assessment for an organization that I worked with. This example is about an orchestra that held concerts several times a week, and wanted to find out how to attract larger audiences
denominator. So we set up a database that related their box-office figures to the music they played. Imagine it as a spreadsheet: each row applied to one concert, and each column applied to a particular piece of information about that concert. Some of the columns were inputs
("effects"). Inputs = (a) The content – the music played, the musicians. (b) The publicity – advertising budget, number of ads, estimated readership of the ads.
Quick Guide to Audience Research
(c) Other factors. In this case there were a lot, including accessibility of venue, day of week, time of day, competition from other attractions, and the weather a few hours before the concert. Outcomes = (a) People – e.g. number of tickets sold. (b) Money – e.g. revenue from tickets sold. (c) Reputation – summarized reviews of the concert, rated on a 5point scale, from Poor to Excellent. In impact assessment, it's important to take time-lags into account – a reputation can lag years behind actuality, for people who don't have a lot of contact with the organization. With an orchestra, maybe people are still staying away because they didn't like the previous conductor, and don't know that he left five years ago. I've noticed in many impact assessment studies that staff of the media organization being studied often assume that the content is by far the main factor in determining outcomes. But in fact, content is more often than not quite a minor factor, because it varies only within a narrow range. Relating inputs to outputs can be treated as a mathematical problem, producing a formula. Perhaps you could find a friendly statistician who could help. For the above exercise, I used a statistical technique known as regression analysis, but different techniques will be needed for different types of situation. For the formula to be reliable, the number of incidents needs to be fairly large: this example was based on more than 100 concerts. Impact assessment can often be improved by doing tiny surveys. Often a sample of 20 is enough, if it's fully typical of the population. "Why did our February concert get double the audience of our March concert?" This example was a one-off study, but usually impact assessment involves setting up an ongoing monitoring system: filling in a spreadsheet on a regular basis, to help keep track of the causes and the effects.
Quick Guide to Audience Research
The survey process
We now move from secondary research to primary: the type of work that most people think of as audience research. With secondary research, there's often no data available about individual consumers; with primary research, data is always collected at an individual level – but it is usually a sample of individuals, not the whole population. Each person in the sample represents a group in the population. Primary
Quantitative research produces outputs in numeric form – such as "We have 37,000 readers on the average day" or "45% of our audience are male, and 55% are female." Qualitative research produces statements that cannot be quantified; the implication is that the statements apply to the large majority of the audience – such as "Hardly any of our readers are interested in news from Mongolia." Section 6 below covers some of the simpler qualitative methods, but before that let's look at the most common quantitative audience research methods: surveys. A survey, as generally understood, has these components: A questionnaire: a fixed set of questions, from which respondents choose one or more preset answers for each question A sample of respondents, forming a representative subset of a defined population Findings, usually in percentage form, but often accompanied by illustrative comments. These are the steps in doing a survey. 1. Decide what you need to find out. Not what you'd like to find out (that's limitless) but what you really need to know. 2. Plan a sample (that is, how to find people who will answer the questions). 3. Write a questionnaire. Then try it out with a small number of people (10 is often enough) to see what problems there are with it. When the problems with the questionnaire have been fixed... 5. Distribute the questionnaire to the sample. This can be done in various ways, each of which is covered in more detail later:
Quick Guide to Audience Research
Interview surveys - which require professional interviewers. These can be either face-to-face surveys or telephone surveys. Self-completion questionnaires - which respondents fill in themselves. These can be distributed by mail, handed to people for them to fill in on the spot, or done through the internet. 6. Process and analyse the completed questionnaires – either using a computer, or manually. 7. Come to conclusions, and distribute the findings - usually as a written report. These steps need to be done in the above sequence. Taking shortcuts by trying to do two of them at once can cause a lot of duplicated effort. There are many types of survey, some using interviewers and some using questionnaires without interviewers. These types are all covered in more detail below, but first we consider the principles common to all surveys: sampling, questioning, and processing and analysis.
A lot of people think all they need to do when preparing a survey is to write a questionnaire. They put a lot of effort into producing a detailed questionnaire, but almost no effort into deciding who the respondents should be. In fact, it's almost more important to have the right respondents as the right questions. The people who are most easily accessible are not always the best ones to interview. Every survey takes a sample of a larger population. If that sample is a true cross-section of the population, the percentages found from the sample can be validly applied to the population. So if 80% of the sample like your program, it follows that 80% of the whole population will like your program – but only if the sample represents the population correctly.
Quick Guide to Audience Research
For a sample to be valid, everybody in the population should have the same chance of being interviewed.
What's the population? "Population" here is not just the number of people: it's also the type of people to survey. You need to define it: both geographically and in terms of people excluded. As it's not feasible to interview young children in a general survey, most surveys have a lower age limit: often around 12 to 18 years. If you need to survey children, different methods are needed, and probably a different project. The other limit is geographical. For most media, this could correspond to their main transmission or circulation area. By following legal boundaries (local government etc) you can probably get census data, and a close estimate of the number of people in any combination of local government areas. Well, it might have been close once - but maybe the last published census figures are 10 years old. In that case, you might be able to get some more recent estimates. Even if the figures are a few percent out, that's close enough for most surveys.
Projection Let's say the population in the area you can survey is 450,000, but 100,000 are below your lower age limit of 15. If you can afford to survey 350 people, that's 1 in every 1000 of those eligible. That doesn't sound like many, does it? But if the sample is carefully selected, about 200 is plenty for most purposes - as long as you don't expect accurate estimates for small sub-groups, such as men aged 35 to 44. So to calculate the estimated number of listeners to your station, just multiply the number of listeners in the survey by 1,000. If you surveyed 350 people, and 120 of them listened to your station, that suggests your total audience is 120,000. It's simple in principle, but in fact things are more messy than that. What if 60% of the respondents are women - and women are less likely to listen to your station than are men? That will mean that the findings under-estimate the number of listeners. Assuming there are an equal number of men and women in the population, one way around this problem is to calculate the numbers of male and female listeners separately, and add the two together to get a (perhaps) more accurate total. This is called weighting. It's one thing to know the total population figure, but quite another to get an accurate cross-section - and to know that it's accurate. The
Quick Guide to Audience Research
most accurate method of sampling is usually to interview people in their homes. That's because (almost) everybody has one and only one home. So if you can give each home an equal chance of being surveyed (it's easier with homes than with people, because homes don't move around) in-home surveys are the most accurate. But they're also slow and expensive. You have to send interviewers to homes all over the survey area. Often it takes several visits to find somebody at home. If you're doing a radio or TV survey, you should know that TV and radio are used most at home. People who are not at home are less likely to be using radio or TV. So if the interviewer doesn't return to find the people who were out on the first visit, you will get audience figures that are too high.
Random sampling A sample is said to be random when every member of the population has an equal chance of being surveyed. The advantage of random sampling is that you can calculate mathematically how wrong your survey results are likely to be. With surveys. there's always a sampling error, because the people you didn't interview might have given a different answer. The larger the sample, the smaller the error – but to halve the expected error, you have to quadruple the sample. This gets expensive. In the end, you learn to live with sampling error. But if you have no information at all about your audience, even a small sample size (with correspondingly large error margin) will give you useful new information.
Quota sampling A quota sample is designed by dividing the population into groups, and interviewing a fixed number in each group. For example, if there are equal numbers of men and women in the population, the quotas for men and women should be equal. But unlike a random sample, where respondents must be first contacted at home, a quota sample can find respondents anywhere. Quota sampling is usually quicker and easier than random sampling. The main problem is that you can't accurately calculate sampling error - which tends to be larger for a quota sample than for a random sample. One way to overcome this problem is by taking several separate quotas, and comparing the results. The example in section 4.1 below, on a series of rapid face-to-face surveys done for a radio network in Indonesia, shows how this can be done.
Quick Guide to Audience Research
Writing a questionnaire
How many questions to ask The number of questions you could ask about your audience is almost infinite. Once I tried to design a comprehensive questionnaire, asking everything about audiences that you might want to know. It included more than 1,000 questions - and the more questions I added, the more gaps I noticed. So it's a good idea to start small: no more than two pages of questions. That's about 15 questions. Beginners often make the mistake of asking about everything they can think of. This greatly increases the cost and time, and may decrease the accuracy (because respondents get bored). If you really need answers to all those questions, do another survey later. The second survey will always produce better results, because you will have learned so much from your first survey.
Deciding what to ask As mentioned above, there are three contrasting approaches to audience research: measuring the audience, understanding the audience, and solving a particular problem. Surveys are most useful in measuring audiences, and addressing specific problems. A problem suitable for a survey can often be expressed, "If we take action A, will that lead to outcome B?" The answer is usually "It depends on the circumstances, C". Therefore a good way to design a problem-based questionnaire is to include three sets of questions: A: possible actions that could be taken B: possible outcomes that might arise from those actions C: possible situations in which the actions might lead to outcomes. However, it's little use asking people what they might do , in some particular situation, if they haven't experience that situation. Results are more usable if they relate to current behaviour, not future behaviour. It's also possible, sometimes to check such answers against other sources.
Quick Guide to Audience Research
Types of question There are four main types of questions: Questions about behaviour – such as "Did you listen to radio yesterday?" Questions about opinions – such as "Do you approve of the local Mayor?" Questions about the respondent – such as "How old are you?" (these are called demographic questions: age group, gender, religion, languages spoken, occupation, income, family size, etc) Questions about objects and possessions.- such as "Do you have a TV at home?" When
Firstly, work out what you need to know (research questions). Then work out what questions need to be asked to get that knowledge (survey questions) For a measurement-oriented survey, taking as an example a newspaper entitled The Chronicle, your research questions might include these: 1. How many readers does The Chronicle have? 2. What proportion of issues do they read? 3. What types of article do they like most? And least? 4. What sort of people are our readers? 5. What do they do as a result of reading our articles? 6. How many people know that The Chronicle exists? 7. How could we get more readers? The corresponding survey questions might be: 1. Do you read The Chronicle at least once a week? 2. In the last seven days, which issues have you read or looked into? 3. Here are some types of article you see in The Chronicle. For each one, please tell me if you like to read it a lot, or a little, or not at all? 4. What age are you? Which sex? What's your occupation? 5. Thinking of the articles in The Chronicle that you've read in the last week, have you taken any actions based on what you read? 6. Which newspapers can you name that circulate in this area? … Any others? [This would have to be one of the first questions, before The
Chronicle was mentioned.] 7. [Ask those who answered No to Q1]: Is there any reason why you don't read The Chronicle?
Quick Guide to Audience Research
[For those who answer No to Q7, and no other barriers exist, their responses to Q3 would be relevant.] Notice how the survey questions are different from the research questions. Often, one research question produces several survey questions - and sometimes one survey question can help answer several research questions. Research questions can be in any order, but survey questions often need to be in a certain order to make sense – for example, question 6 would have to be one of the first asked. Though it seems like more work to write two sets of questions, it usually saves a lot of time in the end, it forces you to consider about what you want to know, and how you could apply the findings.
Gathering data
When you have designed a sample and written a questionnaire, the next step is to get respondents to answer the questions. So you need a distribution method for the survey. Two main options are possible: relying on interviewers, and relying on questionnaires. Each option includes several distribution methods: interviewing can be done faceto-face or by telephone, and questionnaire surveys can be distributed by mail, through print media, personally, and via the internet. Sections 4 and 5 below cover these options in more detail.
Processing the data
Personal interviews can be with one person at a time, or several people (e.g. a household). Telephone interviews are with one person (at a time). With mail and internet surveys, you can not know how many people were involved in answering one questionnaire (unless you include a question about that). When all the questionnaires have been filled in and returned, the hard work is about to begin. Now it's time to tabulate and analyse the results. For this, you need to be well organized. People who are well organized
questionnaire twice, and they hardly ever make clerical mistakes. If you're the sort of person who is impatient with small details, you'd
Quick Guide to Audience Research
better find somebody else to do the survey analysis - perhaps somebody with an accounting or book-keeping background, or a mother who's had a lot of children. The stages of processing and analysis are: 1. The questionnaires completed by each interviewer should come back with a completed log, showing how many questionnaires they have completed. As the completed questionnaires arrive, check that the numbers on each log match the number of questionnaires returned by that interviewer. 2. Take enormous care of the completed questionnaires! If they are lost, all the interviewing work has been wasted. Count them carefully, and keep them in a safe place. Do the data entry as soon as possible. 3. Data entry (recording the results in coded form). 4. Counting the coded results. 5. Analysing the results and producing a report. 6. Presenting the results, and distributing the report. 7. Acting on the results.
Manual processing Most people these days use computers, but if you have only 100 or so questionnaires, and each questionnaire is on a single piece of paper, it's not difficult to analyse them by hand. There are two ways to do this: one question at a time, and one questionnaire at a time.
Method A: One question at a time For each question in turn, sort the questionnaires into heaps, with one heap for each possible answer. Count the number in each heap, write it down, then sort the questionnaires into more heaps for the next question. Don't do this in a windy place!
Method B: One respondent at a time The other way of doing manual analysis is to record all the answers from each questionnaire once. This involves less handling of questionnaires, but requires tallies to be entered on paper. There are two ways of doing this...
Method B1: Set up a tally sheet for each possible answer, and make a mark for each answer. Here's an example, for a survey with just 6 respondents and 3 questions, where each question has two possible answers:
Quick Guide to Audience Research
Q1=yes ////
Q1=no /
This is very quick, but it's all too easy to make a mistake. When the number of tallies doesn't add to the total number of questionnaires, you have to go through all the questionnaires again. This often happens when somebody interrupts your counting. Not so quick after all!
Method B2: A safer way is to give each questionnaire a number, and write that number instead of the tally mark. Then it's possible to recheck - e.g. Line
Q1=no 2
If you enter one questionnaire number on each line, and the lines are numbered, you only need to look up the line number of the last answer in each column, to see how many people gave each possible answer.
Which method is best? The one-question-at-a-time method involves more paper handling, but it's safer - specially if you're interrupted while counting. It works best when questionnaires are on a single piece of paper. Both methods take about the same time to do. One-respondent-at-a-time can be faster, when questionnaires are long but it's easy to make a mistake, by entering a number twice or skipping a number. If you add time for thorough checking, the one-respondent-at-a-time method takes longer. Unless you're very well-organized, and you know that you'll be able to work without interruption, I recommend the one-question-at-a-time method.
Computer processing This has two steps. First, all the answers from all the questionnaires are entered into a computer file. There are two main types of program you can use for this: statistical software, and spreadsheets. Statistical software (such as SPSS and Epidata) is designed for surveys, but not many computers have it. SPSS is very expensive,
Quick Guide to Audience Research
but very common in universities (perhaps you can find somebody who can help you with it). Epidata is free, but not so well known; you can download it from the Web, at www.epidata.dk. Spreadsheet software, such as Excel and Lotus 123, is installed on many computers, but is not easy to use for survey analysis. The Audience Dialogue website has a set of web pages on how to use Excel for analysing surveys – see www.audiencedialogue.org/excel1.html. General advice: it takes a long time to learn to use software well, so if there's some software you (or people on your team) already know well, try that first.
To enter questionnaires onto a spreadsheet Set up a spreadsheet with one line for each questionnaire, and one column for each question. The top line should be the question numbers or brief headings such as Q3 for Question 3, or AGE for age group. The shorter the headings, the less horizontal scrolling will be needed. But the headings need to be long enough that you can check you're entering the correct data. The questionnaires should be numbered, and stored in numerical order, so that if you find a problem on the computer file you can check the original questionnaire. The questionnaire number is usually the first column in each line. In each cell on the spreadsheet, don't enter the full answer, but a code number or letter. For example, to enter answers from the question "Which sex are you?" don't type in "male" or "female" just M or F. This saves a lot of time - as long as you keep a record of what the codes mean. (It's best if codes are printed on the questionnaire.) Save the file after each questionnaire is entered, in case there's a power cut or somebody trips over the cable! After each session of data entry, copy the file to a floppy disk or other storage medium (such as burning a CD, or on a USB flash drive), in case the computer develops a disk problem.
Continuing the spreadsheet example above: for each column in turn (i.e. each question in turn - except the one containing the questionnaire numbers) count the number of different answers. For example, the column for "Which sex are you?" may have 100 entries: 52 might
Quick Guide to Audience Research
be F, and 46 might be M, and 2 might be blank because the interviewer forgot to note the respondent's sex. Ignoring the unanswered questions, this example has 98 answers. 52 out of 98 (53%) are male and 46 out of 98 (47%) are female. Repeat this process for every question. If your spreadsheet software can do Pivot Tables (like recent versions of Excel) and you know how to use these, you can get a result in a few seconds - but beware of problems caused by missing answers. If the answer to a question is a numerical scale, such as "How many marks out of 10 would you give the local news on FM99?" you can use the spreadsheet to calculate an average, instead of (or as well as) the counts for each answer. Again, errors are often caused by missing answers. If somebody doesn't answer that question, don't record the answer as 0 out of 10, but as a blank space. However, analysis is more than just counting: it's working out how the results can be used. For each question in turn, look at the numerical results. It's often easier to understand them if you draw them as a graph - even a rough one. Then ask yourself: What do those numbers mean? What are they telling us? What action do they suggest? If, by themselves, they mean nothing, what other answers do we need to compare them with? It's best to have several people do this, in a meeting. A single person can easily miss a conclusion.
Check the validity You can check the validity of your results by comparing the percentages with known figures. For example, if the population you are taking the sample from has an equal mix of men and women, your sample should have a 50-50 split between the sexes. Most surveys have a slight surplus of women, perhaps because they're easier to interview than men. But if your sample should have 50% women, and it actually has 60% or more, you may need to compensate the results for the lack of men.
Quick Guide to Audience Research
Most surveys include a question on age group. Thus another check on a survey's accuracy is to compare the ages of respondents to the survey with the ages of the whole population in the survey area, using Census data. Dividing the sample into four broad age groups (e.g. 10-19, 20-34, 35-54, and 55-plus) will usually reveal any problems with a broad age group. Many surveys don't include enough young people - they're often harder to interview, because they're away from home a lot. If your survey has a major imbalance in sex or age groups, the best way to correct the problem is to go out and interview more people to correct the imbalance. If you can't afford to do that, you should at least draw attention to the imbalance, and consider it when making any decisions based on the survey.
The need for comparison Suppose your survey produces the result that 37% of people listened to radio station FM99. If you have no previous survey data, you won't know whether that is a lot or a little. This shows that numbers are meaningless
something. Comparisons can arise from four main sources: Previous data from the same station or area Data from other similar stations or areas – often known as "benchmarking". Expectations of those involved with the survey. To obtain realistic comparisons, ask those involved to guess the results before the survey takes lace – then compare the guesses with actual results. In an academically oriented survey, there is often a theory being tested. This often takes the form "A causes B". The measured audience can be either A or B, and some other factor is the cause or effect. The advantage of making comparisons is that it forces you to examine why the measured audience is different from expectations. Without making explicit comparisons, people tend to say "that's interesting" – but not to use the findings to improve the organization commissioning the survey.
Quick Guide to Audience Research
Interviewer surveys
This section describes the two types of survey that use interviewers: in a face-to-face situation, and by telephone. The face-to-face surveys can take place in several types of venue, which often make a difference to the sampling: in respondents' homes, in public places, in workplaces, and at points of sale.
Face to face surveys
A face-to-face survey, as the name suggests, is one where interviewers to out with a heap of blank questionnaires, select members of the public using some sampling system, and interview the people sampled. The questionnaire can be quite complex, because the interviewers fill it in, and they have been trained to use it well. The interviewers also keep a record of the process they have used to choose respondents. Sometimes starting points (such as street corners) are chosen at random, but some elements of the sampling are done by the interviewers. As an example, I've had good results with quota sampling (e.g. in a series of surveys in regional cities in Indonesia), with a design like this: Planned total sample size = 240 people. Interview 80 at home, 80 in public places, and 80 at their workplaces. In each group of 80, interview 40 men and 40 women. Each city to be surveyed was divided (roughly) into areas of high, low, and average wealth. The wealth of the areas was determined informally, using local knowledge - not from any official data which was unobtainable. A quarter of the interviews were in an area where rich people live, a quarter in poor areas, and the other half in an area of average wealth. (Bear in mind that not everybody found in a rich area is rich - or vice versa.) The purpose of segmenting the sample between rich, poor, and average areas was to reach a wide range of the population. In Indonesia, almost everybody speaks Indonesian, but in countries where different groups speak different languages, a languagebased segmentation would be more relevant. Sometimes a
Quick Guide to Audience Research
religion-based segmentation would be more appropriate. Ask yourself "what is the main demographic variable in this area that's likely to be most strongly related to the topic of this survey?" The quota system should reflect that. In the Indonesian studies, each city had about 16 interviewers - who in fact were broadcasters, and had never done research interviews before. They went out in pairs. One person asked the questions, and the other recorded the answers, and kept track of the quotas. About 200 interviews were completed, lasting between 5 and 10 minutes each. Everything was done in one morning. A good way to check the results of a quota survey is to repeat some questions that were included in the census. In this case, we asked everybody's age group and educational achievement, and compared the answers with census data. No difference was more than a few percent, which helped us feel confident that the other results were accurate. Though it's possible that we missed out on some types of people entirely, they'd have had to be the same age groups, sexes, and educational level as the others. To keep track of the quotas, each pair of interviewers has a log to fill in. Every time they get an interview, they number that questionnaire, and write the number in a space on the log. There were 30 spaces on the log, so the total daily quota was 30. (That works out at 4 interviews per hour, with about half the time spent interviewing, and half finding respondents.) The number of each interview is entered on the log twice: once in its location (home, work, public place) and again for sex. Why? Wouldn't it be easier to have a grid, and interview (say) 5 men and 5 women in each type of place? The problem is that sexes are not necessarily evenly spread between locations. Generally, most people interviewed at home are women, and most people interviewed at work are men. In Islamic countries, few women spend much time in public places. So to set an equal quota for sexes in each location could produce a less representative sample. Home Work Public place Men Women
Quick Guide to Audience Research
Rich/poor/average areas didn't appear on the logs, because each pair of interviewers worked in only one area: rich, poor, or average. They were also asked to make no more than 3 interviews in any one street or workplace. Public places included streets, shopping centres, markets, bus stations, etc. People working in public places (e.g. driving buses or serving in shops) were counted in the work quota, not the public place quota. For print media, a good public place to interview people is at the point of sale: near street sellers and news kiosks, and other locations where newspapers and magazines are sold. Combining home, work, and public-place interviews works well because of the mix of age groups obtained. Interviewing people at home finds those who tend to be older than average, and more women. People at work include more men, and more people in the middle age groups. People in public places tended to be the youngest group, including students, the unemployed, and the poor. Did you wonder: why write in the questionnaire number? Why not just tick each box? There's a good reason: writing in the number ensures that a box isn't ticked twice by mistake. That's a very easy mistake to make, when the interviewer is interrupted.
Telephone surveys
If you live in a country where nearly every household has a phone, and where phone surveys haven't (yet) become so common that everybody groans when they get a call from a research company, a phone survey can be a very efficient way of getting results - specially if the sample is widely scattered. Though long-distance calls aren't cheap, they are usually cheaper than the cost of interviewers' travel. The best time of day to do a phone survey of the general public is when most people are at home, and not very busy. In Western countries that's usually evenings and weekends. Most telephone surveys use several interviewers, and it's best if everybody works at the same time, in the same place. Then, if a problem arises, all the interviewers can quickly be told about it. And if everybody is working together, the printed questionnaires that the interviewers fill in are all in the same place, and harder to lose. It's possible to use computers instead of printed questionnaires. However
Quick Guide to Audience Research
the software (called CATI - short for Computer Assisted Telephone Interviewing) is expensive, and not easy to learn, so this is normally used only by professional research companies.
Example of a telephone survey In central Australia, the population is small and widely scattered. It would cost a fortune to do a face-to-face survey there, but as almost everybody has a phone, a telephone survey is the obvious method to use. When I was working for the Australian Broadcasting Corporation (ABC), we organized many telephone surveys for remote areas of Australia, mostly for rural radio. The research questions were usually: What percentage of people in each area are aware of their local ABC radio station? Which other radio stations in their area do they know of? What percentage of people ever listen to each station? What percentage have listened to each station in the last seven days? How much time do they spend listening to each radio station? Which programs on local ABC radio do they listen to regularly? What are their opinions of those programs? These are the basic issues for radio audience measurement: the type of research that focuses on understanding the audience. Because we also asked demographic questions (age group, gender, occupation, education level, ethnicity, etc) we could also discover how listening behaviour and preferences varied among different types of people. Though information collected by telephone is not as accurate as a self-completion diary (particularly in terms of time spent listening to each station), it was the only feasible way of getting audience information across such a huge area.
Political polls by telephone Telephone surveys can work well for political polls, during an election campaign, when public preferences are changing rapidly. The basic question is simple (usually along the lines of "Which party do you intend to vote for in the forthcoming election?�) but results have to be processed within a few days, and published without delay. The main problem with such surveys is response bias: ensuring that the sample accurately reflects the whole population. This is not easy when telephone penetration is low, because a phone survey will reach mainly the richer people, whose voting preferences are often very different from those of poorer people. If a database of previous surveys is available, this can be used for weighting the current results to produce a more accurate estimate.
Quick Guide to Audience Research
Using observation for research is like doing a survey without having respondents. Instead of asking questions of respondents, "questions� are asked of visible objects, usually in public places. There is still an interviewer (relabelled an observer), still a sampling unit (though not a person), and still a questionnaire (perhaps relabelled a unit record form). Questions in media research that can be resolved by observation include: When observing people in front of a TV set: how many are paying attention during advertisements? What percentage of news kiosks in each city is selling our magazine? How does the use of personal audio devices (such as Ipods and MP3 players) vary for different commuter train lines? There are many other examples. Observation is used far less than it could be; a little ingenuity can often produce very useful data.
Quick Guide to Audience Research
Questionnaire surveys
If your audience is highly literate, and happy to fill in printed questionnaires, this is an easy method. The questionnaire survey is the method most often tried by people who have never organized a survey before: it seems so easy. No interviewers are needed, and a well-educated population will complete questionnaires well, as long as the instructions are clear. But before you decide to use this method, consider its disadvantages: The questionnaire needs to be very thoroughly pre-tested, to ensure that the questions as understood by respondents are the same questions that you intended to ask. (You'd be surprised how people can misinterpret a question, if the wording's not crystal clear.) The response rate is often much lower than when interviewers are used. Without an interviewer present to persuade them, people think "Why should I bother to fill in this questionnaire?" – and a lot of them don't. When response rates are usually low, the findings are usually inaccurate, often overstating the incidence of what you're trying to measure. (People who don't use the service that the survey is covering tend not to respond.) The disadvantages are not apparent at first. They usually begin to be obvious after you have sent out questionnaires, and are waiting for hundreds of them to come back to you - but they don't! Alternatively, they do come back, but it's obvious that many of them are not filled in carefully or accurately. Questionnaire surveys can be distributed in four main ways: by mail, in print media, on the spot (either as visitor surveys or in workshops), and by internet.
Quick Guide to Audience Research
Mail surveys
An effective mail survey depends on having an accurate database of names and addresses. In most countries, it is no use doing a mail survey of the general public: response rates are so low that this becomes a very expensive survey method. However, when there is already a database of people who have a strong interest in the improvement of the organization doing the survey, a mail survey can be very effective. To ensure an adequate response, mail surveys normally use replypaid envelopes or freepost addresses. These need to be arranged with postal authorities, and envelopes need to be printed – all of which takes some time. Therefore, when you have prepared for a mail survey, you might as well do a number of them – and gradually learn how to improve them. As an example of a successful mail survey, my group has been organizing annual surveys of grain farmers in southern Australia. They deliver their harvests to bulk storage companies, who store and on-sell the grain on behalf of the farmers. After each harvest, a simple 4-page questionnaire is mailed to all the farmers, to measure their satisfaction with the grain collection process. Because the farmers are shareholders in the companies, their mailing addresses are already known. And because the findings are highly relevant to them, the response rate is very high. Though it takes more than a month for all farmers to send in their questionnaires, this is not a problem, because there's still plenty of time to improve the collection system before the next harvest.
In-publication questionnaires
For print media, it's tempting to include a questionnaire as part of the content. The idea is that readers can complete the questionnaire, cut it out of the publication, and mail it back. For several reasons, this approach doesn't usually work well: the response rate is often so low that it's not safe to draw any conclusions from the findings. If the names and address of readers are known, it's often preferable to do a mail survey, because at least with that it's possible to follow up nonrespondents.
Quick Guide to Audience Research
Instead of including a printed questionnaire in the publication, an alternative is to insert a loose questionnaire in a subset of copies – perhaps every 10th or every 50th copy, and perhaps using a random sample of distribution points. To get a usable response, it's normally necessary to offer an attractive incentive, such as the chance to win a prize. For example, in an insertion survey for a monthly magazine in Australia, the main prize was a trip to Europe for two people. The response rate was so good that it caused problems: we suspected that some people were returning multiple questionnaires, in order to win the prize. To avoid distorting the responses, later surveys for the same magazine offered more prizes (one per region), but each of lesser value.
Visitor surveys
Another way to deliver a self-completion questionnaire is by handing it out to people who are all present in the same place – either because they have spontaneously visited that place (e.g. visitors arriving at a museum or concert) or because they have been invited to an audience workshop. Surveys of visitors (or customers) work much like any other selfcompletion survey, except that they are completed there and then. Response rates can be very high, when the questionnaires are handed out and collected personally. It's tempting to simply leave a heap of questionnaires on a table, with a printed request to fill them in. But response rates in such cases are usually so low that no useful conclusions can be drawn. To have somebody hand a questionnaire to a visitor and request them to fill it in makes a huge difference. Successful visitor surveys make it easy for respondents to fill in questionnaires, by providing facilities: Writing surfaces (such as a table). Pens to fill in the questionnaire with (giving each respondent a free cheap pen). A "mailbox" in which respondents can put their completed questionnaires. The best position for a visitor survey is at the exit to a venue. Visitors will have seen the exhibition, concert, etc, and will at that point be in a good position to make comments.
Quick Guide to Audience Research
Audience workshops
Here's an example of an audience workshop that we ran in Tanzania a few years ago. This was done for a group of radio stations in the capital city, Dar es Salaam. Announcements were made on air on several stations, inviting people to ring a certain phone number if they were interested in attending a workshop. About 40 people telephoned, and were given details of the venue. We prepared a sample tape with about 30 brief extracts from radio programs – about half a minute per extract. The 40 participants sat down in a large room, and listened to this tape. After playing each extract, the tape was paused, and participants were asked to fill in their questionnaire. There were two questions for each extract, basically "How often do you listen to this sort of thing on radio?" and "What's your opinion of the extract you just heard?" Both questions used simple answer scales. At the end of the workshop (which ran for about an hour) we simply collected all the questionnaires, and analysed them as for a normal survey. It would have been difficult to do this research any other way, except by telephone – which would have taken much longer. Normally, you would hold several such workshops, at different times of day, to enable the widest range of people to attend. For this type of research, it's normally necessary to pay the respondents to attend. Otherwise, many of them fail to turn up.
Internet surveys
The main problems with internet surveys are related to sampling: ensuring that the sample is representative, and then ensuring a good response rate. There are also new problems, that don't apply with other types of surveys, such as ensuring that people don't respond twice by mistake, and dealing with the blatant lying that often occurs with online surveys. I recommend using online surveys only when you have a known sample – such as regular customers – almost all of whom have internet access and are comfortable with using the Web. For example, we did a very successful survey with teachers in South Australia. Teachers are always good at filling in forms, they all had
Quick Guide to Audience Research
internet access through their schools, and they knew the education authority that was sponsoring the survey. As for the mechanics of online surveys, this can be surprisingly easy. There are many web-based services that allow you to set up and manage online surveys – either free, or for a small charge per respondent. Among the best known services are: Zoomerang (www.zoomerang.com) – has a wide range of attractive graphical results. Sysurvey (www.sysurvey.com) - one of the most powerful sets of output options. Free Online Surveys (www.freeonlinesurveys.com) and Dubidu (www.dubidu.dk) – not many options, but very easy to use. These services are international: they can be used anywhere in the world. When choosing such a service (they come and go very fast, but the above three have been around for some years) make sure to check their privacy policies and conditions very carefully. Look for sponsorship or recommendation by a government or industry association: there are some rogue organizations that use this market research to develop a database for spamming.
Quick Guide to Audience Research
Qualitative methods
So far, this Guide has covered quantitative research, mainly surveys. You begin with a sample design and a questionnaire, and you end with a set of numbers, from which conclusions can be drawn. Qualitative research is very different. Sampling is less crucial, and there are no formal questionnaires. Conclusions are still drawn, but they are based on words, not numbers. The end-produce is usually a better understanding of the audience, not a measure of it. Because of its verbal emphasis, qualitative research often appeals to people without much mathematical knowledge. It seems easier, at first. But in fact, qualitative research is usually much more work than doing a survey – even though the sample sizes are much smaller. It's also more highly skilled work: qualitative research requires a lot of well-informed thinking, and you can't easily hire people to think for you. This section covers two of the simplest qualitative methods: in-depth interviewing and consensus groups. Other methods, such as focus groups and co-discovery conferences, require more training and experience.
In-depth interviewing
An interviewer questions a respondent. Though the situation is the same for in-depth interviewing and face-to-face survey interviewing, the two methods are quite different. With a face-to-face survey interview, the interviewer works through a planned questionnaire, and records most answers by ticking boxes or writing in short responses. The emphasis is on counting the number of people who give each answer. The interview needs to be trained in how to apply the questionnaire, but doesn't need to understand the background of the study. For in-depth interviewing, the interviewer must be a knowledgeable researcher. Instead of a fixed list of questions to be asked in a predetermined order, there's a short list of issues to be covered – seldom more than a page. An in-depth interview is more like a conversation. Instead of asking questions, the interviewer invites the respondent to speak on the issues of interest. Instead of writing
Quick Guide to Audience Research
down the answers, the interviewer usually records them on tape, and listens to the tape later, transcribing the most relevant sections. Sampling methods often focus on breadth rather than representativeness. The goal is to interview a wide range of stakeholders on the issue being studied. Often, the sample size is not decided in advance; it is typically much smaller than for a survey. 20 or 30 interviews is usually enough: beyond that number, few new insights are found. The main goal of in-depth interviewing is understanding the audience, not measuring it. The most thorough research projects often begin with in-depth interviewing, and the findings from those interviews are used to write a questionnaire. I recommend that for those inexperienced at in-depth interviewing, interviewers work in pairs. One interviews the respondent while the other takes notes and ensures the tape recorder is working. Halfway through the interview, roles are switched. After the interview, the two interviewers discuss it, and record that discussion at the end of the same tape. As an example of in-depth interviewing, Audience Dialogue interviewed groups of "information providers" and "information users" in Victoria, Australia. We were doing a feasibility study for a possible online community information database, and exploring the attitudes and likely behaviour of people who would either provide the information or use it. As the study progressed, we realized that there was also a third group of people: "information intermediaries" who provided community-related information to a wide range of others. If we had done a normal survey, with fixed questions, never changing in wording, we would not have discovered this third group, whose existence made the planned database much more feasible.
Consensus groups
You've probably heard of focus groups, but maybe you haven't heard of consensus groups. The two types of group are similar, but interpreting the findings from a focus group depends very much on the skill of the moderator. It takes a lot of experience to do focus groups well, and to interpret the results well. Consensus groups are safer, because the participants themselves provide the findings.
Quick Guide to Audience Research
To do a consensus group, you need four things: An organizing group, including a moderator and a secretary. A quiet room to hold the meeting, with blank walls where pieces of paper can be posted. An agenda: a list of topics you want to know about. Participants: about 10 of them are selected from the population being studied – such as all people living in the coverage area of your radio station. Consensus groups are done in sets of at least three groups. The reason for this is that any one group may not be typical of the population, and if you do two groups and the results are substantially different, you won't know which is correct. But with three groups, the pattern of results becomes much clearer.
Stages of a consensus group A consensus group normally takes about two hours. This time is divided into several stages... 1. Introductions (about 15 minutes). The moderator asks all participants (in turn) to introduce themselves, usually by describing their habits relating to the subject being studied. (For example, if the study is for a radio station, participants would describe their radio listening habits.) 2. Discussion (about 1 hour). The proposed agenda topics are written on a large sheet of paper, and posted on the wall. The moderator now works through the agenda, inviting participants to discuss each topic in turn - usually 5 to 10 minutes per topic. During the discussion, the secretary writes down statements that seem worth voting on, either because they are of interest to the study organizers, or because they seem to be widely agreed among the participants. So that the discussion can be an informed one, this phase often begins with the presentation of data to be discussed. For example, if the topic of study is a radio program, a recording of that program (or part of it) could be played to respondents. Then, when they discuss the program, they can refer back to examples they have just heard. When the moderator has run out of pre-organized topics, he or she invites participants to add more topics that seem relevant to the subject being studied.
Quick Guide to Audience Research
3. A short break, about 5 minutes. Participants usually have refreshments, while the secretary sticks the tentative statements on the wall. Each statement is written in large letters on a sheet of paper. 4. Voting (about 40 minutes). The moderator (or sometimes, the secretary) lead participants through each of the statements on the wall. Each statement is briefly discussed, and the wording is clarified. When everybody is clear exactly what a statement means, they vote on it. The secretary writes on the piece of paper the number of participants who agree - e.g. 7/10 if 7 of the 10 participants agree.
Participants The best number of participants is normally about 10. With less than about 6 people (unless it's a topic of great interest to participants e.g. childbirth experiences among new mothers) the groups don't function well. And with more than about 12, the groups take a lot longer, are more difficult to manage, and some people don't get to express their opinions.
Moderator and secretary The moderator's job is to chair the meeting: introducing the issues to be discussed, making sure that nobody dominates the conversation, and that everybody has their say. The moderator also needs to encourage participants to discuss the topics with each other, not only with the moderator. It's usually necessary for the moderator to say, early in the discussion, "Talk to each other, don't talk to me." The secretary listens carefully to the discussion, and writes down statements that many people seem to agree with. Later, the secretary records the voting results.
Agenda The types of issue best dealt with by consensus groups are opinions and background details. If you want to measure facts, and record answers in numerical form (or yes/no form), don't do consensus groups - do a survey instead. A good agenda will have about 10 broad topics to discuss - and participants usually add a few more. The agenda topics are not questions, but items to be discussed. The moderator should never ask a question which can be answered Yes or No. Instead of asking "Do you think the news on FM99 is any good?" he or she should say "Please discuss what you most like and dislike about the news on FM99."
Quick Guide to Audience Research
Output The output of a consensus group is a set of statements that most people in the group have agreed with. Usually there are 30 to 50 such statements per group. Several groups often come up with almost
statements from a set of six groups carried out for a weekly magazine in Croatia. This was done as a workshop situation: highselling and low-selling issues were given to group members, and they were asked to identify what they most liked and disliked about each of the two issues. Here is an example of statements related to
Buying publications depending on finances Group
Buying newspapers greatly depends on my current financial situation.
Amount of papers I buy depend on my financial situation.
I would be buying more print if my financial situation was better.
I find buying magazines a serious investment.
Where they buy publications Group
I usually buy papers in the morning, in the shop.
Most often, I buy my newspapers at a kiosk.
I buy print mostly at a kiosk.
I usually buy print at the kiosk.
The format of statements is similar to survey findings, but each statement is given more detail in the recorded comments that lead up to the statement. If the discussion is recorded, the report can include transcripts of the relevant conversions about each statement. From comparing the two issues, and the subsequent discussion, we were able to identify the main factors that led readers to buy some issues, but not others.
Quick Guide to Audience Research
Response cultivation
Some media organizations have a steady stream of inquiries and comments from their audiences. For example, newspapers receive letters to the editor, and many radio stations have talkback (phonein) programs, and musical requests. The principle of response cultivation is to cultivate these responses that are already occurring, encourage more of them, and try to draw some conclusions from them. The big problem with response cultivation is that you can't assume that unsolicited responses from some members of your audience is representative of the whole audience. In fact, some studies have shown that this is definitely not the case: people who spontaneously contact the organization tend to be the most frequent users of its service, and those who have used it for longest. Therefore two important principles of response cultivation are: Set up systems that encourage new and infrequent users to contact you. Try to reduce the effect of bias by comparing trends over a long period (the rationale: if regular users of your service are changing, probably the infrequent users are changing in much the same way). Response cultivation becomes research when you carry out some regular analysis of spontaneous contracts from your audience If the task becomes overwhelming, and you are snowed under with letters and phone calls, you can analyse only a sub-sample. For example, perhaps only using contacts that arrive on certain dates of each month. This is not a problem, as long as there is no reason why the material you choose to analyse may be different from the material you don't analyse. Even keeping a simple count of the number of items received each month can be useful as a way of monitoring audience trends. Though response cultivation need not be qualitative, I've classified it with this group to emphasize that its findings produce suggestions rather than accurate measures. Notice that response cultivation and impact assessment (Section 2.2 above) overlap: data from response cultivation can be used in impact assessment.
Quick Guide to Audience Research
7. 7.1
How to choose a method Whether to do a survey
Do a survey if most of these are true The concepts you want to ask about are very clear. You need exactly the same information from a large number of people. Your research team has a high level of organizing skills: coordinating interviewers, clerical work, etc. You have a computer with statistical or spreadsheet program (or, better still, a skilled researcher) to do the analysis.
Don't do a survey if most of these are true Nobody in the research team is competent at using a spreadsheet or statistical software. You need the results within a week. You're not sure of the exact wording of the questions to ask. You want the result of the research to be understanding rather than numbers. If a survey is appropriate, read on to choose the best delivery method. If a survey is not appropriate, consider using observation, impact assessment, response cultivation, or a qualitative method.
Choosing a survey method
Use personal interviews if most of these are true The survey covers only a small area, so that it takes interviewers no more than an hour or so to reach any part of the area. You have no previous experience at doing surveys. In areas where few people have a telephone (most developing countries) - or response rates on telephone surveys are very low (the USA). Respondents need to be shown something that cannot be described in words only (e.g. screen shots from a TV program). For long questionnaires - when interviews last for half an hour or more. There's a high chance of finding respondents at the planned place of interview.
Quick Guide to Audience Research
Use telephone interviews if most of these are true The survey will cover a large geographical area. Telephone call costs are cheaper than interviewers' travel costs. An interview takes no more than about 15 minutes. It is possible to get an accurate sample list (e.g. from an up-todate and complete telephone directory). You have a number of telephones you can conveniently use. If several of those conditions don't apply, telephone interviewing is probably not a good idea.
Use observation when these are true The behaviour or object you are studying occurs in a public or accessible location. Observation will produce more accurate findings than simply asking people. Observation is often done in conjunction with a normal survey. For example, in a face to face survey, the interviewer will observe the respondent's sex, rather than asking. And when respondents are being interviewed at home, a trained interviewer, though observation, is the best placed person to answer questions such as which short-wave bands are available on radios in the home.
Do a mail survey if most of these are true Your audience is highly educated (e.g. nearly all of them have completed secondary school). You have a complete and up-to-date list of names and addresses. You are in a country with a widespread, fast, and cheap postal system, that provides for business reply mail and/or freepost. You don't need the survey results for a few months (people are slow to respond to mail surveys). Your audience is very interested in the topic of the survey and conscientious at responding. You are highly skilled and experienced at writing questionnaires or you have the patience to test a number of versions of the mail questionnaire by doing it as a personal interview. (If you're a beginner, expect to go through at least 5 versions before the questions are completely clear to respondents.) A forbidding list, isn't it? You'd be wasting your time trying to do a mail survey, unless nearly all of those conditions apply.
Quick Guide to Audience Research
Do an in-publication survey if most of these are true You are in charge of a print medium, and want to survey your present readers. You have strong reason to believe that the response rate will be very high (e.g. by offering a powerful incentive to return the questionnaire, and removing all barriers that might dissuade response). The questionnaire inserted in the edition is intrinsically interesting to fill in.
Do an on-the-spot survey (visitor survey or workshop) if one of these is true You have a fixed venue (such as an art gallery) and want to survey your visitors. You need people to assemble in a group so that they can react to material that you will present to them in a theatre situation.
Do an internet survey if most of these are true Almost all of your audience have internet access (hardly anywhere in the world, yet - maybe only Sweden and South Korea, in 2005). You have the computer skills to set up an internet survey - or you can find somebody to do this for you, exactly as you want it. You thoroughly test the internet survey before making it available to respondents. Most of the time, it's not a good idea to do an internet survey among the general public. The main exception is when you're researching the audience of a specific website.
Use mixed methods when appropriate - for example... With personal interview surveys of business managers, first contact them by telephone to make appointments. When you want people to keep a diary of their activities, seek agreement first by telephone, then send questionnaires by mail. (Because some personal contact will increase the response rate - and a low response rate is the main problem with mail surveys.)
Quick Guide to Audience Research
Choosing a qualitative method
Do a set of consensus groups if most of these are true You need results within a week or two. Most potential participants live fairly close together. There is a suitable venue where they can meet. The budget can afford to pay about 30 people to come to meetings. You want quantifiable data as well as reasons for preferences.
Do in-depth interviewing if most of these are true The budget is very low, but the researchers have plenty of time. There are two people who understand the issues and will work well together on this project, as co-interviewers. Interviewers are able to travel to a venue of the participant's choosing. Interviewers have a high level of skill in conceptual thinking (e.g. university-level training in social sciences). There is no need to numerically extrapolate results to a population.
Use response cultivation if most of these are true A significant proportion of your audience spontaneously contacts you regularly. (How much is "significant"? - Perhaps more than 10% of the audience each year. If you don't get that already, encourage more feedback.) You have no reason to expect large fluctuations in your audience from one month to the next. You can't afford to conduct regular surveys. (Bear in mind that response cultivation is a second-best option.) You have at least one staff member who's interested in this type of work, and well-organized enough to do it accurately and regularly.
Quick Guide to Audience Research
Use your findings!
It's amazing how often media organizations do audience research is done, but don't use the findings. When a market research company does the study, at least it gets paid, but when the organization does its own study and ignores the results, that's a real waste of time and money. Making sure that the results are used is something you need to plan before the study begins. These factors help: Focus either on making a decision, or understanding the audience - but avoid trying to do both in the same study. For making a decision If you need to make a decision, and need information about the audience to make a better-informed decision, you should be able to work out exactly what you need to know, and use that information to make the decision. For this purpose it's usual to do a survey. Very few questions need to be asked: only those relevant to the decision. The sample needs to be just large enough that one alternative is clearly preferred to the others. For understanding the audience If there's no specific decision to be made, but you feel that you don't understand your audience well, a qualitative method, such as consensus groups or a co-discovery conference, is better. The type of data that never seems to be used is from a comprehensive survey. This happens when the managers decide "let's find out all we can about our audience." So they do a huge survey, asking 50 or 100 questions. When they get the results back, they realize the situation is much more complex than they had expected. And because there are so many questions, it usually takes months to prepare the report. By then, it's often too late. And such a report usually raises as many questions as it answers. The common conclusion is "more research is needed" and no action results. Involve everybody who matters in the initial stages. If they don't feel connected to the survey, they won't implement the results. Timing is important. Doing audience research (even if you know exactly what you're doing and have a lot of experience) usually
Quick Guide to Audience Research
takes at least two weeks. For beginners, two months is quick. If you really need the results next week, and you start now, there's no point in doing a survey. But don't pretend you need the results next week, if you really don't. Results need to be presented in a usable way. Long academic reports are not helpful, because they take weeks to write, and the results are out of date by the time the report is finished - and if the report is too long, most people won't bother to read it. But it's also not useful to produce reports that leave out so much detail that the results are meaningless. Usually 10 to 20 pages is a good length. More than that, and it won't be read. Less than that, and it won't be credible, or detailed enough to be useful.
That's all! You have now reached the end of this Quick Guide. If your purpose in reading it was to help commission research from a professional company, this may be enough. But if you are seriously interested in doing your own research, you'll need more detail. This extra detail could come from... Attending a course on audience research. These are rare, but Audience Dialogue is planning to run online courses on basic audience research. Persuading an expert to help you. These are rare too, but there may be a university near you with a school of journalism, media, or communications, that has a practically oriented trainer. Reading a book on audience research methods, and doing some practice. My book Know Your Audience can be read online at www.audiencedialogue.org/kya.html Good luck!
Quick Guide to Audience Research
Further reading This highly selective bibliography focuses on publications that are both relevant to prospective researchers and not difficult to get – available through major publishers, large libraries, or online.
Audience and media research Barwise, Patrick, and Andrew Ehrenberg. Television and its Audience. Sage, London, 1988. A study of the almost-predictable habits of television audiences. Berger, Arthur Asa. Media and Communication Research Methods. Sage, USA, 2000. Focuses more on studying media output than on audiences, so a useful complement to this book. Detailed coverage of qualitative methods, with a lively and attractive style of writing. Gunter, Barrie. Media Research Methods. Sage, London, 2000. Discusses the findings rather than the methods of research, the only one of these books to do so. List, Dennis. Know Your Audience: A Practical Guide to Media Research. Original Books, New Zealand, 2005. A more comprehensive treatment of the areas covered in this manual. An Indonesian translation of the previous edition has been published by UNESCO, Jakarta (2001), under the title Memahami Khalak Anda. Mytton, Graham. Handbook on Radio and Television Audience Research. UNICEF, Paris, 1999. Covers why to do and how to use audience research, rather than how to carry it out. Aimed at managers of media organizations. Radio WKSU. 2001. Listener Survey Toolkit. Produced for public radio in the USA. Short and succinct. Available online (February 2006) at www.wksu.org/toolkit Webster, James G, and others. Ratings Analysis. Lawrence Erlbaum, USA, 2000. Focuses on commercial broadcasting in the US; of particular use to media planners. Wimmer, Roger D, and Joseph R Dominick. Mass Media Research: An Introduction (6th edition). Wadsworth, USA, 2000. Perhaps the clearest of the standard textbooks on media research, though with a strong bias towards the US context.
Research methods in general Of the hundreds of books on research methods (other than audience research) these few have been selected as being exceptionally clear and useful. Bernard, H Russell. Research Methods in Anthropology. Sage, USA, 1994. Covers a wide variety of quantitative and qualitative research methods, and very clearly written. Aimed at anthropologists, but most of the methods described are useful in media research.
Quick Guide to Audience Research
Crouch, Sunny. Market Research for Managers. ButterworthHeinemann, London, 1996. An English book, focusing on market research rather than audience research, but very clearly written, particularly the chapters on using research findings. Dillman, Donald A. Mail and Internet Surveys: The Tailored Design Method. Wiley, New York, 1999. One of the leading experts in mail surveys turned his attention to internet surveys, and produced this authoritative book. Slightly outdated now, given the pace of change on the Net, but a new edition is on the way. Patton, Michael Quinn. Qualitative Research and Evaluation Methods. Sage, USA, 2001. The previous (2nd) edition), entitled Qualitative Evaluation and Research Methods (1990) is equally useful, though less comprehensive. Wadsworth, Yoland. Do-it-Yourself Social Research. Allen and Unwin, Australia, 1997. A concise overview of many research methods.
Development communications Laws, Sophie. Research for Development. Sage, UK, 2003. A useful handbook for researchers in international development – not focusing directly on communications, but still providing useful advice in this area. Mikkelsen, Britha. Methods for Development Work and Research (2nd edition). Sage, India, 2005. Focuses on participatory research methods in developing countries. Not specifically about media research, but methods are usable in that area. Servaes, Jan. Communication for Development: One World, Multiple Cultures. Hampton Press, New Jersey, 1999. A more philosophical book than Mikkelsen's, focusing more on approaches and perspectives than on how research is done.
Evaluation Logic Model Development Guide. W K Kellogg Foundation, USA, 2004. Online at www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf (72 pages). Program logic modelling is a powerful tool for evaluating how a program (in the broadest sense) produces results, and this is one the clearest guides to logic modelling. Patton, Michael Quinn. Utilization-focused Evaluation (3rd edition). Sage, USA, 1996. Perhaps the most comprehensive book available on how to design evaluations so that their findings can be used. Wadsworth, Yoland. Everyday Evaluation on the Run. Allen and Unwin, Australia, 1997. A concise and clear guide to evaluation thinking, which helps frame research in the context of its use.
Quick Guide to Audience Research
Appendix 1: Example of questionnaire used in rapid quota survey Q1. Which of these media do you use at least once a week... (Tick all that apply) [] TV? [] Radio? [] Newspapers and magazines? If answer did not include Radio, please skip ahead to *Q6. Q2. Which radio stations do you listen to at least once a week? ...Any others? (Tick all that apply) [] Aditya [] Graha [] RRI Pro 2 [] Bharabas [] Gress [] RRI Pro 3 [] Chyntia Rama (CBS) [] Soreram [] Warna [] Cendana [] RRI Pro 1 [] Other . . . . . . . . . . . . . . . Q3. Which one radio station do you listen to most often? (Circle station name above) Q4. In which hours do you normally listen to radio? (Tick all that apply)
Q5. What types of program do you like to listen to on RRI? (Tick all that apply) [] News [] Children's [] Education [] Information [] Music [] Contests, etc [] Entertainment [] Sport [] Telephone contact [] Interactive dialogue [] Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . * Q6. Do you have a radio at home? (Tick one only) [] Yes [] No Q7. Do you have a TV at home? (Tick one only) [] Yes [] No Q8 Sex of this respondent (Tick one only) [] Male [] Female Q9. Which age group are you in? (Tick one only) [] Under 25? [] 25 to 34? [] 35 to 44?
[] 45 or over?
Q10. What is your highest education level? (Tick one only) [] No school [] Primary only [] Junior secondary (SLTP) [] University [] Upper secondary (SLTA) [] Akademi / D3 Q11. Finally, are there any comments you'd like to make about radio in Pekanbaru? ............................................................................................................... ............................................................................................................... ............................................................................................................... Thank you very much - that's the end of the survey.
Quick Guide to Audience Research
Appendix 2: Glossary of audience research terms This glossary includes the most common terms used in audience research. There are also many less common terms, which can be found in the glossaries on the Audience Dialogue website, at www.audiencedialogue.org/gloss.html Analysis: Processing the results of a survey: the activity that comes after gathering the data, but before presenting a report. Analysis involves making conclusions about the findings. Moderator: the person who facilitates or manages a group discussion. Multiple-response question: A question that offers a limited set of answers, e.g. "What is your gender: male or female?" See also open-ended question. Open-ended question: A question that does not suggest an answer, e.g. "What is your occupation?" See also multiple-response question. Participant: a person who takes part in a group discussion – a more active role than a respondent. Projection: multiplying data from a sample to make an estimate about the population that the sample came from. E.g. if a survey interviewed 1 person in each 1000 in the population, a count of 20 people in the survey is projected to 20,000 in the population. Population: everybody who is eligible to take part in a research study. See also sample. Qualitative: research that uses words as data, rather than numbers. See also quantitative. Quantitative: research that uses numbers as data – normally from surveys. See also qualitative. Questionnaire: A list of questions and answers, used in a survey. A common error is to call a questionnaire a survey, e.g. "50 people filled in the surveys" – but they didn't fill in surveys, they filled in questionnaires. Quota: the number of people of each type (e.g. male or female) to be interviewed in a survey. Respondent: a person who responds to a request to take part in a survey. See also participant. Sample: the people who participate in research study. See also population. Survey: A type of research study that uses questionnaires, mostly with multipleresponse questions, to gather data from a sample, and project it to the population.
Quick Guide to Audience Research
Dennis List is principal of the communications evaluation consultancy Audience Dialogue, as well as being a part-time Research Fellow at the School of Management at the University of South Australia, where he recently completed a PhD. He has been working in media research and associated areas for more than 20 years, with the New Zealand and Australian Broadcasting Corporations, and has done consulting work for international agencies including the Swedish Media Development Office, the Media Development Loan Fund, and several other high-profile international organizations. He has published several books on media research methods, including Know Your Audience: A Practical Guide to Media Research (2002, 2005) and Participative Marketing for Local Radio (2003), both available from this publisher. The text of most chapters of those books is available on the Audience Dialogue website, www.audiencedialogue.org
Quick Guide to Audience Research
An overview of simpler research methods and which ones to use for audience research
Published on Jan 9, 2010
An overview of simpler research methods and which ones to use for audience research