Ricky Buchanan ([info]rickybuchanan) wrote,
@ 2008-09-26 15:11:00
Previous Entry  Add to memories!  Tell a Friend!  Next Entry
What do you know about survey design
Ok, this comes under "my flist knows everything"... so hello, very smart flist people!

Who knows about designing research questionnaires for people to answer? I know there's ways to do it so you'll get the biggest amount of valid information using the least questions possible. People don't like complicated or long questionnaires, so this is best!

Why? I'm designing some qualitative research about the stuff I wrote about in the letter here - the problem of people who can't physically attend events because of chronic illness/disability related problems. And I want the research to be an un-crappy as possible.

I already know it's going to be a "straw poll" type thing so it won't be numerically valid in terms of how many people have these problems, well except for the extreme lower bound being the number of people who answer "yes" to the questionnaire! But the goal is to get an idea of the range/types of issues that prevent people from attending events in person, the range/types of possible solutions people have used to this problem, and the range/types of solutions people would be interested in using if they were offered. Plus, possibly another questionnaire aimed at event organizers/facilitators which asks the inverse questions - are they aware of the issue abstractly, do they know if anybody has been excluded from events they've run because of this, what things do they think prevent people from attending, what solutions would the be most willing to offer, and so forth. The second questionnaire is the icing on the cake though, the first one is the most important.

Even if you know the name of the field of study relevant to designing good questionnaires of this type, that would be a help! Additional bonus points for offering good URLs and top marks if to anybody who has knowledge to share and could actually chat/email with me while I design the thing.

I really want this to be good. It's a topic that matters.

Thanks!
Ricky

(Post a new comment)


[info]claudine_c
2008-09-26 08:32 am UTC (link)
I don't know whether there is a field of study specifically about questionnaire design, though maybe there should be -- I have seen many baaaad questionnaires in my time.

In my previous job I helped analyse the data that came out of these questionnaires, so while I don't have much grounding in the theoretical principles behind them, I do have a sense for what does and doesn't work. Do you have any questions to start with? Unless this is something you want to get done in a week, I'd be happy to look over drafts and make suggestions.

*hugs*
Claudine

(Reply to this) (Thread)

[info]rickybuchanan
2008-09-26 09:50 am UTC (link)
Thanks! I'll fire a draft at you as soon as I have one :)

r

(Reply to this) (Parent)

[info]meepettemu
2008-09-26 09:14 am UTC (link)
Ok, if you're writing a questionnaire to get qualitative responses, it doesn't need to be numerically valid. You just need to be aware (as you are) that you're pointing it towards a specific audience.

What you're looking for, i think, are psychology journals. specifically, there's http://scholar.google.co.uk/ and sticking search terms in will give you journals.

as for journals, many of them require a uni access user/password, but some have free papers up

http://www.informaworld.com/smpp/title~content=g795399049~db=all is the first journal i found.

But, honestly, what you're talking about is qualitative psychology. And what tends to happen is that people design their own questionnaires, to get views on things that need covering - so you ask all the questions that you think need asking, making sure that there's a "is there anything else you'd like to add that i haven't covered" question.

I'm in the middle of having some papers published about women with PCOS. Like your idea, the original tohught behind the PCOS surveys was novel, and the questionnaire that was designed was simply designd to cover as many areas as the interviewer (who wasn't me) could think of.

Once those answers come back, they can then be analysed in terms of themes etc.

Once you have those kinds of responses, it's then more possible to start creating standardised questionnaires.

Does that help at all?

(Reply to this) (Thread)

[info]rickybuchanan
2008-09-26 09:55 am UTC (link)
Yes it helps lots, thank you. You're right, qualitative psychology is exactly what I was failing to remember the name for. Yay for smart flists!

And the suggestions helped. I actually asked one of my brothers (the one who's doing his grad cert is psych currently) and he said almost the same as you, which is nice and reassuring :)

One thing I'm sorta stuck on is that I've brainstormed with a couple of people on possible reasons why people might not be able to go to stuff and we have a list of about 10 reasons but I'm totally sure it's not all the reasons there are because I'm only human!

So one part is I want to ask if the person has missed out on events, and then if yes has it been because of a, b, c, d, e, f, g, h or "other", but I know if there's a non-negligable number of people whose answer falls within "other" that'll end up under reported because by asking specifically about reasons a through h I'm bringing those to mind and people may not recall their other reasons at the time, or may not be bothered to explain them because they're complicated. Is there a better way to structure things to bias this less?? I suspect if I don't use *any* prompts then I'll end up with *everything* under-reported which is worse!

Thanks again!
r

(Reply to this) (Parent)(Thread)

[info]meepettemu
2008-09-26 10:16 am UTC (link)
:o) you're welcome!

I would suggest that it might be worth your time picking a couple of people you know and trust, who are able to spend a while writing, and send out your survey. It's known as a pilot study, and this would enable you to find out roughly how many people are going to fill things under "other" and what that "other" might be. this is often exactly what's done in quali psych, to make sure that the interviewer has got, as you say, the right "feel" for the questions.

:o)

(Reply to this) (Parent)(Thread)

[info]rickybuchanan
2008-09-26 01:56 pm UTC (link)
Bwahaha! Pilot survey! That sounds like a VERY cool and helpful idea.

I shall think upon it.

I miss academia sometimes *wry grin*

r

(Reply to this) (Parent)

[info]claudine_c
2008-09-26 11:48 am UTC (link)
If you don't offer any options for a question, but just offer a huge chunk of space for people to give their own reasons, some people may not remember everything, and prompts *may* help; on the other hand, some people may not need much prompting and may write *a lot* (whether you provide space or not!).

A fully qualitative approach like this is more difficult to analyse statistically -- but if numbers are small, this is not such a huge issue.

(Reply to this) (Parent)(Thread)

[info]rickybuchanan
2008-09-26 01:55 pm UTC (link)
My gut feeling is that given all the subjects will be homebound, it's probably that the vast majority will have major constraints on their energy, ability to use a computer, health, or other stuff and therefore will be less likely to be willing-and-able to write much in blank boxes.

I know that I, for example, am 10x more likely to complete a questionnaire where I tick boxes or similar than one with open-ended questions. I very much don't want to exclude those with the most severe limitations on energy/ability.

I guess that leaves me with the option of prompting AND leaving a chunk of "do you have anything else to add..." space for each area of questions.

Also, I don't need to analyse it statistically in an academically rigorous fashion - at least at this point. Nor do I have the ability to do that. I just need to be able to make sense of it, I guess, so that I can find out what to do next...

These questions/suggestions from everybody are being very helpful in making me think about exactly what I'm trying to accomplish - it's cool!

r

(Reply to this) (Parent)

[info]paula_angela
2008-09-26 01:29 pm UTC (link)
At first I was picturing this as "focus group" research, but that implies that you get respondents in the same room responding and discussing together. You are definitely talking about a questionnaire and not a survey here. A lot of the early research on a topic is qualitative, and indeed interviews are commonly used, as they allow the researcher to discuss questions, ask follow-up questions, and discover new things.

This early qualitative stage *may* be followed up by the researcher developing a model, and using surveys to numerically test their model(s).

(Reply to this) (Thread)

[info]rickybuchanan
2008-09-26 01:50 pm UTC (link)
Unfortunately it's sorta inherently impossible to get together a focus group or even interviews when both researcher and subjects are homebound! Ironic, really...

So yes, I'm stuck with questionnaires with fixed questions rather than interviews where you can change the questions depending on previous answers.

r

(Reply to this) (Parent)(Thread)

[info]azurelunatic
2008-09-26 05:14 pm UTC (link)
Is this to be paper, or online? One can dynamically program survey paths online and ask more in-depth questions that are thought out beforehand depending on the respondent's answer to screener questions.

You can also ask if they would be willing to be contacted for a follow-up, in case you think of more things to ask.

(Reply to this) (Parent)

[info]squishsplash
2008-09-27 09:05 am UTC (link)
Also, no need to get hung up on a particular model of how a questionnaire should work.

Remember that what you are doing is qualitative. If you like you can think of your research as a "modified interview" methodology. If you think of it as modified-questionnaire or modified-survey, you may find yourself following rules that are unnecessary. Qualitative interviews can specifically tap into the interviewer's intuition.

So emailing a discussion back and forth is all fine! Plenty of qualitative interview studies have several rounds of interviews with the same subjects as the interviewer's knowledge increases. They don't always make this explicit in their write up - but when you read closely enough you can see it.

And when you are ready to make up your "model" you still have the option of writing up a survey, and sending this back to your same group.


This all sounds like a *lot* but it doesn't have to be. If you are comfortable with one round of questionnaire answers - GREAT! If you want to delve deeper in parts, that's good too. In the end, the quality of the research has more to do with the researcher's insight than on any single tool.

Like with anything in academia, you have quite a lot of latitude as long as you don't pretend to be doing something you are not. And qualitative-modified-interview methodology gives you a lot of leeway.

Frankly, you *could* use a standard internet bulletin-board, and explain in the write up that it was a modified-focus-group methodology, developed to access a group who cannot come together in any other way. I think you'd get academic kudos for this. Just be explicit that you are doing it.




(Reply to this) (Parent)(Thread)

[info]squishsplash
2008-09-27 09:08 am UTC (link)
Oops - that was me!!! [info]paula_angela. Just forgot to change IDs. Sorry!!!!

(Reply to this) (Parent)

[info]azurelunatic
2008-09-26 05:12 pm UTC (link)
I have a few years of administrating telephone surveys and giving feedback on how well the respondents reacted to them under my belt; I'd be willing to give a draft a look over to see if there's anything I see that's skewed, missing, bad and wrong, fuzzy, et cetera.

(Reply to this)


[info]ltlpengy
2008-09-26 10:02 pm UTC (link)
I' glad the Flist came through for you. I scratched my head and thought I know the answer, but recalling it is another question.
Good luck with getting your head around the statistics (urg)

(Reply to this)


[info]griffen
2008-10-05 04:30 pm UTC (link)
Sent here by [info]pauamma.

Get thee to a used bookstore that sells textbooks and pick up the most recent edition you can find of Earl Babbie's "The Practice of Social Research." That's one of the sociological bibles for designing good survey research.

You might want to get in contact with researchers in the sociology department of your local college or university, too - your results could have a much greater impact if you could do this under their wing (plus, the institutional review board would ensure that it wasn't considered unethical research on human subjects).

(Reply to this)

Create an Account
Forgot your login or password?
Login w/ OpenID
English • Español • Deutsch • Русский…