If any of you are planning to do a survey, I highly recommend you check out a series of six papers (six in all) from the ACM SIGSOFT Software Engineering Notes:
- Principles of Survey Research: Turning Lemons into Lemonade
- Principles of Survey Research: Designing a Survey
- Principles of Survey Research: Constructing a Survey Instrument
- Principles of Survey Research: Questionnaire Evaluation
- Principles of Survey Research: Populations and Samples
- Principles of Survey Research: Data Analysis
I also find some
class slides on the papers. I wish I had read this series before we started working on the survey--it would've saved us a load of headaches.
Let's talk about something specific:
Likert Scales. For those of you who are not familiar with them, here's an example:
Indicate to what degree you agree or disagree with the below statement :
1.) I prefer using Linux to Windows.
|Strongly Disagree|Disagree|Neither Agree nor Disagree|Agree|Strongly Agree|
Kitchenham and Pfleegar in part 6 of the series call answers to the above questions
ordinal data. Can we treat orindal data as
nominal data? In other words, can I simply convert the above (e.g., strongly disagree = 0, strongly agree = 0) example into a number scale and perform the standard
ANOVA? ANOVA is a staple of HCI research. It will tell you if you can, with confidence, state that differences between two means are a result of some treatment and not just random effects.
From the paper: "In general, if our data are single peaked and approximately Normal, our risks of misanalysis are low if we convert to numerical values."
So, I guess using F-test + post-hoc tests like
Bonferonni are OK to use if the results seem to have some sort of bell shape. What do you do if you don't have such a shape? You can try converting values (multiple/divide by a factor, take the log, etc.) or if the shape is bimodal, trying to split the data further. Any other tips people have? In fact, I haven't really found a paper or book that will tell you--1) first run this test for normality, 2) if it meets this threshold, then you can safely use the F-test. Any pointers? I'm wondering, is a Chi-squared test appropriate for Likert scales as well?
-*-
Some minor corrections to our blog were pointed out by
moondial [japanese only]. We were inconsistenly spelling the word blog in Japanese. The Japanese language allows one to explictly specify a
glottal stop (a sort of hesitation, or pause)...sometimes its hard for me to correctly find whether an English word should have a glottal stop at a certain place when it is translated to Japanese. burogu or burrogu (the double r indicates the glottal stop)? Correct answer: burogu