How to make a better political quiz?

[Anders Hallin]

Since everyone always complains about political quizzes, I'd like to get some discussion going on how to actually make a decent political quiz. How many axes should there be? Is it possible to make one that has validity and meaning globally, or are we constrained to relatively homogeneous regions, like Europe (or maybe even country-level), and US?

The normal axes are social and economic, are these valid?

The first thing that comes to mind when I think of political scales are those based on the cleavage theory of political science, the church/state, centre/periphery, land/industry and capitalist/worker (though there is some debate on which ones exist and are relevant, you can certainly add urban/rural and the later "post-modern" (environmentalism and other values) cleavage), that gave rise to the party system of today in Europe, but the issues of today and how those relates to the cleavages is not always so obvious, and may well have played out very differently in every country.

[Robert Sharp]

I actually don't have a problem with the axes. To me the problem is that you need MORE questions than these things give. They only hit on the really hot button issues, and usually with only one question. Quick example:

A quiz might say: Do you believe Americans have the right to own whatever arms they wish?

My answer would be no. However, I do believe in the Second Amendment overall. I just think there are limits based on what counts as personal defense versus what counts as an assault weapon. Simply allowing me to slightly disagree instead of greatly disagree insufficiently captures my position.

[Lurb]

Maybe some kind of weighted graph thing, no axes. Axes are bad because more than 2 is confusing and 2 are not enough.
So you make Hitler/Gandhi/Blah Blah Blah nodes, map the test responses to them and then get the user node and place it using his answers to connect him to the archetype nodes, making the links stronger the more you agree with the guy. Things would arrange themselves in a pretty map.

And of course most of them are US-centric. Guns and universal healthcare aren't really in the agenda here, so those would be mostly meaningless.

[bago]

Exactly, each participant should have as many axis as they want to define endpoints for and place themselves accordingly. A single issue voter might have one axis, while a more nuanced voter might have 17 axis to make judgments upon. Throw in a Bayesian weighting of the information used to determine the axis and the positions, and you are talking some seriously hardcore shit.

[lesslucid]

No political quiz will ever be perfect. I would just try to make one that's interesting and different. For example, you could have an "Individualist' vs "Communitarian" axis (ie, to what extent does the person believe they have a moral duty toward other members of the society? None = anarchist, a little (in terms of property rights) = libertarian, substantial = social democrat, total = socialists and christians) and a "Traditional Values" vs "Enlightenment Values" axis (ie, do your values come from the pre-Enlightenment heritage of Christian Europe, or from the tradition of Voltaire and Rousseau and Russell... and if so, how far out on that limb are you?).

[Tim James]

I always thought it was cute that the World's Smallest Political Quiz tended to make people more libertarian than they actually are. First thing is to get the bias out of it, I guess.

I actually don't have a problem with the axes. To me the problem is that you need MORE questions than these things give. They only hit on the really hot button issues, and usually with only one question. Quick example:

A quiz might say: Do you believe Americans have the right to own whatever arms they wish?

My answer would be no. However, I do believe in the Second Amendment overall. I just think there are limits based on what counts as personal defense versus what counts as an assault weapon. Simply allowing me to slightly disagree instead of greatly disagree insufficiently captures my position.

Seems like yes and no questions fail too, even when there is a range of agree/disagree answers. I may say "no" to the question above as well, but Robert's use of the buzzword "assault weapon" indicates that we are probably radically different in our knowledge of Second Amendment rights and support for firearms.

In a similar way, you have to deal with unknown unknowns like a person believing they support the Second Amendment and then getting the practical modern legislation all wrong. (I'm sure there's a fancy term for this.)

Aren't people able to label themselves fairly well? Why have a quiz at all?

[Unicorn McGriddle]

Robert's use of the buzzword "assault weapon" indicates that we are probably radically different in our knowledge of Second Amendment rights and support for firearms.

"Assault weapons" are, for practical purposes, a set of weapons/modifications/accessories that are banned. There is some correspondence with weapons that are more dangerous or have less legitimacy as tools for something other than homicide, but the correspondence is not total. However, there is still a category of banned weapons called "assault weapons." What Robert is saying is that some weapons should be permitted and others should be banned, which is everyone's position.

[Tim James]

True, I should have given him the benefit of the doubt. Assault weapon was derived from assault rifle (which actually has a clear definition), but today it's usually employed as a bad word so I reacted to it.

That just brings up another problem with political quizzes -- ill-defined modern language.

[Phil_Stein]

The two conventional axes (social and economic) are fine with me.

You could construct a quiz that was very general, and worked on a world scale, but I think it would be measuring somewhat different things. For example, the rights enshrined in the US Constitution and Bill of Rights have special meaning, BECAUSE they are so enshrined. I may not agree with every aspect of them, but I like the concept as a whole, and therefore, accept the bad with the good. But for a Frenchmen, or a Peruvian, the US Constitution doesn't hold much personal meaning. So, bottom line, I think a good political quiz for Americans needs to be US specific.

I think the main problem with the ones I've seen/take is the questions. Generally there aren't enough questions, and the ones that are asked are often poorly constructed. I may say yes (no) in a very literal sense to a question, even though I think I am against (in favor of) the underlying political concept they're going after. This is especially true when questions are asked in an absolute sense or with absolute words (always, never, and so on). Even though I have a lot of core political beliefs, many/most of them are not absolute, and adding an absolute word to a question's phrasing can force me, if I'm trying to be accurate, to give an answer that I think the quiz software will evaluate to project a different political stance on me than I actually have.

[Phil_Stein]

Here's an example of a quiz question I didn't like, from one of the quizzes linked in the other thread.

"If economic globalisation is inevitable, it should primarily serve humanity rather than the interests of trans-national corporations."

I said strongly agree. Hey, I'd like all of our laws and economic policies to benefit humanity.

But I would assume the quiz makers designed the question as whether you're pro/anti corporations and/or pro/anti globalization/trade. (That's a lot of slashes - have fun parsing that sentence).

I am actually a strong proponent of trade, globalization, and corporations. But I think that's because they DO benefit humanity on the whole. Basically, I think the quiz forced me to make a false choice, and got a bad read on my political beliefs because of a poorly worded question. A better phrasing might have been something like.

"Increased economic globalisation is a good thing for humanity." (strongly agree/agree/neutral/disagree/strongly disagree)

For that matter, the quiz did not have a neutral response option (I added that in my example). There are some issues I don't care about or don't have much of an opinion about.

[ravenight]

Right, I think the axes are reasonable in some sense, but the reasoning behind a response is more important than the actual response, so questions that assume you will agree because of A and disagree because of B tend to be bad questions unless A and B are really the only possible viewpoints.

In America, though, I think the main debate is about moral relativism. America liberals tend to have a Rawlsian view of the world: Government exists to create the system where you'd be happiest taking your chance being born as a randomly selected person. It turns out that we'd rather sacrifice some probability of being poor for some probability of being rich, as long as being poor isn't too horrible. Whereas the American right sees it as the government's job to punish people for immorality (since there is an absolute morality, it is easy to legislate), but not prevent or hinder them from engaging in it beforehand. Obviously most people fall somewhere in between, with their positions influenced by the practical realities of specific questions.

The problem with putting these philosophies on a spectrum is that they aren't actually opposites. You can believe in a Rawlsian system and still want the government to punish immorality, and you can believe that your morals should not be imposed on others, but still want to twist the system to benefit you specifically.

[roboczar]

A lot of political quizzes poison the well in their questions. These things aren't put together by sociologists after all. Joe Bob Webmaster wants people to come to his website and generate ad hits by getting passed around on forums or social networking sites. How to do that? Make a quiz that gets people worked up and plays to their prejudices.

A simple understanding of correct survey methodology and how to avoid bias will make any political quiz better than 99.9% of what's out there.

[Robert Sharp]

"Assault weapons" are, for practical purposes, a set of weapons/modifications/accessories that are banned. There is some correspondence with weapons that are more dangerous or have less legitimacy as tools for something other than homicide, but the correspondence is not total. However, there is still a category of banned weapons called "assault weapons." What Robert is saying is that some weapons should be permitted and others should be banned, which is everyone's position.

Right, and it shows that these questions are just poorly worded and need some shadings. Tim and I MIGHT disagree about the Second Amendment (I wouldn't call it a knowledge vs. ignorance issue though, but rather interpretation) but the question can't capture the nuances of that disagreement by using yes or no, or even degrees of yes or no.

Sadly, what I said would not be everyone's position, but close enough.

[Damien Neil]

In America, though, I think the main debate is about moral relativism. America liberals tend to have a Rawlsian view of the world: Government exists to create the system where you'd be happiest taking your chance being born as a randomly selected person. It turns out that we'd rather sacrifice some probability of being poor for some probability of being rich, as long as being poor isn't too horrible. Whereas the American right sees it as the government's job to punish people for immorality (since there is an absolute morality, it is easy to legislate), but not prevent or hinder them from engaging in it beforehand. Obviously most people fall somewhere in between, with their positions influenced by the practical realities of specific questions.

This doesn't make any sense to me at all, as a characterization of either the American left or the right.

I think you're talking about the left's traditional support for social programs intended to aid the poor, but I don't see where "moral relativism" gets involved in it.

And the right most assuredly does want to prevent or hinder people from engaging in immoral behavior. The people who want to criminalize abortion, for example, are interested in preventing people from getting abortions.

Personally, I think trying to jam either the left or right wings into a simple category is doomed to failure, since both are characterized by an uneasy alliance of disparate interests. Opponents of homosexuality and free marketers are both "right wing", despite having no common origin. Environmentalists and opponents of firearms are both "left wing", again despite having no real common origin. Politics is a complex stew, and the "political spectrum" is a description of tactical alliances of convenience, not political belief.

[roboczar]

Moral relativism is a watchword and talking point, mostly. It has an actual definition and meaning, but it's mostly used as a weapon outside its original connotations.

[Dave Markell]

I actually don't have a problem with the axes. To me the problem is that you need MORE questions than these things give.

Agreed, and they should be very specific questions, too. The example you gave, Robert (should people be allowed to own any arms they wish?) is a perfect illustration why overly broad questions skew the results. Practically everyone doesn't want the neighbors to be packing attack helicopters, tanks, and ICBM's, but answering "no" to that question would group supporters of gun ownership (but not nuke ownership) with those who would ban handguns.

[antlers]

Social and Economic axes of state control are totally inadequate. The concepts are not orthogonal, nor are they comprehensive, nor are there clearly defined points along the spectrum.

[Qenan]

I agree with Robert Sharp; the main problem is that there aren't enough questions and the answers available often aren't nuanced enough. An example for me would be gun control. Do I think Americans "should" be able to own whatever guns they want? No. Do I think we should do anything about it? No. There's not enough political consensus and it isn't a high enough priority with me. But I'm emphatically not an NRA member, either.

[Jojo]

Another issue I have with many political quizzes, is that they assume a certain level of experience/understanding of politics to begin with.

Perhaps I'm going to out myself as a bit of a dumbass here, but anyway I will go ahead and take some examples from the political compass quiz:
"Controlling inflation is more important than controlling unemployment." I just don't know enough about economics to feel like I have an educated answer to that. I know what inflation and unemployment are, I know that both are bad (and inevitable), but I don't understand how they interact with each other enough to do anything other than choose an answer at random.

OK, I can't find the other question that I remember being confused by, but suffice to say that I'm not really interested in, or educated about economics, so those questions get a pretty unreliable answer from me. If someone can click on a question for more information about it, couldn't it theoretically serve a double purpose - to explain some of the terms included, which in turn might reveal any inherent bias from the test creator, or encourage them to provide links to multiple opinions/explanations.