The value of experts

[Phil_Stein]

I was thinking about the role of experts in summarizing complex academic and/or other information for public consumption.

Here's my theory:

Experts, on the average, usually know much more about their topic than I do.

But, the degree to which I trust an expert's opinion should depend on how I 'found' that expert.

If there are ~100 experts in a given field, and I choose one more or less at random and read what he/she has to say, the odds are good that I'm going to get a good summary of expert understanding in the field in question, and garner a better understanding than I had before.

But, if I rely on others to find experts for me, and/or summarize an expert's opinion, there's a decent chance that the intermediary will have chosen an expert near the extremes of expert opinion. The outlier expert may be right, of course, but the fact that perhaps 95 out of 100 other experts disagree with him/her means (IMO) there's also a high chance they're wrong, and that if I rely on that expert's opinion instead of my own preconceptions, my understanding of the issue at hand may be worse. In particular, sources advocating a viewpoint may cite experts at extreme ends of the expert opinion bell curve.

Yes, this may be a blindingly obvious conclusion to many of you, and it's not entirely novel to me, but I think it's the reason why lots of folks shrug off expert opinions. I accept that experts, on average, probably know more than me about a given subject and perhaps have a more accurate assessment of what to do going forward, but the specific expert that any particular source quotes is likely to have an outlier assessment of things that is worse than my own gut instinct. This isn't always the case, of course, but it's true often enough to substantially reduce the value of expert opinion in persuading me of a particular argument.

If I'm interested enough in the issue in question, I can seek out a variety of expert opinions on my own, and try to sort of average them to try to find the middle ground. But that's time consuming, so I don't do it all that often.

You can apply the above theory to media (and advocate) presentations on topics like:

Global warming
Various economic proposals
Novel medical claims
Analysis of what's really going in in Iraq/Afghanistan/the Middle East in general

Thoughts?

[Fugitive]

That kind of phenomenon shows up a lot among things like conspiracy theories too, where the "truth-seekers" can trot out their own experts, but it's highly likely that their expert is that 1-in-100 guy who got his degree at NotAMill U and also believes aliens are secretly replacing his Buick every night.

[Sidd_Budd]

Seems logical to me. I can think of one additional case in which people dismiss experts -- when they have a strongly held belief regarding the issue. Using your model, these folks already come from an extreme viewpoint. If they were to pick a random expert that represents mainstream expert consensus (and is most likely to be correct, according to you), they would find the opinion different from their own, and their strong pre-existing biases would make them more likely to dismiss the claim in order to reduce the challenge to their belief system.

So people who simply want to learn more about global warming would distrust preselected experts because of the issues you brought out. People who have a strongly held conviction for or against global warming will distrust both mainstream and preselected experts from the opposing camp, because of their own biases.

EDIT: Fugitive hit on this idea while I was composing my post

I have a related issue. Given that most of us are experts in some area, what are the most effective ways we can overcome these tendencies for distrust, and serve as sources of credible information to the public?

[Lunch of Kong]

Phil is our expert on experts.

[Phil_Stein]

Global warming is an interesting case study.

It's a tremendously complex issue. It involves a lot of areas:

What have the temperature and CO2 levels been in the distant past (often estimated by proxies)?

What have these values been in the more recent past (last 40-150 years or so)? We have more instrument based records for these, but interpreting them requires some expertise, for various reasons.

What will the CO2/greenhouse gas levels be in the future?

What temperatures will those levels lead to?

What impact will a particular temperature increase lead to in terms of sea levels, severe weather, rainfall and so on?

What will the negative (and, to a degree, positive) impacts for humans be as a result of those changes?

What policies can have what impacts on future growth in CO2/GG?

What technologies are promising and merits government support for addressing these issues?

If we understand future impacts and policy alternatives (debatable), how do we strike an appropriate balance between the costs and benefits of reducing global warming impacts?

How do we address the international diplomacy questions involved? (Different countries have different motivations...)

And so on...

Most of these questions are best addressed by separate groups of experts. There are outlier opinions (on both sides) within many of these areas, and even the mainstream view (the center of expert opinion) shifts around over time.

The good news is that there's an international effort to present a consensus view on many of these issues - the IPCC report(s). Of course, there are still experts who criticize IPCC conclusions, from both sides (they claim that the IPCC understates or overstates the problem). Still, as a non-expert, for global warming, I think the latest version of the IPCC report is probably as close to the center of expert opinion as I'll get, without more personal research than I'm currently willing to do.

[BennyProfane]

It also doesn't help when your governmental representatives deliberately choose experts at the fringe and tout them as authoritative, due to a desire to either cover up lack of progress, or actively support continued behaviour that is detrimental to the concern at hand. It just leads, at best, to people being even less likely to believe expert testimony.

[Rywill]

This is also a big deal for lawyers. A fair number of cases involve science or other issues beyond the experience of a normal person, so one or (usually) both sides put on experts to try and explain the evidence in a "battle of experts." Then we count on regular people to sort through the expert testimony from both sides and decide what they believe. They listen to two people, both of whom usually have reasonable qualifications, but who disagree on the key issue and often disagree on a lot of the science behind the key issue. Somehow we expect that twelve people who've had no exposure to any of this before and who probably aren't critical/scientific thinkers by nature to figure out who is right and who is wrong. Does that really work? How do people make those sorts of decisions?

[Qenan]

Is there any objective evidence that adversarial justice works in general? Serious question. I accept that it's better than mob rule, but am skeptical how reliable its decisions are.

[russellmz00]

http://www.slate.com/id/2111894/entry/2112064/

Blink/Tipping Point author Malcolm Gladwell and The Wisdom of Crowds author James Surowiecki discuss experts and other stuff

page 3:
But assuming that there is (at least in theory) an objective answer to the question "Does this mammogram show evidence of cancerous or precancerous cells?" then I have little doubt that if you averaged the judgments of a group of moderately trained people, you would end up with results as good as those produced by a board-certified radiologist. (The group wouldn't have to be as big as 1,000 people, either.) NASA, for instance, recently ran an online experiment called "clickworkers" to test whether the collective judgment of ordinary people would be of any use finding and classifying craters on Mars. You could go to the site, get trained (for a couple of hours, I think), and then click away. The result, in NASA's words: "the automatically computed consensus of a large number of clickworkers is virtually indistinguishable from the inputs of a geologist with years of experience in identifying Mars craters." And these people weren't even being paid.

[russellmz00]

Is there any objective evidence that adversarial justice works in general? Serious question. I accept that it's better than mob rule, but am skeptical how reliable its decisions are.

that depends on what your goal is. if you want someone to go to jail, then the adversarial arrangement works poorly. if your goal is "innocent until proven guilty", it probably ranks pretty highly.

[Rywill]

Is there any objective evidence that adversarial justice works in general? Serious question. I accept that it's better than mob rule, but am skeptical how reliable its decisions are.
I'm not sure what qualifies as "objective evidence" in your opinion. I mean, a lot of times -- particularly in civil suits -- the question at hand is one of judgment, like "Who is most at fault for this?" or something similar, where there isn't really an objective right or wrong answer. Even in cases with an objective true/false outcome ("Did this person shoot the murder victim?"), although there is a definite right or wrong answer, it's hard to somehow pull back the curtain after the trial and see if we got it right. I mean, if we had some other, verifiable way of finding the answer, we probably wouldn't hold the trial in the first place.

I can think of some exceptions, though -- for example, advancements in science sometimes make it possible to determine with much greater certainty whether the adversarial system got it right back when it made the initial decision. Outfits like The Innocence Project do investigations where DNA comparisons are used to determine whether certain convicted criminals did or didn't commit the crimes for which they were sentenced. TIP gets a lot of press, and rightly so, for their many successes (i.e., freeing wrongly convicted people). I would be interested to know what their "failure" rate is -- that is, how many times they do the DNA test and it turns out we locked up the right guy after all. (And note that TIP concentrates, as we would expect they would since they have limited resources, on cases where chances of a misidentification are high.)

Adversarial justice has a lot of problems, but it's sort of like Churchill's description of democracy: the worst form of adjudication, except for all the others.

[shift6]

Thoughts?
It applies to many other topics as well. Even those of us here on QT3 (above average educations, exposure to more points of view, etc) digest most of our information from intermediaries rather than experts and then spit it out as the gospel truth. Proof: wikipedia and blog links.

It's also the major thing that I bemoan about most of modern Christianity. So many of them (us) simply digest what some guy in a collar tells them as correct and base life decisions on it. Take for example Westboro Baptist Church (http://en.wikipedia.org/wiki/Westboro_Baptist_Church). There are literally hundreds of passages in the New Testament (which they claim to follow) talking about how we are to treat our fellow man. If you exclude the leadership of the church, that leaves maybe 50ish-100ish people who trust the leadership's agenda and "interpretation" of Scripture instead of simply reading it on their own. The same applies for other hot-button issues, including abortion and young-earth creationism.

And I think Sidd_Budd hits it on the head about why: once someone gets something in their head, there's no way to get it out. I'm not talking about an atheist changing his entire POV or about complex theological nuances; I'm talking about a "Bible follower" who doesn't even follow a simple reading of the Bible. I imagine the same thing is why you can have Islamic murderers following the same Koran as peaceful muslims.

[Phil_Stein]

I have a related issue. Given that most of us are experts in some area, what are the most effective ways we can overcome these tendencies for distrust, and serve as sources of credible information to the public?

I have an idea for a website that would, in part, try to parse and summarize a lot of expert opinion (from academia and such) in a manner useful to the general public.

The problem is that users, in general, are probably unlikely simply to trust my parsing of expert opinion (because I could be skewing things in my presentation, among other things). My rough solution would be to present information hierarchically - simply summaries at the top, with the ability to drill down for more detail, and a broad survey of expert opinion.

[Qenan]

I'm not sure what qualifies as "objective evidence" in your opinion. I mean, a lot of times -- particularly in civil suits -- the question at hand is one of judgment, like "Who is most at fault for this?" or something similar, where there isn't really an objective right or wrong answer. Even in cases with an objective true/false outcome ("Did this person shoot the murder victim?"), although there is a definite right or wrong answer, it's hard to somehow pull back the curtain after the trial and see if we got it right. I mean, if we had some other, verifiable way of finding the answer, we probably wouldn't hold the trial in the first place.

I guess I was thinking along the lines of made up cases (where all the facts are known): is there good evidence that adversarial justice will determine guilt or innocence correctly? As opposed, say, to the investigating magistrate system used by the French or Swiss?

You make a good point that many questions answered in civil suits involve more shades of gray, but I think it would be illuminating to look at the ability of the system to get simpler questions right.

I have a bias, or suspicion, which is that adversarial justice tilts the scales in favor of whichever side has the more talented, better-prepared team -- usually, whichever side has more resources. I'm not sure there's a better system, but I'm not very sanguine about the one we have.

[RickH]

Yes, this may be a blindingly obvious conclusion to many of you, and it's not entirely novel to me, but I think it's the reason why lots of folks shrug off expert opinions. I accept that experts, on average, probably know more than me about a given subject and perhaps have a more accurate assessment of what to do going forward, but the specific expert that any particular source quotes is likely to have an outlier assessment of things that is worse than my own gut instinct. This isn't always the case, of course, but it's true often enough to substantially reduce the value of expert opinion in persuading me of a particular argument.

* * *

Thoughts?

Two things. First, I'm not sure you're accounting for the power of incentive in your assessment of the value of expert opinions. Put simply, why is the expert in question giving his opinion? Does he want a grant? Is he being paid by one side of an argument? Is he trying to discredit a rival? Is he serving a particular political agenda? Is he lazy? Is he an egomaniac? Is he seeking press coverage?

Any of these factors can influence an opinion. In many ways, that's why the collective is usually more reliable than the individual, and why we seek out and value consensus among experts. Of course, consensus opinions are subject to influence as well.

Second, knowledge is knowledge, no matter whether you're a grad student or a scientist at the end of a distinguished career. Personally, I would just as soon have the facts, apply well-proven theories, and see where that puts me. Once you get beyond that realm, anything goes. Just like the example of global warming you cite above, there's just so many variables at work, how can anyone be as certain as they claim to be? Placing the line between what is known and what can be determined from what is known and what the experts think could be true is critical. Trouble is, most folks want to jump right to theory (the more dramatic the better) without firming up the foundations.

[Anaxagoras]

Adversarial justice has a lot of problems, but it's sort of like Churchill's description of democracy: the worst form of adjudication, except for all the others.

But that's the thing... is it? Why is it better than the Napoleonic model? Or the Mandarin model?

It seems like all justice systems ultimately rely on the quality of the people involved. And at that point.... is any system better than any other? (Note: I'm pretty sure the answer is definitely "yes" when comparing some of the systems. I'm just not convinced that the adversarial justice system comes out on top, as you assert.)

[Cad]

I have an idea for a website that would, in part, try to parse and summarize a lot of expert opinion (from academia and such) in a manner useful to the general public.

The problem is that users are unlikely simply to trust my parsing of expert opinion (because I could be skewing things in my presentation). My rough solution would be to present information hierarchically - simply summaries at the top, with the ability to drill down for more detail, and a broad survey of expert opinion.
The more the better. This is already happening in some areas, although slowly. For example, consider the Psychology Wikia:
http://psychology.wikia.com/wiki/Expert_article

If you're into wikis, you know they're probably the best bet for this sort of thing. If not, you might want to look into mediawiki (sql) or dokuwiki (txt files, no db needed).

The net/web is definitely at the point where we need to use reputation as a primary filter of content, because searching isn't good enough. This is already happening in a weak (yet useful) way with blogs being both news and analysis of varying reputation, but I think for now wikis are the best way to go, particularly if there's a combination of expert opinion and a layer of public comment. The public layer can add a lot of value, assuming there's a general community sense of conduct and purpose. They also can be a royal pain, but in so doing they may very well reveal public misconceptions that need clearing up. The nuts can always be kicked from the community. Transparency of sources and foundational information with links will make fact-checking that much easier for doubtful visitors, and increase the reputation of you and the site. People want reputable information, and the sharp ones will run with what you provide, and build on it.

I'll end this before I begin any "future of the net" ramblings, but good luck, and thanks.

[caesarbear]

Any expert at some point becomes a typical person. A legal expert, a wine expert, they'll both default to personal tastes at some point, but it's kinda unsatisfying to find yourself reduced to "I really don't know anything for sure." This is probably where Common Sense arrives to make us more secure, but it so often isn't what's true. It's more like the Common Uncertainty, and no expert is immune from relying on it.

[tiohn]

I have an idea for a website that would, in part, try to parse and summarize a lot of expert opinion (from academia and such) in a manner useful to the general public.


At least in my experience, you are talking about Wikipedia. Granted, I really only use it as a jumping-off point for math and physics junk, but I find that it often has parsed and summarized information that is useful on varying levels. I understand that this certainly isn't the case for all topics, but isn't wikipedia supposed to exemplify Gladwell's thoughts from Tipping Point and whatnot, at least to an extent?

[RickH]

But that's the thing... is it? Why is it better than the Napoleonic model? Or the Mandarin model?

It seems like all justice systems ultimately rely on the quality of the people involved. And at that point.... is any system better than any other? (Note: I'm pretty sure the answer is definitely "yes" when comparing some of the systems. I'm just not convinced that the adversarial justice system comes out on top, as you assert.)

Because it places the responsibility for bringing forward the best evidence and strongest possible arguments in the hands of the parties with the greatest incentive to do their best.

As opposed to, for example, a bureaucrat who will not be personally impacted and get paid anyway no matter what the outcome.

EDIT: I no spel gud.

[Rimbo]

Arguments from authority carry little weight (in science there are no "authorities").

Carl Sagan's Baloney Detection Kit (http://users.tpg.com.au/users/tps-seti/baloney.html)