Estimation Quiz

[Phil_Stein]

This quiz is based on something I read about people's ability/inability to make reliable estimates.

All questions in this quiz can be answered as a positive integer value (rounded to the nearest, where applicable).

For each question in this quiz, provide an answer that you think has a 90% chance of including the correct value.

So for instance, if the question was, "What year was Ludwig van Beethoven born?", instead of simply taking a stab (say, 1780), you should instead have a range that you think gives you a 90% chance of being correct (let's say, 1710-1830). FWIW, the correct answer is 1770.

The answers are mostly based on Wikipedia, and the sources will be provided with the answer key.

Please answer these off the top of your head, without further research. You can obviously think about the questions a bit to arrive at a reasonable answer, but don't go to Wikipedia or otherwise research or overthink things.

Try to strike a balance between providing meaningful information but not guessing so narrowly that you'll likely miss more than 10% (1 in 10) of them.

>>>>>

1) What is the weight, in pounds, of an adult male African elephant?

2) In what year did Christopher Columbus die?

3) What is the diameter of the earth, in miles, at the equator?

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)

6) How long (in miles) is the Rhine river?

7) How many home runs did Stan Musial hit in his MLB career?

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08)

9) How many "member states" are in the United Nations?

10) How many prime numbers are there under 1000?

<<<<<

Note that I'm not intending trickiness with these questions - in a few places, I tried to narrow terms or definitions a bit, and tried to use what was easiest to find on Wikipedia and/or seemed like the sensible answer. Also note that units are miles, pounds, and such - all you Frenchie metric types please post your answers in imperial units. Answers are based on what I found on the web as of 6/9/2008.

REMEMBER - TRY TO PROVIDE A RANGE THAT HAS A ~90% CHANCE OF INCLUDING THE CORRECT VALUE.

The answers are in post 3 (below the spoiler space post immediately following this one).

If you post your answers in this thread, and we get a decent number of answers, I may do some summary analysis. Posting your answers here grants permission for such analysis :)

[Phil_Stein]

Spoiler Space
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

[Phil_Stein]

Answers:
1) 12,000 pounds (http://en.wikipedia.org/wiki/African_elephant)
2) 1506 (http://en.wikipedia.org/wiki/Christopher_Columbus)
3) 7926 miles (http://www.nasa.gov/worldbook/earth_worldbook.html)
4) 11,326 (http://en.wikipedia.org/wiki/Closing_Milestones_of_the_Dow_Jones_Industrial_Ave rage)
5) 1,499,402 (http://en.wikipedia.org/wiki/List_of_U.S._states_by_population)
6) 820 miles (http://en.wikipedia.org/wiki/Rhine)
7) 475 (http://en.wikipedia.org/wiki/Stan_Musial)
8) 115 (http://www.quartertothree.com/game-talk/search.php?searchid=1161120)
9) 192 (http://en.wikipedia.org/wiki/United_Nations_member_states)
10) 168 (http://en.wikipedia.org/wiki/List_of_prime_numbers)

[Phil_Stein]

More Spoiler Space
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.

[ElGuapo]

1) 3500-4500 lbs

2) 1510-1540 AD

3) 8000-8400 miles

4) 10,300-12,000 points

5) 1.2 - 1.5 million people

6) 900-1800 miles

7) 50-200 Home runs (no idea really)

8) 100-150 threads

9) 130-170 nations

10) 300-360

Edit ... clearly I don't know much about baseball stats and Elephants are HEAVY!

[Phil_Stein]

Please post your score too - yes, if your answers are posted, your score can be calculated, but it will be a little easier to get the gist of things by skimming posts and looking at scores.

The idea for this was based on something I read a while ago - I think an experiment by a psychologist showing that most folks are quite overconfident about their knowledge. I can't find the link to the original study right now (assuming there was a study), though I did find an article pretty much on the same thing here:

http://www.bookofjoe.com/2007/12/05/index.html

According to the article, less than 1 percent of people got either 9 or 10 right - i.e. most folks answered with ranges that were too narrow, reflecting overconfidence.

Of course, it's possible that you could accurately answer each question with a 90% confidence range but miss more than 1 due to bad luck (though I suppose this depends a bit on how we define terms). But if the vast majority of answerers miss 2 or more, then that would tend to confirm the overconfidence idea.

[Enidigm]

This is, imo, a bad quiz; it's testing whether or not you understand to frame and describe your boundaries, which is really unintuitive, imo, in our culture today, raised on the idea that your answers are right or wrong.

The correct answer for the elephant is, say 1lb to 100,000 lbs. The correct answer for Columbus' death is 1 million BC to today. Ect. Because these "correct" responses are not only ridiculous but reveal the fact that you're admitting that you don't know, most people would probably hazard a narrower range just to prevent themselves looking ignorant. Yet, once you know the nature of the questionnaire, anything other than a completely open-ended answer is nothing but a risk.

In other words, imo, you need clearer language about the limits of your answer; in which case, it just becomes another knowledge quiz like any other. If you limit the guessing weight to +/-500lbs, but your knowledge of elephants is 0, what does it prove? That you're overconfident, or that you have no knowledge of elephants?

[Quaro]

Maybe bold the 'get a most one wrong' part. YOU LOSE the GAME if you get more than one wrong. I know my brain keeps pushing to aim for narrow ranges, to hit a bulls eye on a dartboard. Stupid brain.

[MikeJ]

1. 4500-6500 lbs
2. 1500-1520 AD
3. 8000-9000 miles
4. 9000-11000 points
5. 500,000-1,500,000 people
6. 400-900 miles
7. 10-200 home runs
8. 50-500 threads
9. 100-150 member states
10. 40-80 primes

(incorrect bolded)
Overall only 4/10 correct. Was just a little too narrow on several.

[Phil_Stein]

The correct answer for Columbus' death is 1 million BC to today.

That would be correct if the quiz asked you for 99.99%+ confidence.

I was quite specific that I was looking for 90% confidence. 90% is a reasonable target - IMO, it strikes a balance between providing meaningful information (i.e. avoiding the Columbus example you cite), and yet understanding that in many domains, the typical person's knowledge is imprecise, and that should be taken into account in decision making, providing information, and so on.

[Talisker]

1) 10,000-15,000 pounds
2) 1495-1535
3) 7,000-8,000 miles
4) 8,000-11,000 points (wrong)
5) 1mil - 10mil people
6) 100-1000 miles
7) 300-500 HR's
8) 50-300 "favorites"
9) 100-150 member nations (wrong)
10) 100-200 primes

80%.

[Equis]

I read this recently in the Black Swan. (http://www.amazon.com/Black-Swan-Impact-Highly-Improbable/dp/1400063515/ref=pd_bbs_sr_1?ie=UTF8&s=books&qid=1213035379&sr=8-1) Something to do with Epistemological arrogance and the inability of people to willingly make themselves look stupid by admitting they don't know. And a correlation to certain academics making narrower ranges in fields they're supposed to be experts in, but still get wrong.

Not that I won't guess, but I'll straight up tell you that I wouldn't make a decent estimation on anything I don't have even a passing familiarity with. And the ones that I do, I will probably estimate far beyond the accurate range. For instance, I am not American. I have no studied Christopher Columbus and I'll be damned if I can even get the right century of America's discovery. 17th century maybe? 16th? So asking me to give a range, I would hazard a wide range. How many prime numbers under a thousand, somewhere between 50 - 1000, just to be correct. Why hazard a guess on something so obscure as number of threads with the word favourite.

Truthfully though, before I read this, I didn't realize people would do that. Now that I do, I'm taking several steps to lessen that "epistemological arrogance" etc. If I don't know something, I'd rather admit, and be given time for research, then stand around making an eventual fool of myself.

Or I could just pretentiously quote Socrates "Wisest is he who knows he does not know."

[BrewersDroop]

10,000 - 12,000

1510 - 1520

10,000 - 14,000

7,000 - 8,500

300,000 - 400,000

800 - 1,000

200 - 300

40 - 70

180 - 220

20 - 30

[Phil_Stein]

Quaro - I think the initial post is clear enough, without turning this into a game. The point is to see if people are instinctively overconfident/overprecise.

If people like this thread, perhaps I will post another with a new set of questions and make it super-clear that it's basically a test of overconfidence, the 1 in 10 thing, and so on.

[Equis]

Quaro - I think the initial post is clear enough, without turning this into a game. The point is to see if people are instinctively overconfident/overprecise.

If people like this thread, perhaps I will post another with a new set of questions and make it super-clear that it's basically a test of overconfidence, the 1 in 10 thing, and so on.

I also think to be more accurate in gathering that sort of data. You should post questions which are closer to the QT3 hivemind knowledge base.

Like

How many have been banned from QT3. What is the total post by all of Tim's alts, how many people has Bill D mocked mercilessly, computer games, etc, etc, etc.

[Brendan]

1) What is the weight, in pounds, of an adult male African elephant?

2) In what year did Christopher Columbus die?

3) What is the diameter of the earth, in miles, at the equator?

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?

5) What is the current population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)

6) How long (in miles) is the Rhine river?

7) How many home runs did Stan Musial hit in his MLB career?

8) How many threads on Quarter to Three have the word "favorite" in their title? (With that spelling, and as of this writing, 6/9/08)

9) How many "member states" are in the United Nations?

10) How many prime numbers are there under 1000?



Well, let me show you all how stupid I really am.

1. 6000-8000lbs (Under estimated)
2. 1510-1520 (Over estimated but close)
3. 10 000 - 12 000 miles (Over estimated)
4. 6000 - 8000 (Under estimated)
5. 5 000 000 - 6 000 000 (Over estimated by a large margin)
6. 800 - 1 000 miles? (Correct)
7. 200 - 300 (Under estimated)
8. 400 - 500 (Way over estimated)
9. 180 - 200 (Correct)
10. 127-150 (Under estimated)

[skedastic]

The correct answer for the elephant is, say 1lb to 100,000 lbs. The correct answer for Columbus' death is 1 million BC to today. Ect.

Phil asked for 90% belief regions, not regions which contain the true value with certainty. He wants intervals which you believe will cover the correct answer 90% of the time. If your answers are always "correct" in the sense that you give intervals which always cover the correct answer, you're not answering correctly.

Consider the date of Columbus's death. Almost everyone will remember that Columbus must have been alive in the year 1492, so if you knew nothing else about you might guess that he was somewhere around 40 (neither too old nor too young to lead such an expedition) in 1492. It's very implausible, then, that he lived beyond about 1550, and obviously we can place zero weight on his his death occurring prior to 1492. If you knew nothing else about him, you might guess that he lived to about 70, placing his death around 1520, but he could easily have died earlier. So someone who remembered nothing about Columbus except 1492 might place 90% probability on the interval 1495 to 1525 covering the data of Columbus's death.

I would answer Phil's questions, but I glanced at the correct answers and spoiled it.

[Gremlinclr]

1> 8,000 - 12,000
2> 1693 - 1750
3> 30,000 - 40,000
4> 1,000 - 2,000
5> 6 - 10 million
6> 600 - 1000
8> 100 - 200
7> 500 - 700
9> 50 - 75
10> 200 - 400

EDIT: My brain doesn't seem to be wired for random fact retention, unless it's crap like my high school locker combination 20 years ago. Sigh.

EDIT II: On #2 I really did know the rough answer I just brain farted an extra 200 years up there. Really.

[Phil_Stein]

But really, you should be able to take a ~90% stab at just about any question, whether you have a good deal of knowledge about the subject matter or not. If you know little, use a wide range, and vice versa.

Even if I had asked for the birthdate of [some person you've never heard of], you could reasonably infer that the person was born within the bounds of recorded history and frame an answer that way. And I think that for the 10 questions provided, most folks have at least heard of the person/place/thing/concept, even if they know little about it/them beyond a faint ring of familiarity. And for most of the questions, I think the typical QT3 poster's knowledge is at least moderately solid.

I can believe that academics would choose a too narrow range in fields they are supposed to know. There's an element not only of overconfidence in this kind of thing, but also, that we feel we SHOULD provide a confident answer (narrow range) even if our confidence is not very high.

This may be part of the reason why we sometimes see two acknowledged experts in a field disagreeing strongly about aspects of it.

[Mr_PeaCH]

Elephant............. 2500-25000
Columbus............ 1492-1542
Earth.................. 3333-6666
1999DOW............ 5000-15000
Idaho.................. 250000-2500000
Rhine................... 75-750
MusialHR.............. 199-599
Favorites.............. 3-333
UNmembers........... 100-200
Prime<1000............ 10-100

I tried to leave them, imo, pretty wide in an effort to get 90% right but missed 3/10.

I feel pretty stupid about the earth's diameter and looking back it is one of the guesses that I made 'too narrow'.

[Hanacker]

1) 2000-8000 Those suckers are heavy

2) 1493-1530

3) 10,000-40,000 Guess I was thinking km

4) 6,000-10,500 Huh, our economy used to be pretty good.

5) 500,000-4,00,000

6) 15-2000

7) 50-500

8) 1-10 I thought we were only talking about the first page of each forum.

9) 100-400

10) 5-50 Yeah, should have thought about this one more.

Woo, posted what I thought were ridiculously large ranges and missed half. Go me!

[JPR]

6/10

1) 2000 - 4000
2) 1502 - 1530
3) 5000 - 7000
4) 8000 - 10000
5) 2000000 - 4000000
6) 300 - 2000
7) 30 - 400
8) 20 - 200
9) 100 - 200
10) 30 - 100

[Phil_Stein]

The mention of The Black Swan and the term "Epistemological arrogance" helped me to find a relevant passage in that book. (I think this is what has caused this to be in the back of my mind for a while.)

They describe (starting on page 138) an experiment in which people were asked to make range statements with 98% confidence (i.e. they should be VERY confident, and would, if they were thinking things through correctly, set very wide ranges). In fact, according to the book, Harvard Business School studends had an error rate of 45% (i.e. instead of their ranges being right 98% of the time, they were right 55% of the time).

According to the book, other populations like janitors and cabdrivers were less overconfident, and typical populations have error rates of 15-30% when trying to achieve 2% error rates.

[Nengjanggo]

1. 2000 - 4000
2. 1492 - 1544
3. 4000 - 10000
4. 5000 - 15000
5. 2,000,000 - 10,000,000
6. 100- 1000
7. 10 - 1000
8. 1000 - 500,000
9. 100 - 500
10. 100 - 600

7/10 correct

I feel that one of those needs explanation - my guess on how many threads have "favorite" in the title. In my defense, I had no idea whatsoever, but I thought I remembered seeing on the front page that there were 15 million threads (apparently my brain mushed together the number of posts and the number of members.

Also, re. number 5, I'm from Los Angeles and I can't imagine a city, much less a state, with so few people.

At least I'm as smart as a cabdriver.

[Alan Au]

I dunno, this quiz relies too much on trivia and not enough on estimation. I mean, I'm pretty lousy at estimation, but many of these questions don't provide any point of reference. When I think of estimation, I think about taking something I know well and extrapolating it out to some unknown set of conditions. Unfortunately, for these, I don't know where to start. You may as well have asked about the average annual rainfall in the Amazon River basin (80 inches), or the original height of the Great Pyramid of Cheops (480.9 feet).

- Alan

[Reldan]

>>>>>

1) What is the weight, in pounds, of an adult male African elephant?

4,000-5,000 lbs

2) In what year did Christopher Columbus die?

1500-1520

3) What is the diameter of the earth, in miles, at the equator?

6,000-8,000 miles

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?

15,000-18,000

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)

1.2-1.4 million

6) How long (in miles) is the Rhine river?

100-500 miles

7) How many home runs did Stan Musial hit in his MLB career?

300-500

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08)

400-600

9) How many "member states" are in the United Nations?

100-120

10) How many prime numbers are there under 1000?

70-80

[Destarius]

I dunno, this quiz relies too much on trivia and not enough on estimation. I mean, I'm pretty lousy at estimation, but many of these questions don't provide any point of reference. When I think of estimation, I think about taking something I know well and extrapolating it out to some unknown set of conditions. Unfortunately, for these, I don't know where to start. You may as well have asked about the average annual rainfall in the Amazon River basin (80 inches), or the original height of the Great Pyramid of Cheops (480.9 feet).

Agreed. What is the population of HeiLongJiang, China? zzz.

[Lurb]

5/10 some wild guesses hit, some that I knew a bit about went boom.

1) What is the weight, in pounds, of an adult male African elephant?
5k -8k pounds

2) In what year did Christopher Columbus die?
1500-1530

3) What is the diameter of the earth, in miles, at the equator?
3k - 4k miles

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?
10000-12000 whatyacallits

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)
500k - 2 million people?

6) How long (in miles) is the Rhine river?
700-1300 miles

7) How many home runs did Stan Musial hit in his MLB career?
150-300

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08)
400-800

9) How many "member states" are in the United Nations?
150-200

10) How many prime numbers are there under 1000?
50-100

[skedastic]

According to the book, other populations like janitors and cabdrivers were less overconfident, and typical populations have error rates of 15-30% when trying to achieve 2% error rates.

An alternate interpretation is that people have a very hard time understanding probabilities. The statistical notion here is really quite difficult: it's almost but not quite a confidence interval. Graduate students in statistically intensive disciplines often have a hard time grasping what a confidence interval is. Are people answering these sorts of quizzes really overconfident, or do they simply not understand what they're being asked?

When researchers need to elicit probabilistic responses, they go to great lengths to use various visual aids to get at probabilities. They never just ask for numerical responses or state questions in terms of probabilities, as it's well-known that people won't understand. Do you have a cite to the study discussed in the book, Phil?

[Mordrak]

They describe (starting on page 138) an experiment in which people were asked to make range statements with 98% confidence (i.e. they should be VERY confident, and would, if they were thinking things through correctly, set very wide ranges). In fact, according to the book, Harvard Business School studends had an error rate of 45% (i.e. instead of their ranges being right 98% of the time, they were right 55% of the time).

According to the book, other populations like janitors and cabdrivers were less overconfident, and typical populations have error rates of 15-30% when trying to achieve 2% error rates.

I think Socrates (or at least Plato as Socrates) had something to say about this during his Apology.

[skedastic]

Agreed. What is the population of HeiLongJiang, China? zzz.

But you can answer without knowing anything at all about the question.

I've never heard of HeiLongJiang. I don't know if it's a city or something else, I don't know whether it's big or small, I don't even know if it's a real place. I am to give an interval which I think has the property: there is a 10% chance that the true population is either greater than the upper limit or smaller than the lower limit.

Well, I think it's pretty unlikely Destarius just made up the name, it sounds like a real place. I also infer that, if it's a city, it can't be extremely large, because I know the names of the most well-populated Chinese cities. But it could be a province, including a province which contains one or more very large cities. I can't rule out the possibility that the population of this place is very large, many hundreds of millions, but I also guess that I'd know the name if it were that big.

So, my answer would be 100,000 to 50,000,000. That is, given the information I have, I think there is a 10% chance that this place has a population greater than 50M or smaller than 100,000. That's a very wide interval, reflecting the fact that I am not able to precisely estimate the population of this place.

[Phil_Stein]

The book mentions the original discoverers of this phenomenon as Albert and Raiffa, but does not appear to have a cite for a specific paper.

This paper concerns the same general topic:
"Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments" by Kruger and Dunning

There is further discussion in a non-academic article here: http://www.arthurdevany.com/webstuff/montier-follyofforecasting.pdf

That article replicates a quiz (that my quiz is inspired by) by Russo and Schoemaker ("Decision traps: Ten barriers to brilliant decision making and how to overcome them")

I agree that the concept of a confidence interval is a bit tough for the uninitiated, but I've read and seen enough statements of bald (and, IMO, unmerited) confidence (X will happen by Y date, or we know A and B about issue C) that I believe the general problem goes well beyond the inability to grasp percentage confidence intervals.

The Montier piece includes a few great examples of another problem that's probably related to this: anchoring.

If you ask people to solve 1*2*3*4*5*6*7*8, they estimate 512 (median), but if you present it as 8*7*6*5*4*3*2*1, they estimate 2250 (median).

Using essentially random values (a number from a wheel of fortune, the last 4 digits of people's phone numbers) to frame an initial guess (above or below the 'random' value), then asking them for a specific guess, shows that the initial 'random' value has a great deal of influence, even when the question shifts from "above/below" to "what's your guess" and the latter ignores that random value.

[JPR]

Somewhere in this thread, people switched from bolding their incorrect answers to bolding their correct answers.

[Bill]

Or maybe most people are lazy and just picked a single number, knowing that if it actually mattered they would create some arbitrarily large range. That's what I did at least.

I agree Alan's point that many of these questions aren't really about estimation. For example, Stan Musial? No idea who this even is. I think the type of question that would be better is something along the lines of "How many home phones (non cell phones) are there in the US?" It makes you estimate how many homes there are in the US and how many phones are in each.

[skedastic]

There appears to be a pretty big, and not entirely consistent, literature on the subject of overconfidence. The Alpert and Raiffa piece is

Alpert, M., & Raiffa, H. (1982). A progress report on the training of probability assessors. In D. Kahneman, P. Slovic, & A. Tversky (Eds.), Judgment under uncertainty: Heuristics and baises (pp. 294–305). New York: Cambridge University Press.

but the idea goes back farther. After a brief read, I'm not entirely convinced. For example, if you ask people to give 90% confidence regions, their regions should cover the correct answer 90% of the time, but they are usually wrong far more often. On the other hand, if you ask people the same battery of questions and then ask them to guess how many of those question they got correct, they are underconfident in that they on average guess they're right at lower frequency than they actually are right. There is some debate in the literature on what all this means.

Anchoring effects are one type of framing effect, which means that the answers you get depend on how you frame the question. I don't think such effects are directly related to overconfidence, except inasmuch as people studying overconfidence ought to be designing their instruments to minimize framing effects.

[Robert Sharp]

Here are my guesses, without reading anyone elses:

Edit: to Save space, since I didn't seem to get the question. I honestly have no idea how to make ranges that would create a 90% chance of being right about something that I have no way of estimating.

[skedastic]

Now that I've checked, I wasn't even close on most of them. But I was practically guessing wildly on most of those.

Then why aren't your ranges much wider? Did you not understand the question? Your answers to 4 and 7 in particular suggest not.

I don't know from baseball. I don't watch it, I don't play it, I don't give a shit about it. I've never heard of the guy in Phil's question. But if I had to give a 90% interval containing the number of home runs he hit, I'd think: well, he's almost certainly a famous baseball player. Probably he's famous because he hit lots of home runs. How many is a lot of home runs? Colloquially I might say "I have no fucking clue," but of course I do. I'd bet a lot money that he didn't hit more than 100,000 home runs, for example, and I'm certain he didn't hit fewer than zero. I can narrow it down more than that by guessing that a really good MLB player might be in the league for 10 years, play 50 fifty games a year, and hit one home run a game, so 500 might be a reasonable guess. But I could be quite markedly off in any of those guesses, so for me there's a great deal of uncertainty. I want to give a range with the property that I think there's a 10% chance that the true value falls outside the range. I'd say, then, 50 to 1,000. This is a very wide range because I know fuck all about the question.

[Robert Sharp]

I guess I didn't. So I was supposed to make the range as big as necessary to insure I would be right? Why not make it 1-1 trillion then? I have no way of making 90% chance of being right into anything meaningful as far as ranges are concerned.

[skedastic]

I guess I didn't. So I was supposed to make the range as big as necessary to insure I would be right? Why not make it 1-1 trillion then? I have no way of making 90% chance of being right into anything meaningful as far as ranges are concerned.

You're supposed to report a range which covers the true value 90% of the time, not 100% of the time. Suppose Phil asked you a very, very long list of questions such as the ones he did ask. You should get about 9 in 10 of those questions correct. If you get more than 9 in 10 correct, you're giving ranges which are too large. If you get less than 9 in 10 correct, you are giving ranges which are too small.

[magnet]

Suppose Phil asked you a very, very long list of questions such as the ones he did ask. You should get about 9 in 10 of those questions correct. If you get more than 9 in 10 correct, you're giving ranges which are too large. If you get less than 9 in 10 correct, you are giving ranges which are too small.

As you pointed out in a concurrent thread, this is only true in the long run. Getting 8/10 on Phil's quiz doesn't necessarily mean your ranges were too small. The problem is that quizzes like this are usually "passed" or "failed" at the end, but only ten questions are not enough to come up with a full assessment of your overconfidence.

It might be easier to consider it in terms of wagering. For example, what would you consider to be a fair spread for a 9:1 bet on the weight of an elephant (i.e. you would be willing to cover either side of the bet)?

[MikeJ]

As you pointed out in a concurrent thread, this is only true in the long run.

This is why he specified a "very, very long list of questions".

[Phil_Stein]

magnet, you're correct on both counts. It would take a longer list of questions to have at least a reasonably good idea if any one person was able to hit the ~90% range with reasonable accuracy, and yes, the betting analogy you suggest is apt.

Of course, if I'd posted a 50 question test, few would have answered it (and it would have taken more time than I cared to commit).

But in any case, while the fact that a particular person only got 8 or 7 or even 6 right does not prove with certainty that they were wrong in their ranges (they could have been unlucky), the results of the group as a whole suggest that the average person is overconfident (or at least, not very well able to frame answers as ranges with a 90% confidence level).

I had seen this phenomenon written about, based on studies that presumably had far more rigor than this little internet quiz, but I wanted to see the phenomenon in action for myself.

===

Edit: One clarification on translating this into a 9:1 bet. Such a bet would be acceptable only if the side you had to take was randomly chosen. If I express my confidence range and then allow someone to bet inside or outside at 9:1, they will make that bet based on THEIR knowledge, which might be far more precise than my own. i.e. They might have a narrow range for a 90% confidence interval for a question for which my own range is broad.

[Robert Sharp]

To me, the more interesting data show that people tend to be more overconfident the less competent they are. For example, studies show that people who know little about a subject are more likely to overestimate their abilities than someone who is an expert, who tends to more or less know what they do or don't know. So dabblers in Greek history might guess that they got a 75% on a test on the subject, when they only got a 45%. While an expert might guess that he/she got an 85%, which is roughly what he/she DID get.

In other words, people who are ignorant often don't know how ignorant they are. That makes sense on one level, but on another it's interesting.

[Phil_Stein]

Assuming I have tabulated/scored correctly (and that the self-scorers reported correctly):

Twelve respondents have given their answers so far. Remember, the 'expected' score is a 9.

The best score so far is a 8 and the worst is a 2. The median is 4.5, and the mean is 4.7.

Hanacker missed question 8 due to arguably unclear wording in the original (since cleaned up) and Gremlinclr missed question 2 due to brain fart, so give a very small nudge of extra credit if you'd like.

[Lazy Shiftless Bastard]

In the spirit of not reading the thread and feebly trying to guess things I have no idea of, here I go:

1) 5000-8000 pounds.

2) 1500-1550 AD. I mean, he probably died somewhere in there, right?

3) 10000-15000 miles. I probably just failed geography.

4) 100-500. Off the top of my head I don't even know what the range of the stock market IS.

5) 20-30 million.

6) 100-500 miles. It's in... uh... Europe, right?

7) Who? 60-70?

8) 100-10000 threads? This is a question that deserves "Shit, boners!" if I ever heard one.

9) 150-250. I am a stupid American, how would I know?

10) 20-40.


It's depressing how I have no goddamn clue what any of those answers are off the top of my head.

I just went back and read the original post and saw the "Try to give a range for 90% chance of correctness." I should've really widened my ranges. Actually, I should've just answered 1-million for all of them. So I only got like three right instead of all 10.

[Talisker]

The best score so far is a 7 (two people have scored that) and the worst is a 2. The median is 4.5, and the mean is 4.7.
By my estimation, Phil has overlooked between 1 and 10 posts (I got 8 of 'em right, so nyah!).

[Phil_Stein]

Doh!

I had your score in my tally, and I used Excel functions for median and average (mean), but I just eyeballed for the low and high values.

But I never said I was 90% confident in my tabulation - so there!

[chemdem]

1.) What is the weight, in pounds, of an adult male African elephant? 7000-30,000

2) In what year did Christopher Columbus die? 1500-1550

3) What is the diameter of the earth, in miles, at the equator? 4000-10000

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999? 10500-11500
This one is my best guess. I remember that around that time the dow had reached 11000 but I could not tell if the date was before or after that event.

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate) 1 Million-4Million

6) How long (in miles) is the Rhine river? 100-5000

7) How many home runs did Stan Musial hit in his MLB career? 0-500
The name does not ring a bell, so it was possible he never even played baseball. The upper limit is a threshold for not knowing his name.

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08) 5-500

9) How many "member states" are in the United Nations? 80-140

My memory of the number of nations in the world was short by 100 or so.

10) How many prime numbers are there under 1000? 100-350

[Sidd_Budd]

I have read none of the previous responses, to not be influenced by other posters.

1) Elephant weight - 4000 to 12000 pounds
2) Columbus death - 1492 to 1570 CE
3) Diameter of Earth - 6450 to 10750 miles
4) Dow sometime in 99 - 2775 to 3700
5) Population of Idaho - 1 million to 4.5 million people
6) Length of Rhine - 400 to 1000 miles
7) Some person's career home runs - 10 to 1000
8) "Favorite" titled posts on Qt3 - 400 to 9500
9) Number of nations in UN - 82 to 147
10) Number of prime numbers from 1 to 1000 - 67 to 250

SPOILERS BELOW, WHERE I INDICATE HOW MANY OF THE ABOVE WERE CORRECT
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
..
I got 7 of 10 correct. The hypothesis "Sidd can self-select error bars for general knowledge questions with 90% confidence" can be rejected at a confidence level (p-value) of .07.

The overconfidence bias is really amazing. I did a few quick back-of-the-envelope calculations to reach the above answers (just using logical reasoning; not looking at any outside sources) and I was astounded how wide I had to make my error bars in some cases to give myself a 90% confidence. Even with the wide bands, I still overestimated my knowledge.

[Phil_Stein]

Can I just point out here that y'all need to read up on your baseball.

Stan Musial = The Man

http://en.wikipedia.org/wiki/Stan_Musial

22 seasons, 475 HR (putting him 6th on the all-time list at the time he retired), .331 lifetime average, 3 time NL MVP, Top 10 all time in games, at bats, runs, hits, total bases, doubles, RBIs and probably some other stuff. He was a consistent, oustanding career with a long and productive career (including being on 3 World Series winners).

Yeah, ok, he played for the St. Louis Cardinals, not the New York Yankees, but still...

[Phil_Stein]

[Deleted - Attempt at creating spoiler space here belatedly recognized as pointless...]

[Phil_Stein]

Bumping this old thread that I had fun with (and I think some of you enjoyed). Perhaps some folks missed it the first time around.

If you HAVEN'T read this thread, don't skim through any of the responses above - go directly to the first post, read it, and take the quiz.

Anyways, if you found this interesting, perhaps you may want to try the much simpler prediction game I just posted here (http://www.quartertothree.com/game-talk/showthread.php?t=50937). Go to that thread, send me your prediction, and hopefully we'll have a nice discussion about the results in a few days.

[Tman]

1) 8,0000-16,000 lbs
2) 1500-1540
3) 7000-8000 miles
4) > 10,000
5) 800,000-1,500,000
6) 800-1400 miles
7) 150-300 Home runs
8) 200-400 threads
9) 200-400 nations
10) 20-30

[Fugitive]

I missed this thread before, but did something pretty much like it as part of a software schedule estimation course. The point there was to show just how bad people are at estimating things, that it's better to go for too broad than too narrow, and to help pinpoint where the 'clearly impossible' edges of estimation are.

[Ed Solomon]

1) What is the weight, in pounds, of an adult male African elephant?

12,000 - 15,000

2) In what year did Christopher Columbus die?

1515-1530

3) What is the diameter of the earth, in miles, at the equator?

22,000 - 24,000

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?

1,200 - 1,400

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)

1,000,000 - 1,500,000

6) How long (in miles) is the Rhine river?

750 - 1000

7) How many home runs did Stan Musial hit in his MLB career?

250 - 400

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08)

1,500 - 2,500

9) How many "member states" are in the United Nations?

175 - 200

10) How many prime numbers are there under 1000?

75 - 125

Edit: Crap! I thought it was circumference of the Earth, not diameter. Damn reading comprehension failures.

[Squirrel Killer]

I'm including exact guesses for my own curiosity.

1) What is the weight, in pounds, of an adult male African elephant?
I'd guess 10,000 pounds, and would give a range of 8,000-15,000 to get 90% accuracy.
90% Correct!

2) In what year did Christopher Columbus die?
I'd guess 1515, and would give a range of 1510-1525 to get 90% accuracy.
90% Wrong!

3) What is the diameter of the earth, in miles, at the equator?
I'd guess 8,500, and would give a range of 7,000-9,000 to get 90% accuracy.

EDIT: I, too, read it as circumference. I'd guess 5,500, and would give a range of 4,000-7,000 to get 90% accuracy based on the real question. I realize the basic geometry of the two guess is likely wildly at odds with each other.
90% Wrong!

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?
I'd guess 7,500, and would give a range of 6,000-8,500 to get 90% accuracy.
90% Wrong!

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)
I'd guess 3 million, and would give a range of 2 million to 3.5 million to get 90% accuracy.
90% Wrong!

6) How long (in miles) is the Rhine river?
I'd guess 450, and would give a range of 350-750 to get 90% accuracy.
90% Wrong!

7) How many home runs did Stan Musial hit in his MLB career?
I'd guess 300, and would give a range of 125-500 to get 90% accuracy.
90% Right!

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08)
I'd guess 150, and would give a range of 75-300 to get 90% accuracy.
90% Right!

9) How many "member states" are in the United Nations?
I'd guess 220, and would give a range of 175-250 to get 90% accuracy.
90% Right!

10) How many prime numbers are there under 1000?
I'd guess 35, and would give a range of 20-50 to get 90% accuracy.
90% Wrong!

4 out of 10. I haven't read the thread closely yet, but I think it's interesting that the answers I felt most confident about (#2, 3, 4, 5, 9), I only got one right.

[Kael]

I missed 2 because I read the diameter as circumference, and I know nothing about baseball (and I thought my 0-300 range was wide enough).

[Rightbug]

Incorrect estimations bolded...

1) What is the weight, in pounds, of an adult male African elephant?

2,500 - 4,500 Off by a 7,500 lbs!!!

2) In what year did Christopher Columbus die?

1470 - 1510

3) What is the diameter of the earth, in miles, at the equator?

6,000 - 8,500

4) What was the closing value of the Dow Jones Industrial Average on August 25, 1999?

7,500 - 11,000 Off by only 326 pts!

5) What is the population of Idaho? (Per July 1, 2007 U.S. Census Bureau estimate)

500,000 -- 1,500,000

6) How long (in miles) is the Rhine river?

150 - 800 Off by only 20 miles!

7) How many home runs did Stan Musial hit in his MLB career?

200 - 800

8) How many threads on Quarter to Three have the word "favorite" in their title? (i.e. How many total threads can be found through an Advanced Search, titles only, for "favorite", with that spelling only, not case-sensitive, and as of this writing, 6/9/08)

15 - 200

9) How many "member states" are in the United Nations?

50 - 300

10) How many prime numbers are there under 1000?

5 - 50 Off by 118

So, only 6 correct which is a definite fail but I was sort of interested to see that two of the incorrect answers were really, really close.

[Flowers]

A better use for the data collected in this thread is the level of resentment when ignorance of a subject or fact is exposed. With the exception of Stan Musial and the Dow Jones at a certain point, those are all facts that you have been told, probably in grade school, and either forgotten or tuned out. But man, did some people not enjoy not being able to answer your questions off the top of their head. You could use that information to create a purposefully infuriating test to measure other factors.

[Phil_Stein]

What I think is interesting is the default method for widening from a point estimate to a 90% confidence range.

Often, folks seem to think that if their point estimate is 100, than an adequate 90% confidence range is about 75-150 or so. In many cases, to hit 90% confidence, you need to be thinking in orders of magnitude, not on a more linear scale.

For instance, a lot of folks set a relatively small range for a question (number of prime numbers under 1000) that, apparently, few have much intuitive grasp of. Assuming you know what a prime number is, it's easy enough to scale down the upper end of the range to 501 (eliminate even numbers other than 2), and to bring it somewhat lower (there's gotta be a bunch of multiples of 3, 5, and 7 in there, right?), but folks still seem to set rather aggressively small and low ranges for this answer.

If you don't know the answer, you can (and should) compensate with the size and even order of magnitude of your bounding range.

[Tman]

What I think is interesting is the default method for widening from a point estimate to a 90% confidence range.

Often, folks seem to think that if their point estimate is 100, than an adequate 90% confidence range is about 75-150 or so. In many cases, to hit 90% confidence, you need to be thinking in orders of magnitude, not on a more linear scale.

For instance, a lot of folks set a relatively small range for a question (number of prime numbers under 1000) that, apparently, few have much intuitive grasp of. Assuming you know what a prime number is, it's easy enough to scale down the upper end of the range to 500 (eliminate even numbers other than 2), and to bring it somewhat lower (there's gotta be a bunch of multiples of 3, 5, and 7 in there, right?), but folks still seem to set rather aggressively small and low ranges for this answer.

If you don't know the answer, you can (and should) compensate with the size and even order of magnitude of your bounding range.

The real question is do I feel less stupid by putting a range for prime numbers like 5-800?? I think Flowers is on right track, as the brain turmoil was pretty high when I knew I should have a better clue but didn't.

[Pogo]

1) 10,000-15,000 lb

2) 1510-1530 AD

3) 8,000-12,000 miles

4) 760-840 points

5) 3.5-5 million

6) 500-1000 miles

7) 25-35 homers

8) 100-200 threads

9) 140-160 countries

10) 6-12 primes

Bleh, I suck. I thought the homers was a season count, not career. And I'm confusing the industrial average with... something else.