Examples of Probability

What is the probability of a health-reform bill this year?

What is the probability of a health-reform bill with a public option this year?

What is the probability of a health-reform bill without a public option this year?

What is the probability of the senate finance committee approving a bill by then end of next week?

What is the probability of a health-reform bill this year if the senate finance committee approves a bill by next week?

What is the probability of a health-reform bill this year if the senate finance committee does not approve a bill by next week?

What is the probability of the finance committee having approved a bill by the end of next week if there is a health-reform bill this year?

Normative Theory of Probability

What is probability?
A numerical measure of the strength of a belief in a certain proposition: p(proposition).

Theories: frequency, logical, personal.

Rules of coherence: addition, multiplication, conditional probability, independence.

Theories

The Frequency Theory

The proportion of times it might have happened in the past that it actually did, e.g., p("Get run over crossing 38th street") = 
   (Number of times people got run over crossing 38th st.)
   -------------------------------------------------------
         (Number of times people crossed 38th st.)
But why this denominator?

Other theories

The Logical Theory

The proportion of all possible exchangeable (i.e., equally likely) worlds which entail our proposition of interest being true. E.g. Playing cards.

But how often can we apply this in the real world?

The Personal Theory

Probability is a subjective judgment based on all of the knowledge and beliefs you have. There is no objectively perfect way to determine the `correct' probability. Reasonable people can disagree, because they have different evidence available to them.

Examples:
Iowa Electronic Markets
Intrade
NWS
National Hurricane Center

Rules of coherence

p(A) + p(not A) = 1

(not A is called the "complement" of A)

rainingnot raining

e.g. "It is absolutely certain that either "It is raining" is true or "It is raining" is not true.

Additivity

Mutually Exclusive

Propositions A and B are "mutually exclusive" if they cannot both be true at the same time.

I.e., if one of the propositions is true, that "excludes" the possibility of the other being true: the two propositions "mutually exclude" each other.

When propositions A and B are mutually exclusive: p(A or B) = p(A) + p(B)

e.g. p(Bill) = p(BillPub) + p(BillNopub)

BillPubBillNopub

Where does additivity come from?

From betting. The expected value of a bet on an event is its probability times the amount to win. (Later we'll see that this works for "expected utility.")

For example, the expected value of "$10 if a coin comes up heads" is $5.

What is the EV of "$10 if a coin comes up heads twice (in 2 flips)"?

EV is (roughly) the average value if the bet were repeated.

EV and additivity

We can bet on two events at once.

The value of a bet on one event should not change when you break it into two events.

For example, the expected value of (and willingness to pay for)

$10 if "Bill"

should be the same as:

$10 if "BillPub"
and
$10 if "BillNopub"

Example, continued

Suppose you are willing to pay or accept $4 for $10 if "Bill"

$3 for $10 if "BillPub"

$2 for $10 if "BillNopub"

Someone could buy the first bet for $4, sell you the second and third, together, for $5, then trade the second and third for the first, since they are logically equivalent, and so on.

Your values must add up.

Conditional Probability Defined

1. Before we know.

Bill, Senfin   no Bill, Senfin
Bill, no Senfin   no Bill, no Senfin

2. After we know Senfin.

Bill, Senfin          no Bill, Senfin   

The symbol | means "given" or "given that we know that..."

The conditional probability of proposition A given proposition B is the probability that we would assign to A if we knew that B were true, that is, the probability of A conditional on B being true. We write p(A|B) or p(A/B). This does not mean "divided by".

Multiplication Rule - p(A and B)

p(Bill | Senfin) = p(Bill AND Senfin) / p(Senfin)

The Conditional Probability Rule is: p(A | B) = p(A and B) / p(B)

(This time the / does mean "divided by".)

In other words: p(A and B) / p(B) = p(A | B)

Then we multiply both sides by p(B) to get p (A and B) = p(A | B) × p(B), the multiplication rule.

Conditioned assessment (Kleinmuntz et al., 1996)

Estimate the probability of some event E.

Estimate the probability of E given some other event F, p(E|F)

Estimate p(E|not-F)

Estimate p(F)

Compute E' as p(F)×p(E|F) + p(not-F)×p(E|not-F)

Example of conditional assessment

p(m)=.5, p(C|m)=.4, p(C|f)=.1

p(C) = p(m)×p(C|m) + p(f)×p(C|f) = .5×.4 + .5×.1 = .25

Independent Propositions

Two propositions A and B are independent if believing that A is true does not change your belief about whether B is true. (The reverse always holds.)

Are "It is raining" and "Melissa is mowing the lawn" independent?

Are "The first car to pass us will be a Dodge" and "The second car to pass us will be a BMW" independent?

When A and B are independent, then the multiplication rule can be simplified, because p(A | B) = p(A). Hence: p(A and B) = p(A) × p(B).

Why the multiplication rule?

Your choice should not change if you narrow down the space of possibilities to what matters. Or if a choice is conditional on some event. Consider the choice of taking Option A or not.

Choice A. ($1 if Rain, p=.75), (-$3 if no rain, p=.25)
vs. nothing.

Choice B. Choice A if heads, nothing if tails.

Choice C. ($1 if Rain, p=.375), (-$3 if no rain, p=.125)
vs. nothing.
R SVG Plot! H T .5 R $1 N -$3 .75 .25

Bayes's Theorem as an equation

H = Hypothesis, "Teri has cancer"
D = Datum, e.g., "Mammogram result is positive"

When Dr. Ayes examines Teri, she assigns a prior probability of 10% to the hypothesis H: p(H) = 0.1. The prior probability is the probability that the Hypothesis is true, based on what we know prior to doing a test such as a mammogram.

In thinking about whether to do the mammogram, Dr. Ayes knows:

 p(H) = 0.1 The prior probability is 10%
 p (not-H) = 0.9 Because p(H) + p(not-H) = 1
 p(D | H) = 0.9 Test has a "hit rate" of 90%
 p(D | not H) = 0.2 Test has a "false alarm rate" of 20%

Dr. Ayes wants to know what the probability of Teri having cancer will be if the test is positive. In symbols, she wants to know:
p(H | D) - The posterior (after the test result) probability.

How do we turn the information she has into the information she wants?

Bayes's theorem, pictured

not-H H not-H & D H & D not-H & D H & D

p(not-H & D) = p(D | not-H) × p(not-H)

p(H & D) = p(D | H) × p(H)

Thus: p(H | D) = p(H & D) / [ p(not-H & D) + p(H & D)] =

            p(D|H)×p(H)
-----------------------------------------------
[ p(D|H)×p(H) + p(D|not-H)×p(not-H) ]

Another way to picture this

probability tree A long but good tutorial.

Deriving Bayes's Theorem

The Conditional Probability Rule tells us a way to calculate p(H | D):

(1) p(H | D) = p(H and D) / p(D)

But we don't know p(H and D) or p(D). Let's try to use what we do know to find out what they are. The Multiplication Rule is:

(2) p(D and H) = p(D | H) × p(H)

We can change (D and H) to (H and D), since they mean the same thing:

(3) p(H and D) = p(D | H) × p(H)

So now we can work out p(H and D) from two things we know: p(D | H) and p(H). If we use (3) to change (1), we can get a form of Bayes's theorem:

(4) p(H | D) = p(D | H) × p(H) / p(D)

Getting p(D)

(we have 4) p(H | D) = p(D | H) × p(H) / p(D)

Now we just need to get p(D). We can use the fact that D must happen either with H or with not-H:

(5) p(D) = p(D and H) + p(D and not-H)

Then we can use the Multiplication Rule again to replace the "and" terms:

(6) p(D) = p(D | H) × p(H) + p(D | not-H) × p(not-H)

By replacing "p(D)" in (4) with our expression from (6) we get:

(7)

                    p(D|H)×p(H)
p(H|D) = ---------------------------------------------
           [ p(D|H)×p(H) + p(D|not-H)×p(not-H) ]

Finishing the example

(we have 7)

                    p(D|H)×p(H)
p(H|D) = ---------------------------------------------
           [ p(D|H)×p(H) + p(D|not-H)×p(not-H) ]

So Dr. Ayes could use that formula to figure out whether a mammogram would be worthwhile. Putting in the numbers

                .9×.1          .09     .09
p(H | D) = --------------- = ------- = --- = .33
           [.9×.1 + .2×.9]   .09+.18   .27

Ratio form of Bayes's theorem

                    p(D | H) × p(H)
p(H | D) = ---------------------------------------------
           [ p(D | H) × p(H) + p(D | not-H) × p(not-H) ]

                         p(D | not-H) × p(not-H)
p(not-H | D) = ---------------------------------------------
               [ p(D | H) × p(H) + p(D | not-H) × p(not-H) ]

  p(H | D)         p(D | H) × p(H)          p(D | H)     p(H)
------------ = ----------------------- = ------------ × --------
p(not-H | D)   p(D | not-H) × p(not-H)   p(D | not-H)   p(not-H)

posterior odds = diagnostic ratio × prior odds

log(posterior odds) = log(diagnostic ratio) + log(prior odds)