Lecture 13: Exercises

a. Number the investments 1-4 and let binary $x_i = 1$ if investment $i$ is chosen. The ILO is

b. Compute the total return $R$ and budget spend $B$ of the chosen policy. Simulate a budget realization $x$ from $X$: if $x < B$ the simulated return is $R - 3$, if $x \ge B$ it is $R$. Repeat this simulation many times; the sample average of the simulated returns approximates the expected return.

c. The 95% confidence interval is

Since 0 lies in this interval, we cannot conclude with 95% confidence that one policy is significantly better than the other.

d. Run more simulations, or use common random numbers (the same simulated budget realizations for both policies) to reduce variance.

a. The 95% confidence interval is

Since 0 is not in this interval, we can conclude with 95% confidence that A is significantly different from (better than) B.

b. Scenario A has the larger sample standard deviation, so adding simulations there reduces the confidence interval width the most: with the extra 200 on A, the width becomes approximately $1.96\sqrt{\tfrac{8^2}{600} + \tfrac{6.5^2}{400}} \approx 0.90$; with them on B, approximately $1.96\sqrt{\tfrac{8^2}{400} + \tfrac{6.5^2}{600}} \approx 0.94$. So the extra simulations should go to A.

c. Test $H_0(B, A)$: B is not worse than A, and $H_0(C, A)$: C is not worse than A, both at significance level $\alpha = 1 - \sqrt{1 - 0.05} \approx 0.025$ (adjusted for testing two comparisons at once), whose corresponding critical value is approximately 1.96. If both null hypotheses are rejected, A is significantly better than both B and C.

Since $16.2 \le 17.3 - 1.96\sqrt{6.5^2 + 8^2}/\sqrt{400} = 16.28$, $H_0(B, A)$ is rejected. Since $15.7 \le 17.3 - 1.96\sqrt{12.4^2 + 8^2}/\sqrt{400} = 15.85$, $H_0(C, A)$ is also rejected. So, with 95% confidence, A is better than both B and C.

a. Let $X, X'$ be the two experiments’ outcomes, with $n = 100$ samples each, and let $Y = X - X'$. By the central limit theorem,

approximately, with $EY = EX - EX' \approx 4.5 - 3.6 = 0.9$ and, since $X$ and $X'$ are independent, $\sigma^2(Y) = \sigma^2(X) + \sigma^2(X') \approx 2.4^2 + 3.7^2 = 19.45$. So the difference of sample averages is approximately $N(0.9, 0.1945)$: mean 0.9, standard deviation $\approx 0.44$.

b. Using part a, the 95% confidence interval is

c. Test the three null hypotheses that experiment 2, 3, or 4 (respectively) has a higher outcome than experiment 1. With three simultaneous comparisons, adjust the confidence level to $\sqrt[3]{0.95} \approx 0.983$, with critical value approximately 2.12. All three null hypotheses must be rejected to conclude experiment 1 has the significantly highest outcome.

The first test asks whether $4.5 > 3.6 + 2.12\sqrt{2.4^2 + 3.7^2}/\sqrt{10}$; working out the right-hand side gives $4.5 > 4.53$, which is false, so this null hypothesis cannot be rejected already. We therefore cannot conclude that experiment 1 has the significantly highest outcome (the other two comparisons, $4.5 > 3.97$ and $4.5 > 3.76$, would in fact have been rejected, but that no longer matters once one comparison fails).

Rather than splitting the whole budget of 2000 evenly across all 21 candidate values of $\pi$, use a two-phase ranking-and-selection approach. In an orientation phase, simulate every candidate a smaller number of times (e.g. 30 each, spending $30 \times 21 = 630$ of the budget) and use a conservative $t$-test (accounting for the number of comparisons made) to discard candidates that are significantly worse than some other candidate. In a second phase, divide the remaining budget ($2000 - 630 = 1370$ simulations) over only the candidates that were not discarded.

Many problems have candidates that are clearly worse than the top few; identifying and discarding these early frees up simulation budget to more precisely compare the remaining, more promising candidates in the second phase. This concentrates statistical power where it matters, increasing the chance of correctly identifying the true best $\pi$ compared to spreading the budget uniformly over all 21 candidates from the start.

a. y holds the average profit $y(\pi)$ of each order size after the first round, s the sample standard deviation $s(\pi)$, and m0 the number of runs $m_0$ of every order size in the first round. With $k = 5$ order sizes, $\alpha^* = 1 - \sqrt[4]{0.95} \approx 1 - 0.9873 = 0.0127$.

b. beta is about 2.27, and since all standard deviations are equal, every comparison uses the same margin $2.27 \times 2.12 / \sqrt{100} \approx 0.48$. An order size is discarded if its average is at least 0.48 below the average of another order size. The highest average is 1.55 (order size 8), so order sizes with an average at most $1.55 - 0.48 = 1.07$ are discarded: 4 (0.98) and 12 (0.45). The code prints [6, 8, 10].

c. With equal standard deviations the margin is the same for every pair. If an order size is beaten by some other order size, so $y(\pi) \le y(\pi') - \text{margin}$, then it is certainly beaten by the order size with the highest average, because that average is at least $y(\pi')$. So comparing with that one order size gives the same result.

d. Remember the order size that beats it, and print it after the inner loop:

for order in y:
    discard = False
    for other in y:
        if other != order:
            margin = beta * np.sqrt(s[order] ** 2 + s[other] ** 2) / np.sqrt(m0)
            if y[order] <= y[other] - margin:
                discard = True
                beaten_by = other
    if discard:
        print(order, "is beaten by", beaten_by)
    else:
        survivors.append(order)

e. Replace the line beta = stats.t.ppf(1 - alpha_star, m0 - 1) by beta = stats.t.ppf(1 - alpha, m0 - 1). Then beta is about 1.66, the margin is $1.66 \times 2.12 / 10 \approx 0.35$, and order sizes with an average of at most $1.55 - 0.35 = 1.20$ are discarded: 4, 10 and 12. This is not a good idea because each order size is tested against four others: even if it is the best one, the probability that at least one of the four tests discards it by chance is then larger than 5%.

f. The first round used $k m_0 = 5 \times 100 = 500$ runs, so 900 runs are left:

budget = 1400
n_each = (budget - k * m0) // len(survivors)
print(n_each)

With three survivors, each gets 300 extra runs.