\begin{center}
{{4. INTERVAL ESTIMATION}}
\end{center}

Interval estimation is more informative than
point estimation, and is very important in practice.

{\em  Confidence Interval} is the most commonly used
interval estimation.
More general type is  {\em Confidence Set} which may
consist of several intervals.


{\textbf  4.1 What is an interval estimator}

{\textbf Example 1}. Let us start with a simple example. A random
sample $X_1, \cdots, X_n$ are drawn from $N(\mu, 1)$. Then
\[
\sqrt n(\bar X -\mu) \sim N(0,1).
\]
Hence
\[
P( -1.96 \le \sqrt n(\bar X -\mu) \le 1.96) =0.95,
\]
or
\[
P( \bar X - 1.96/\sqrt n < \mu < \bar X + 1.96/\sqrt n) = 0.95.
\]
So a 95\% confidence interval for $\mu$ is
\[
(\bar X - 1.96/\sqrt n, \; \bar X + 1.96/\sqrt n).
\]

Suppose $n=4$, $\bar X = 2.25$. Then a 95\% C.I. is
$(2.25-0.98, 2.25+0.98) = (1.27, 3.23)$.

{\sl  Question}: what is $P(1.27 < \mu < 3.23)$? --- Note $\mu$ is a
unknown constant!

{\sl  Answer}: $(1.27, 3.23)$ is one realisation of the {\textbf random
interval} $(\bar X - 0.98, \; \bar X + 0.98)$ which covers $\mu$ with
probability 0.95.



{\textbf Definition}. Suppose the joint distribution of $\bX =(X_1,
\cdots, X_n)$ depends some parameter $\theta$.

If $\theta_l (\bX) $ and $\theta_u(\bX)$ are two statistics
for which
\[
P\{ \theta_l (\bX) < \theta < \theta_u(\bX) \} = 1-\alpha,
\]
$(\theta_l (\bX), \; \theta_u(\bX))$ is called a {\sl $100(1-\alpha)\%$
 confidence interval} for $\theta$.

{\textbf Remark}. (i) $1-\alpha$ is called the confidence level, which
is usually set at 0.95, 0.99 or 0.999. Naturally for given $\alpha$, we
shall search for the interval with the shortest length
$\theta_u(\bX)-\theta_l (\bX)$, which gives the \textbf{ most accurate}
estimation.

(ii) We may have $\theta_l (\bX) =-\infty$ or $\theta_u(\bX)=\infty$,
giving a one-sided interval.

(iii) In general we may use non-interval type set $S(\bX) \subset \Theta$
as the estimator for $\theta$, i.e.
\[
P\{ \theta \in S(\bX) \} = 1 -\alpha.
\]
We call $S(\bX)$ a $100(1-\alpha)\%$ {\textbf  confidence set}.



{\textbf  4.2  Method of pivotal functions}

\textbf{Set-up}: Observations -- $\bX$, Parameter of interest --
$\theta$ (possible other parameters).

{\textbf Definition}. A function of $\bX$ and $\theta$ alone is a {\sl
 pivotal function} if its distribution does not depend on
any unknown parameters {\small (i.e. can be calculated numerically)}.

{\textbf Examples of pivotal functions}: Let $X_1, \cdots, X_n$ be a
random sample.

1. for exponential distribution with
mean $\theta$, $\theta^{-1} \sum_j X_j$ is a pivotal,

2. for $N(\mu, 1)$,  $\bar X - \mu$ is a pivotal,

3. for $N(\mu, \sigma^2)$, $\sqrt n(\bar X - \mu)/S$ is a pivotal.



\textbf{Construction of C.I. based on a pivotal}:

Step 1: Find a pivotal function $T= T(\bX, \theta)$, and identify
its distribution

Step 2: Use  the distribution to identify a range of values $t_1, t_2$
such that
\[
P(t_1 < T < t_2) = 1-\alpha.
\]

Step 3: Manipulate the inequalities
\[
T>t_1 \quad {\rm and} \quad T< t_2
\]
to find a set of values for $\theta$. The values included in this set
form a 100$(1-\alpha)$\% C.I. for $\theta$.



{\textbf Example 2}. ({\sl  Student's $t$-interval})\\
 $(X_1, \cdots, X_n)$ is a random sample
from $N(\mu, \sigma^2)$.
A pivotal function is
\[
T = \sqrt n (\bar X -\mu) / S \sim t_{n-1},
\]
where $S^2 = {1 \over n-1} \sum_j (X_j - \bar X)^2$.

For $n=10$, we have
$P(-2.26 < T < 2.26) = 0.95$.
Note that $T> -2.26$ is equivalent to
\[
\mu < \bar X + 2.26S/\sqrt n.
\]
(Similar for $T< 2.26$.)
A 95\% C.I. for $\mu $ is $(\bar X - 2.26S/\sqrt n, \;
\bar X + 2.26S/\sqrt n)$.



{\textbf Note}. (i) In general it is not easy to identify a pivotal
function.

(ii) For location parameter $\mu$, $\bar X - \mu$ is \textbf{likely} to
be pivotal.

(iii) The MLE is asymptotically pivotal:
\[
\hat \theta \sim N(\theta, \calI^{-1}(\theta) ).
\]
Hence, $\sqrt{\calI(\theta)} ( \hat \theta - \theta)  \sim N(0,1)$ approximately
for large $n$.

{\textbf  4.3 Inverting Tests}

One easy way to get a confidence set is to invert a test; see
Example 1.

{\textbf Theorem}. Suppose that we have a size $\alpha$ test for the
Null Hypothesis $H_0:\;\theta=\theta_0$. For each $\theta_0\in\Theta$,
suppose the set $A(\theta_0)$ is the collection of $\bx$ for which
$H_0$ is {\sl not rejected}. Then the set
\[
S(\bx)=\{\theta :\bx\in A(\theta), \theta\in\Theta\}
\]
is a family of confidence sets for $\theta$ at confidence level $1-\alpha$.\\

The proof of this is obvious, because when $\theta$ is the true parameter value
\[
P_\theta[\theta\in S(\bX)] = P_\theta[\bX\in A(\theta)] = 1-\alpha.
\]

{\textbf Example 3}. Suppose that $X_1, X_2,\ldots ,X_{n}$ is a random
sample from an exponential distribution with mean $1/\lambda$.  The
UMPT with size 5\% for
\[
H_0:\;\lambda=\lambda_0 \quad {\rm against} \quad
H_1:\;\lambda >\lambda_0
\]
is to reject $H_0$ when $\sum X_i <
\frac{1}{2\lambda_0}\chi^2_{(2n),0.05}$, where $\chi^2_{(2n),0.05}$
is the lower 5\% point of the distribution $\chi^2_{2n}$.
The acceptance region here is $\sum X_i >
\frac{1}{2\lambda_0}\chi^2_{(2n),0.05}$,  which on inversion gives
\[
\left(
\frac{
      \chi^2_{(2n),0.05}}{ 2 \sum X_i}, \; \infty\right)
\]
as the confidence set for $\lambda$ at the 95\% level of confidence.\\



{\textbf  4.4 Bootstrap confidence intervals}


Let $X_1, \cdots, X_n$ be sample from unknown distribution
of $F$. We are interested in constructing confidence intervals
for some characteristic $\theta = \theta(F)$ of distribution $F$.

We adopt the so-called { \sl nonparametric bootstrap method} now. It
{draws a bootstrap sample
\[
X_1^*, \cdots, X_n^*
\]
independently from the uniform distribution over $n$ discrete
points $\{ X_1, \cdots, X_n \}$}.

Now in principle the distribution of any statistic based on $(X_1^*,
\cdots, X_n^*)$ is known, {\sl  conditionally on $(X_1, \cdots, X_n)$}.




{\textbf  4.4.1 Bootstrap percentiles}

Let $\wh \theta = T (X_1, \cdots, X_n)$ be an estimator
for $\theta$. Define
\[
\wh \theta^* = T( X_1^*, \cdots, X_n^*).
\]
Let $l^*_\alpha$ and $u^*_\alpha$ be, respectively,
 the lower and upper $\alpha$-point of
the distribution $\wh \theta^*$, i.e.
\[
P\{ \wh \theta^* \le l^*_\alpha | X_1, \cdots, X_n \} = \alpha,\]
\[
P\{ \wh \theta^* > u^*_\alpha | X_1, \cdots, X_n \} = \alpha.
\]

Then \textbf{ the $(1-\alpha)$100\%-th bootstrap interval} for $\theta$
is defined as $$( l^*_{\alpha/2},\; u^*_{\alpha/2}].$$


In practice we draw $B$ sets of bootstrap samples for some large $B$,
resulting in
 $B$ bootstrap estimates $\wh \theta^*_1, \cdots, \wh \theta^*_B$.
Then
$l^*_{\alpha/2}$ and $u^*_{\alpha/2}$ are,
respectively, the $[B\alpha/2]$-th smallest
and the $[B\alpha/2]$-th largest values among $\wh \theta^*_1,
\cdots, \wh \theta^*_B$.\\

{\textbf Note}.  Both $l^*_{\alpha/2}$ and $u^*_{\alpha/2}$ are
statistics depending on $\{ X_1, \cdots, X_n\}$.

Intuitively, we {\textbf  must} start with a {\sl good} estimator $\wh
\theta$. In fact, if $\wh \theta = T(X_1, \cdots, X_n)$ fulfils some
conditions, it holds that as $n\to \infty$,
\[
P( l^*_{\alpha/2} < \theta \le u^*_{\alpha/2} ) \to 1 - \alpha.
\]



{\textbf  4.4.2 Bootstrap-$t$}

Suppose we have a {\sl  studentised `pivot'}
\[
V = (\wh \theta - \theta)/ \wh \sigma,
\]
where $\wh \theta$ is an estimator for $\theta$, and $\wh \sigma^2$ is
{ an estimator for Var$(\wh \theta)$}. Define its bootstrap version
\[
V^* = (\wh \theta^* - \wh \theta)/ \wh \sigma^*.
\]
Let
\begin{eqnarray*}
\alpha/2 &=&
P\{ V^*\le  l^*_{\alpha/2}  |X_1, \cdots, X_n\}\\
&=&
 P\{ V^* > u^*_{\alpha/2} |X_1, \cdots, X_n\} .
\end{eqnarray*}

Then the 100$(1-\alpha)\%$-th bootstrap-$t$ interval for $\theta$ is
\[
[\wh \theta - \wh \sigma u^*_{\alpha/2}, \; \wh \theta - \wh \sigma l^*_{\alpha/2}).
\]

{\textbf  Remark}. The estimator $\wh \sigma^2$ may  be obtained by
bootstrap, then  $\wh \sigma^{2*}$ will be calculated via a {\textbf
 nested bootstrap}.




Ideally we require that the distribution, or at least the asymptotic
distribution,  of $V$ { does not depend on anything unknown}. This will
make the approximation
\[
P \big\{
\wh \theta - \wh \sigma u^*_{\alpha/2} \le \theta < \wh \theta - \wh \sigma l^*_{\alpha/2}
\big\} \approx \alpha
\]
more accurate.



{\small
We give a heuristic argument for the above assertion.

If $V$ is an asymptotic pivot in the sense that
\begin{equation} \label{41}
P(  V\le x ) = \Phi(x) + {1 \over \sqrt{n} }
g(x, \theta) + O({1 \over n }),
\end{equation}
similarly in the bootstrap world we have
\begin{eqnarray} \label{42}
\nonumber
&&
P(  V^*\le x| X_1, \cdots, X_n ) \;
=  \;\Phi(x) + {1 \over \sqrt{n} }
g(x, \wh \theta) + O_p({1 \over n })\\
&& \; \; = \; \Phi(x) + {1 \over \sqrt{n} }
g(x, \theta) + O_p({1 \over n }).
\end{eqnarray}
Hence
\[
l_{\alpha} = l_{\alpha}^* + O_p({1 \over n }).
\]

In contrast if $\Phi(x)$ depends on $\theta$, then $\Phi(x)$ should be
replaced by $\Phi(x, \theta)$ in (\ref{41}), and by $\Phi(x, \wh \theta)$ in
(\ref{42}). Now
\[
l_{\alpha} = l_{\alpha}^* + O_p({1 \over \sqrt{n} }).
\]
}
