\begin{center}
{\bf \colorbox{grey}{4. INTERVAL ESTIMATION}}
\end{center}

Interval estimation is more informative than
point estimation, and is very important in practice.

{\em  Confidence Interval} is the most commonly used
interval estimation. 
More general type is  {\em Confidence Set} which may
consist of several intervals.


{\bf \color{dblue} 4.1 What is an interval estimator}

{\bf Example 1}.
Let us start with a simple example.
A random sample
$X_1, \cdots, X_n$ are drawn from $N(\mu, 1)$.
Then
\[
\sqrt n(\bar X -\mu) \sim N(0,1).
\]
Hence
\[
P( -1.96 \le \sqrt n(\bar X -\mu) \le 1.96) =0.95,
\] 
or
\[
P( \bar X - 1.96/\sqrt n < \mu < \bar X + 1.96/\sqrt n) = 0.95.
\] 
So a 95\% confidence interval for $\mu$ is
\[ 
(\bar X - 1.96/\sqrt n, \; \bar X + 1.96/\sqrt n).
\]

Suppose $n=4$, $\bar X = 2.25$. Then a 95\% C.I. is
$(2.25-0.98, 2.25+0.98) = (1.27, 3.23)$.

{\sl \color{dred} Question}: what is $P(1.27 < \mu < 3.23)$? ---
Note $\mu$ is a unknown constant!

{\sl \color{dred} Answer}: $(1.27, 3.23)$ is one realisation of the
{\bf random interval} $(\bar X - 0.98, \; \bar X + 0.98)$ which
covers $\mu$ with probability 0.95.

\newpage

{\bf Definition}. Suppose the joint distribution of
$\bX =(X_1, \cdots, X_n)$ depends some parameter $\theta$.

If $\theta_l (\bX) $ and $\theta_u(\bX)$ are two statistics
for which
\[
P\{ \theta_l (\bX) < \theta < \theta_u(\bX) \} = 1-\alpha,
\]
$(\theta_l (\bX), \; \theta_u(\bX))$ is called a {\sl \color{dgreen}
$100(1-\alpha)\%$
 confidence interval} for $\theta$.

{\bf Remark}.
(i) $1-\alpha$ is called the confidence level, which is usually set
at 0.95, 0.99 or 0.999.
Naturally for given $\alpha$, we shall search for the interval with
the shortest length $\theta_u(\bX)-\theta_l (\bX)$, which gives
the \underline{\color{dblue} most accurate} estimation.

(ii) We may have $\theta_l (\bX) =-\infty$ or $\theta_u(\bX)=\infty$,
giving a one-sided interval.

(iii) In general we may use non-interval type set $S(\bX) \subset \Theta$
as the estimator for $\theta$, i.e.
\[
P\{ \theta \in S(\bX) \} = 1 -\alpha.
\]
We call $S(\bX)$ a $100(1-\alpha)\%$ {\bf \color{dred} confidence set}.

\newpage

{\bf \color{dblue} 4.2  Method of pivotal functions}

\underline{Set-up}: Observations -- $\bX$, Parameter of interest -- $\theta$
(possible other parameters).

{\bf Definition}. A function of $\bX$ and $\theta$ alone is a
{\sl \color{blue} pivotal function} if its distribution does not depend on
any unknown parameters {\small (i.e. can be calculated numerically)}.

{\bf \color{dmagenta} Examples of pivotal functions}: Let $X_1, \cdots, X_n$
be a random sample.

1. for exponential distribution with
mean $\theta$, $\theta^{-1} \sum_j X_j$ is a pivotal,

2. for $N(\mu, 1)$,  $\bar X - \mu$ is a pivotal,

3. for $N(\mu, \sigma^2)$, $\sqrt n(\bar X - \mu)/S$ is a pivotal.


\newpage
\underline{\color{blue} Construction of C.I. based on a pivotal}:

Step 1: Find a pivotal function $T= T(\bX, \theta)$, and identify
its distribution

Step 2: Use  the distribution to identify a range of values $t_1, t_2$
such that
\[
P(t_1 < T < t_2) = 1-\alpha.
\]

Step 3: Manipulate the inequalities
\[
T>t_1 \quad {\rm and} \quad T< t_2
\]
to find a set of values for $\theta$. The values included in this set
form a 100$(1-\alpha)$\% C.I. for $\theta$.

\newpage

{\bf Example 2}. ({\sl \color{blue} Student's $t$-interval})\\ 
 $(X_1, \cdots, X_n)$ is a random sample
from $N(\mu, \sigma^2)$.
A pivotal function is
\[
T = \sqrt n (\bar X -\mu) / S \sim t_{n-1},
\]
where $S^2 = {1 \over n-1} \sum_j (X_j - \bar X)^2$.

For $n=10$, we have
$P(-2.26 < T < 2.26) = 0.95$.
Note that $T> -2.26$ is equivalent to
\[
\mu < \bar X + 2.26S/\sqrt n.
\]
(Similar for $T< 2.26$.)
A 95\% C.I. for $\mu $ is $(\bar X - 2.26S/\sqrt n, \;
\bar X + 2.26S/\sqrt n)$.

\newpage

{\bf Note}. (i) In general it is not easy to identify a pivotal function.

(ii) For location parameter $\mu$, $\bar X - \mu$ is \underline{likely} to
be pivotal.

(iii) The MLE is asymptotically pivotal:
\[
\hat \theta \sim N(\theta, \calI^{-1}(\theta) ).
\]
Hence, $\sqrt{\calI(\theta)} ( \hat \theta - \theta)  \sim N(0,1)$ approximately
for large $n$.

\newpage
{\bf \color{dblue} 4.3 Inverting Tests}

One easy way to get a confidence set is to invert a test; see 
Example 1.

{\bf Theorem}.
Suppose that we have a size $\alpha$ test for the Null Hypothesis
$H_0:\;\theta=\theta_0$. For each $\theta_0\in\Theta$, suppose the set
$A(\theta_0)$ is the collection of $\bx$ for which $H_0$ is {\sl not
rejected}. Then the set
\[
S(\bx)=\{\theta :\bx\in A(\theta), \theta\in\Theta\}
\]
is a family of confidence sets for $\theta$ at confidence level $1-\alpha$.\\

The proof of this is obvious, because when $\theta$ is the true parameter value
\[
P_\theta[\theta\in S(\bX)] = P_\theta[\bX\in A(\theta)] = 1-\alpha.
\]

{\bf Example 3}.
Suppose that $X_1, X_2,\ldots ,X_{n}$ is a random sample from an
exponential distribution with mean $1/\lambda$.  The UMPT with
size 5\% for
\[
H_0:\;\lambda=\lambda_0 \quad {\rm against} \quad
H_1:\;\lambda >\lambda_0
\]
is to reject $H_0$ when $\sum X_i <
\frac{1}{2\lambda_0}\chi^2_{(2n),0.05}$, where $\chi^2_{(2n),0.05}$
is the lower 5\% point of the distribution $\chi^2_{2n}$.
The acceptance region here is $\sum X_i >
\frac{1}{2\lambda_0}\chi^2_{(2n),0.05}$,  which on inversion gives
\[
\left(
\frac{
      \chi^2_{(2n),0.05}}{ 2 \sum X_i}, \; \infty\right)
\]
as the confidence set for $\lambda$ at the 95\% level of confidence.\\



\newpage
{\bf \color{dblue} 4.4 Bootstrap confidence intervals}


Let $X_1, \cdots, X_n$ be sample from unknown distribution 
of $F$. We are interested in constructing confidence intervals
for some characteristic $\theta = \theta(F)$ of distribution $F$.

We adopt the so-called {\color{blue} \sl nonparametric bootstrap
method} now. It {\color{dmagenta}draws a bootstrap sample 
\[
X_1^*, \cdots, X_n^*
\]
independently from the uniform distribution over $n$ discrete
points $\{ X_1, \cdots, X_n \}$}.

Now in principle the distribution of any statistic based on
$(X_1^*, \cdots, X_n^*)$ is known, {\sl \color{dgreen} conditionally on 
$(X_1, \cdots, X_n)$}.


\newpage

{\bf \color{dred} 4.4.1 Bootstrap percentiles}

Let $\wh \theta = T (X_1, \cdots, X_n)$ be an estimator
for $\theta$. Define
\[
\wh \theta^* = T( X_1^*, \cdots, X_n^*).
\]
Let $l^*_\alpha$ and $u^*_\alpha$ be, respectively,
 the lower and upper $\alpha$-point of
the distribution $\wh \theta^*$, i.e.
\[
P\{ \wh \theta^* \le l^*_\alpha | X_1, \cdots, X_n \} = \alpha,\]
\[
P\{ \wh \theta^* > u^*_\alpha | X_1, \cdots, X_n \} = \alpha.
\]

Then \underline{\color{blue}
the $(1-\alpha)$100\%-th bootstrap interval} for $\theta$ is defined
as $$( l^*_{\alpha/2},\; u^*_{\alpha/2}].$$

\newpage
In practice we draw $B$ sets of bootstrap samples for some large $B$,
resulting in
 $B$ bootstrap estimates $\wh \theta^*_1, \cdots, \wh \theta^*_B$.
Then
$l^*_{\alpha/2}$ and $u^*_{\alpha/2}$ are,
respectively, the $[B\alpha/2]$-th smallest
and the $[B\alpha/2]$-th largest values among $\wh \theta^*_1,
\cdots, \wh \theta^*_B$.\\

{\bf Note}.  Both $l^*_{\alpha/2}$ and $u^*_{\alpha/2}$ are statistics depending
on $\{ X_1, \cdots, X_n\}$.

Intuitively, we {\bf \color{dred} must} start with a {\sl 
\color{dred}good} estimator $\wh \theta$.
In fact, if $\wh \theta =
T(X_1, \cdots, X_n)$ fulfils some conditions, it holds 
that as $n\to \infty$,
\[
P( l^*_{\alpha/2} < \theta \le u^*_{\alpha/2} ) \to 1 - \alpha.
\]


\newpage

{\bf \color{dred} 4.4.2 Bootstrap-$t$}

Suppose we have a {\sl \color{blue} studentised `pivot'}
\[
V = (\wh \theta - \theta)/ \wh \sigma,
\]
where $\wh \theta$ is an estimator for $\theta$, and 
$\wh \sigma^2$ is {\color{dgreen} an estimator for
Var$(\wh \theta)$}. Define its bootstrap version
\[
V^* = (\wh \theta^* - \wh \theta)/ \wh \sigma^*.
\]
Let 
\begin{eqnarray*}
\alpha/2 &=&
P\{ V^*\le  l^*_{\alpha/2}  |X_1, \cdots, X_n\}\\
&=& 
 P\{ V^* > u^*_{\alpha/2} |X_1, \cdots, X_n\} .
\end{eqnarray*}

Then the 100$(1-\alpha)\%$-th bootstrap-$t$ interval for $\theta$ is
\[
[\wh \theta - \wh \sigma u^*_{\alpha/2}, \; \wh \theta - \wh \sigma l^*_{\alpha/2}).
\]

{\bf \color{dred} Remark}. The estimator $\wh \sigma^2$ may  be
obtained by  bootstrap, then  $\wh \sigma^{2*}$ will be calculated 
via a {\bf \color{dgreen} nested bootstrap}.


\newpage

Ideally we require that the distribution, or at least the 
asymptotic distribution,  of $V$ {\sl \color{blue}
does not depend on anything unknown}.
This will make the approximation
\[
P \big\{
\wh \theta - \wh \sigma u^*_{\alpha/2} \le \theta < \wh \theta - \wh \sigma l^*_{\alpha/2}
\big\} \approx \alpha
\]
more accurate.

\askip
\askip


{\small
We give a heuristic argument for the above assertion.

If $V$ is an asymptotic pivot in the sense that
\begin{equation} \label{41}
P(  V\le x ) = \Phi(x) + {1 \over \sqrt{n} }
g(x, \theta) + O({1 \over n }),
\end{equation}
similarly in the bootstrap world we have
\begin{eqnarray} \label{42}
\nonumber
&&
P(  V^*\le x| X_1, \cdots, X_n ) \; 
=  \;\Phi(x) + {1 \over \sqrt{n} } 
g(x, \wh \theta) + O_p({1 \over n })\\
&& \; \; = \; \Phi(x) + {1 \over \sqrt{n} } 
g(x, \theta) + O_p({1 \over n }).
\end{eqnarray}
Hence 
\[
l_{\alpha} = l_{\alpha}^* + O_p({1 \over n }).
\]

In contrast if $\Phi(x)$ depends on $\theta$, then $\Phi(x)$ should be
replaced by $\Phi(x, \theta)$ in (\ref{41}), and by $\Phi(x, \wh \theta)$ in
(\ref{42}). Now
\[
l_{\alpha} = l_{\alpha}^* + O_p({1 \over \sqrt{n} }).
\]
}



