I. Parameter Estimation

1 Statistical Modeling and Quality Criteria

• Statistical Estimation

The goal is to determine the probability distribution of the random variables based on avaliable samples. / The stochastic model is a set of probability spaces and the task of statistical estimation is to select the most appropriate candidate based on the observed outcomes of a random experiment.

• Statistical Model

In a statistical model, data is assumed to be generated by some underlying probability distribution, and the goal is to estimate the parameters of this distribution.

$\mathbb{X}$$\mathbb{X}$: a set of all possible observations

$\mathbb{F}$$\mathbb{F}$: a subset of $\mathbb{X}$$\mathbb{X}$ that satisfies certain properties. Here, $\mathbb{F}$$\mathbb{F}$ is the collection of events that can be assigned probabilities

$\mathrm{d}F(\cdot ;\theta )$$\mathrm{d}F(\cdot;\theta)$: a set of optional probability measures that assign probabilities to events in $\mathbb{F}$$\mathbb{F}$, they depend on parameter $\theta$$\theta$

$\theta$$\theta$: selects the probability measure $\mathrm{d}F(\cdot ;\theta )$$\mathrm{d}F(\cdot;\theta)$. They are unknown and need to be estimated based on the available data.

• Statistics

$X_i:\mathrm{\Omega }\to \mathbb{X}$$X_i: \Omega \rightarrow \mathbb{X}$, a random variable, also called a statistic, maps outcomes in the sample space $\mathrm{\Omega }$$\Omega$ to values in set $\mathbb{X}$$\mathbb{X}$, representing the values of interest that we want to study in our statistical analysis

$x_i\in \mathbb{X}$$x_i\in\mathbb{X}$: actual values that we observe or measure in our study

$T:\mathbb{X}\to \mathrm{\Theta }$$T:\mathbb{X}\rightarrow \Theta$, a special type of statistic, called an estimator, maps observations $x_i\in \mathbb{X}$$x_i\in\mathbb{X}$ to $\hat{\theta }\in \mathrm{\Theta }$$\hat{\theta}\in\Theta$

• $P_{\theta }(\cdot )$$P_{\theta}(\cdot)$ refers to the probability of an event according to the probability distribution selected by $\theta$$\theta$. The same holds for $E_{\theta }[\cdot ]$$E_{\theta}[\cdot]$ and ${\text{Var}}_{\theta }[\cdot ]$$\text{Var}_{\theta}[\cdot]$.

1.1 Introductory Example: Estimating upper bound $\theta$$\theta$

Given $N$$N$ i.i.d. statistics $X_1,\dots ,X_N$$X_1,\dots,X_N$ of a uniformly distributed random variable $X\sim \mathcal{U}(0,\theta )$$X\sim \mathcal U (0,\theta)$ where $\theta$$\theta$ is deterministic but unknown. We have that $E[X]=\theta /2,Var[X]={\theta }^2/12$$E[X]=\theta/2,Var[X]=\theta^2/12$.

Solution 1: Use $E[X]=\theta /2$$E[X]=\theta/2$

Solution 2: Intuition $\hat{\theta }=\underset{i=1,\dots ,N}{max}\{X_i\}$$\hat{\theta}=\max_{i=1,\dots,N}\{X_i\}$

1.2 Consistency and Unbiasedness

Consistent estimator $T$$T$: $\underset{N\to \mathrm{\infty }}{lim}T(x_1,\dots ,x_N)\to \theta$$\lim_{N\rightarrow\infin}T(x_1,\dots,x_N)\rightarrow \theta$

$T_1$$T_1$ and $T_2$$T_2$ are both consistent.

$T_1$$T_1$:

Using Chebyshev inequality, we derive the law of large numbers,

Using law of large numbers, we get

$T_2$$T_2$:

Unbiased estimator $T$$T$: $E[T(X_1,\dots ,X_N)]=\theta$$E[T(X_1,\dots,X_N)]=\theta$

• $T_1$$T_1$ is unbiased:

• $T_2$$T_2$ is asymptorically unbiased:

Proof. For $0\le \xi \le \theta$$0\leq \xi\leq\theta$,

• $T_2^{\prime }$$T_2'$ is unbiased:

1.3 Variance: $\text{Var}[T]=E[(T-E[T]{)}^2]$$\text{Var}[T]=E[(T-E[T])^2]$

A further quality measure for an estimator is its variance.

1.4 Mean Squared Error(MSE): $E[{\epsilon }^2(T)]=E[(T-\theta {)}^2]$$E[\varepsilon^2(T)]=E[(T-\theta)^2]$

An extension of the variance is the MSE(mean squared error), where $\theta$$\theta$ is the true parameter.

$T_1$$T_1$ and $T_2^{\prime }$$T_2'$ are unbiased: $E[T_1]=E[T_2^{\prime }]=\theta$$E[T_1]=E[T_2']=\theta$, therefore,

$T_2$$T_2$ is biased, its MSE is obtained by

1.5 Bias/Variance Trade-Off

MSE of an estimator $T$$T$ is defined as

and can be decomposed into its bias and variance

• Choose $\alpha$$\alpha$ to get optimal $\alpha T_2$$\alpha T_2$

Therefore, an unbiased estimator is not necessarily the optimal estimator, but for large $N$$N$, it is.

2 Maximum Likelihood Estimation

2.1 Maximum Likelihood Principle

The maximum likelihood principle suggests to select a candidate probability measure such that the observed outcomes of the experiment become most probable. A maximum likelihood estimator $T_{ML}$$T_{ML}$ picks the $\theta$$\theta$ that maximizes the likelihood function $L(x_1,\dots ,x_N|\theta )$$L(x_1,\dots,x_N|\theta)$.

The likelihood function depends on the statistical model, assuming all observations are iid, we obtain,

Normally, we use log-likelihood function,

• In the slides, $L(x;\theta )$$L(x;\theta)$ is used rather than $L(x|\theta )$$L(x|\theta)$.

2.2 Parameter Estimation

Channel Estimation

Consider an AWGN channel $y=hx+N$$y=hx+N$ with $N\sim \mathcal{N}(0,{\sigma }^2)$$N\sim \mathcal{N}(0,\sigma^2)$. We know that $Y\sim \mathcal{N}(hs,{\sigma }^2)$$Y\sim \mathcal{N}(hs,\sigma^2)$, and with $\theta =h$$\theta=h$, we have

Given $N$$N$ observations $y$$y$ and training signals $x$$x$, the maximum-likelihood estimation is

• The ML estimator is obviously identical with the least squares estimator, which changes drastically when the statistics $Y_i$$Y_i$ are correlated or when $N$$N$ is non-Gaussian distributed.

Introductory Example: Estimating upper bound $\theta$$\theta$

Suppose the distribution of observations is uniform, the likelihood function of $N$$N$ observations is

$1/{\theta }^N>0$$1/\theta^N>0$, therefore, to maximize the likelihood function,

$N$$N$ Bernoulli Experiments

Given $N$$N$ observations, the likelihood function of success number is

and $E[X]=N\theta ,Var[X]=N\theta (1-\theta )$$E[X]=N\theta,Var[X]=N\theta(1-\theta)$.

The ML-estimator is obtained by

In the following, we analyze the quality of $T_{ML}$$T_{ML}$

Since the estimator is unbiased, the MSE is equal to the variance of the estimator,

However, biased estimator can have less MSE and thus provide better estimates than unbiased estimator.

Alternative Solution:

2.3 Best Unbiased Estimator

ML estimators are not necessarily the best estimators. However, a wide class of estimators is defined by minimizing the MSE under an unbiasedness constraint.

We call an estimator $T$$T$ Best Unbiased Estimator if $E[T(X_1,\dots ,X_N)]=\theta$$E[T(X_1,\dots,X_N)]=\theta$ and

for any alternative unbiased estimator $S$$S$.

Best unbiased estimators are also referred to as UMVU(Uniformly Minimum Variance Unbiased) estimators.

3. Fisher's Information Inequality

An universal lower bound for the variance of an estimator can be introduced, if the following consition is fulfilled:

We define the score function as the slope of $\log L(x;\theta )$$\log L(x;\theta)$ with respect to $\theta$$\theta$:

3.1 Cramer-Rao Lower Bound

With $E[g(X;{\theta }_{true})]=E[\frac{\partial }{\partial \theta }\log L(x;\theta )]=0$$E[g(X;\theta_{true})]=E[\frac{\partial}{\partial\theta}\log L(x;\theta)]=0$, the Fisher Information term is defined as

which can be interpreted as the negative mean curvature of the log-likelihood function at ${\theta }_{true}$$\theta_{true}$.

The variance of an estimator can be lower bounded by the Cramer-Rao lower bound

If $T$$T$ is unbiased, $E[T(X)]=\theta$$E[T(X)]=\theta$, then

Properties of the Fisher Information:

• $I_F(\theta )$$I_F(\theta)$ depends on given observations $x_1,\dots ,x_N$$x_1,\dots,x_N$ and the unknown parameter $\theta$$\theta$

• A large value of $I_F(\theta )$$I_F(\theta)$ corresponds to a strong curvature and more information in $x_1,\dots ,x_N$$x_1,\dots,x_N$. A small value of $I_F(\theta )$$I_F(\theta)$ corresponds to a weak curvature and little information in $x_1,\dots ,x_N$$x_1,\dots,x_N$.

• $I_F(\theta )$$I_F(\theta)$ is monotonically increasing with the number of independent observation $N$$N$ statistics.

3.2 Exponential Models

A exponential model is a statistical model with