I. Parameter Estimation

1 Statistical Modeling and Quality Criteria

• Statistical Estimation

The goal is to determine the probability distribution of the random variables based on avaliable samples. / The stochastic model is a set of probability spaces and the task of statistical estimation is to select the most appropriate candidate based on the observed outcomes of a random experiment.

• Statistical Model

In a statistical model, data is assumed to be generated by some underlying probability distribution, and the goal is to estimate the parameters of this distribution.

$\mathbb{X}$$\mathbb{X}$: a set of all possible observations

$\mathbb{F}$$\mathbb{F}$: a subset of $\mathbb{X}$$\mathbb{X}$ that satisfies certain properties. Here, $\mathbb{F}$$\mathbb{F}$ is the collection of events that can be assigned probabilities

$\mathrm{d}F(\cdot ;\theta )$$\mathrm{d}F(\cdot;\theta)$: a set of optional probability measures that assign probabilities to events in $\mathbb{F}$$\mathbb{F}$, they depend on parameter $\theta$$\theta$

$\theta$$\theta$: selects the probability measure $\mathrm{d}F(\cdot ;\theta )$$\mathrm{d}F(\cdot;\theta)$. They are unknown and need to be estimated based on the available data.

• Statistics

$X_i:\mathrm{\Omega }\to \mathbb{X}$$X_i: \Omega \rightarrow \mathbb{X}$, a random variable, also called a statistic, maps outcomes in the sample space $\mathrm{\Omega }$$\Omega$ to values in set $\mathbb{X}$$\mathbb{X}$, representing the values of interest that we want to study in our statistical analysis

$x_i\in \mathbb{X}$$x_i\in\mathbb{X}$: actual values that we observe or measure in our study

$T:\mathbb{X}\to \mathrm{\Theta }$$T:\mathbb{X}\rightarrow \Theta$, a special type of statistic, called an estimator, maps observations $x_i\in \mathbb{X}$$x_i\in\mathbb{X}$ to $\hat{\theta }\in \mathrm{\Theta }$$\hat{\theta}\in\Theta$

• $P_{\theta }(\cdot )$$P_{\theta}(\cdot)$ refers to the probability of an event according to the probability distribution selected by $\theta$$\theta$. The same holds for $E_{\theta }[\cdot ]$$E_{\theta}[\cdot]$ and ${\text{Var}}_{\theta }[\cdot ]$$\text{Var}_{\theta}[\cdot]$.

1.1 Introductory Example: Estimating upper bound $\theta$$\theta$

Given $N$$N$ i.i.d. statistics $X_1,\dots ,X_N$$X_1,\dots,X_N$ of a uniformly distributed random variable $X\sim \mathcal{U}(0,\theta )$$X\sim \mathcal U (0,\theta)$ where $\theta$$\theta$ is deterministic but unknown. We have that $E[X]=\theta /2,Var[X]={\theta }^2/12$$E[X]=\theta/2,Var[X]=\theta^2/12$.

Solution 1: Use $E[X]=\theta /2$$E[X]=\theta/2$

Solution 2: Intuition $\hat{\theta }=\underset{i=1,\dots ,N}{max}\{X_i\}$$\hat{\theta}=\max_{i=1,\dots,N}\{X_i\}$

1.2 Consistency and Unbiasedness

Consistent estimator $T$$T$: $\underset{N\to \mathrm{\infty }}{lim}T(x_1,\dots ,x_N)\to \theta$$\lim_{N\rightarrow\infin}T(x_1,\dots,x_N)\rightarrow \theta$

$T_1$$T_1$ and $T_2$$T_2$ are both consistent.

$T_1$$T_1$:

Using Chebyshev inequality, we derive the law of large numbers,

Using law of large numbers, we get

$T_2$$T_2$:

Unbiased estimator $T$$T$: $E[T(X_1,\dots ,X_N)]=\theta$$E[T(X_1,\dots,X_N)]=\theta$

• $T_1$$T_1$ is unbiased:

• $T_2$$T_2$ is asymptorically unbiased:

Proof. For $0\le \xi \le \theta$$0\leq \xi\leq\theta$,

• $T_2^{\prime }$$T_2'$ is unbiased:

1.3 Variance: $\text{Var}[T]=E[(T-E[T]{)}^2]$$\text{Var}[T]=E[(T-E[T])^2]$

A further quality measure for an estimator is its variance.

1.4 Mean Squared Error(MSE): $E[{\epsilon }^2(T)]=E[(T-\theta {)}^2]$$E[\varepsilon^2(T)]=E[(T-\theta)^2]$

An extension of the variance is the MSE(mean squared error), where $\theta$$\theta$ is the true parameter.

$T_1$$T_1$ and $T_2^{\prime }$$T_2'$ are unbiased: $E[T_1]=E[T_2^{\prime }]=\theta$$E[T_1]=E[T_2']=\theta$, therefore,

$T_2$$T_2$ is biased, its MSE is obtained by

1.5 Bias/Variance Trade-Off

MSE of an estimator $T$$T$ is defined as

and can be decomposed into its bias and variance

• Choose $\alpha$$\alpha$ to get optimal $\alpha T_2$$\alpha T_2$

Therefore, an unbiased estimator is not necessarily the optimal estimator, but for large $N$$N$, it is.

2 Maximum Likelihood Estimation

2.1 Maximum Likelihood Principle

The maximum likelihood principle suggests to select a candidate probability measure such that the observed outcomes of the experiment become most probable. A maximum likelihood estimator $T_{ML}$$T_{ML}$ picks the $\theta$$\theta$ that maximizes the likelihood function $L(x_1,\dots ,x_N|\theta )$$L(x_1,\dots,x_N|\theta)$.

The likelihood function depends on the statistical model, assuming all observations are iid, we obtain,

Normally, we use log-likelihood function,

• In the slides, $L(x;\theta )$$L(x;\theta)$ is used rather than $L(x|\theta )$$L(x|\theta)$.

2.2 Parameter Estimation

Channel Estimation

Consider an AWGN channel $y=hx+N$$y=hx+N$ with $N\sim \mathcal{N}(0,{\sigma }^2)$$N\sim \mathcal{N}(0,\sigma^2)$. We know that $Y\sim \mathcal{N}(hs,{\sigma }^2)$$Y\sim \mathcal{N}(hs,\sigma^2)$, and with $\theta =h$$\theta=h$, we have

Given $N$$N$ observations $y$$y$ and training signals $x$$x$, the maximum-likelihood estimation is

• The ML estimator is obviously identical with the least squares estimator, which changes drastically when the statistics $Y_i$$Y_i$ are correlated or when $N$$N$ is non-Gaussian distributed.

Introductory Example: Estimating upper bound $\theta$$\theta$

Suppose the distribution of observations is uniform, the likelihood function of $N$$N$ observations is

$1/{\theta }^N>0$$1/\theta^N>0$, therefore, to maximize the likelihood function,

$N$$N$ Bernoulli Experiments

Given $N$$N$ observations, the likelihood function of success number is

and $E[X]=N\theta ,Var[X]=N\theta (1-\theta )$$E[X]=N\theta,Var[X]=N\theta(1-\theta)$.

The ML-estimator is obtained by

In the following, we analyze the quality of $T_{ML}$$T_{ML}$

Since the estimator is unbiased, the MSE is equal to the variance of the estimator,

However, biased estimator can have less MSE and thus provide better estimates than unbiased estimator.

Alternative Solution:

2.3 Best Unbiased Estimator

ML estimators are not necessarily the best estimators. However, a wide class of estimators is defined by minimizing the MSE under an unbiasedness constraint.

We call an estimator $T$$T$ Best Unbiased Estimator if $E[T(X_1,\dots ,X_N)]=\theta$$E[T(X_1,\dots,X_N)]=\theta$ and

for any alternative unbiased estimator $S$$S$.

Best unbiased estimators are also referred to as UMVU(Uniformly Minimum Variance Unbiased) estimators.

3. Fisher's Information Inequality

An universal lower bound for the variance of an estimator can be introduced, if the following consition is fulfilled:

We define the score function as the slope of $\log L(x;\theta )$$\log L(x;\theta)$ with respect to $\theta$$\theta$:

3.1 Cramer-Rao Lower Bound

With $E[g(X;{\theta }_{true})]=E[\frac{\partial }{\partial \theta }\log L(x;\theta )]=0$$E[g(X;\theta_{true})]=E[\frac{\partial}{\partial\theta}\log L(x;\theta)]=0$, the Fisher Information term is defined as

which can be interpreted as the negative mean curvature of the log-likelihood function at ${\theta }_{true}$$\theta_{true}$.

The variance of an estimator can be lower bounded by the Cramer-Rao lower bound

If $T$$T$ is unbiased, $E[T(X)]=\theta$$E[T(X)]=\theta$, then

Properties of the Fisher Information:

• $I_F(\theta )$$I_F(\theta)$ depends on given observations $x_1,\dots ,x_N$$x_1,\dots,x_N$ and the unknown parameter $\theta$$\theta$

• A large value of $I_F(\theta )$$I_F(\theta)$ corresponds to a strong curvature and more information in $x_1,\dots ,x_N$$x_1,\dots,x_N$. A small value of $I_F(\theta )$$I_F(\theta)$ corresponds to a weak curvature and little information in $x_1,\dots ,x_N$$x_1,\dots,x_N$.

• $I_F(\theta )$$I_F(\theta)$ is monotonically increasing with the number of independent observation $N$$N$ statistics.

3.2 Exponential Models

A exponential model is a statistical model with

$b_{\theta }$$b_{\theta}$ is for normalization. Then the respective Fisher Information can be directly obtained by

If $f_X(x;\theta )$$f_X(x;\theta)$ can be further arranged such that $E[t(X)]=\theta$$E[t(X)]=\theta$, the unbiased estimator can directly be obtained as $T=t(X)$$T=t(X)$.

Mean Estimation Example

Consider the estimation of the unkown mean value $\theta =\mu$$\theta=\mu$ of $X\sim \mathcal{N}(\mu ,{\sigma }^2)$$X\sim\mathcal{N}(\mu,\sigma^2)$ based on a single observation.

Since $E[X]=\mu =\theta$$E[X]=\mu=\theta$, the single UMVU estimator is $T(x)=x$$T(x)=x$, which minimizes the variance, i.e., the MSE and gets

3.3 Asymptotically Efficient Estimators

An estimator is asymptotically efficient if (convergence in distribution)

• A ML estimator is asymptotically efficient if …

II. Examples

4. ML Principle for Direction of Arrival Estimation

4.1 Signal Model and ML Estimation

We consider the estimation of the Direction Of Arrival(DoA) $\theta$$\theta$ of an impinging planar waveform by means of an attenna array with $M$$M$ arrays and distance interval $d$$d$. The signal vector at time instant $t$$t$ is

Then the signal at the $m$$m$-th antenna is

where $\lambda$$\lambda$ is the wavelength of the assumed narrowband signal. The signal is assumed to be corrupted by the Gaussian noise vector $\mathbf{\text{N}}(t)\sim \mathcal{N}(\mathbf{\text{0}},C_{\mathbf{\text{N}}})$$\textbf{N}(t)\sim\mathcal{N}(\textbf{0},C_\textbf{N})$. $\xi$$\xi$ represents the attenuation which the transmitted signal $s(t)$$s(t)$ experiences over the transmission path.

Furthermore, we assume a Uniform Linear Array(ULA) with $M$$M$ antenna elements, i.e.,

In the case of a single observation and AWGN $\mathbf{\text{N}}(t)\sim \mathcal{N}(\mathbf{\text{0}},{\sigma }_N^2\mathbf{\text{I}})$$\textbf{N}(t)\sim\mathcal{N}(\textbf{0},\sigma^2_N\textbf{I})$, we have

! ${\alpha }^{\ast }\alpha =1\Rightarrow {\mathbf{\text{a}}}^H\mathbf{\text{a}}=M-1$$\alpha^*\alpha=1\Rightarrow \textbf{a}^H\textbf{a}=M-1$.

4.2 Cramer-Rao Bound for DOA Estimation

The likelihood function of the given estimation problem obviously belongs to the family of exponential distributions, where $\theta$$\theta$ is the parameter to be estimated.

III. Estimation of Random Variables

5. Bayesian Estimation

In the previous ML principle, we don't have any statistical information about the parameter $\theta$$\theta$, and only use the likelihood function to decide $\hat{\theta }$$\hat{\theta}$.

The Bayesian Estimation Method is based on a specific Statistical Model for the unknown parameter, here, it is $\theta$$\theta$. So we assume the statistical information about $\theta$$\theta$, consequently, $\theta$$\theta$ is now considered as a random variable, i.e.,

where $\sigma$$\sigma$ is the parameter for the statistical model of R.V. $\mathrm{\Theta }$$\Theta$, for example, a specific PDF.

Furthermore, instead of using notation $f_X(x;\theta )$$f_X(x;\theta)$, we have to use the conditional PDF, because $f_X$$f_X$ can only be obtained after $\mathrm{\Theta }$$\Theta$ is realized and a certain value of $\theta$$\theta$ is used

Now, the MSE of estimator $T(x)$$T(x)$ is calculated with respect to both random variables $X$$X$ and $\mathrm{\Theta }$$\Theta$,

5.1 Conditional Mean Estimator/Bayes Estimator – Minimizing the MSE

Conditional mean estimator $T_{CM}$$T_{CM}$

• Theorem

The conditional mean estimator (Bayes estimator) $T_{CM}$$T_{CM}$ is MSE optimal, i.e., minimizes the MSE cost criterion:

• Alternative cost criterion

Although the mean MSE is the most popular cost criterion, other criteria have been proposed and applied. The mean modulus is defined as

The conditional median estimator minimizes it,

5.2 Binomial Experiment

The ML estimator is obtained as

Now we assume $f_{\mathrm{\Theta }}(\theta )=1$$f_{\Theta}(\theta)=1$ if $\theta \in [0,1]$$\theta\in[0,1]$, then the MSE estimator is

Note that

We get

5.3 Mean Estimation Example

We consider the estimation of the unknown mean value $\theta$$θ$ of Gaussian Distributed random variable

assuming $\mathrm{\Theta }\sim \mathcal{N}(m,{\sigma }_{\mathrm{\Theta }}^2)$$\Theta\sim\mathcal{N}(m,\sigma^2_{\Theta})$, given $N$$N$ observations $X_1,\dots ,X_N$$X_1,\dots,X_N$ taken from the joint conditional likelihood of iid observations

Conditional mean estimator (two steps needed):

• Computing conditional PDF

• Computing conditional mean

Discussion

• Given large $N$$N$ or small conditional variance ${\sigma }_{X|\mathrm{\Theta }=\theta }^2$$\sigma^2_{X|\Theta=\theta}$ or large variance ${\sigma }_{\mathrm{\Theta }}^2$$\sigma^2_{\Theta}$, it is recommended to rely on the ML estimator.

• Given small variance ${\sigma }_{\mathrm{\Theta }}^2$$\sigma^2_{\Theta}$ or large conditional variance ${\sigma }_{X|\mathrm{\Theta }=\theta }^2$$\sigma^2_{X|\Theta=\theta}$, it is recommended to rely on the mean value of $\mathrm{\Theta }$$\Theta$.

Minimum Mean Square Error

MSE is minimized at $T_{CM}$$T_{CM}$, and is given by

which can be obtained once we have the information about the joint distribution $(X,\theta {)}^{\mathsf{T}}$$(X,\theta)^\mathsf{T}$

5.4 Jointly Gaussian Random Variables – Multivariate Case

Given random vectors and the covariance matrix from the joint distribution:

the multivariate conditional mean estimator is obtained as:

The respective MMSE is equal to the trace of the conditional covariance matrix ${\mathbf{\text{C}}}_{\mathbf{\Theta }|\mathbf{\text{X}}}$$\textbf{C}_{\bold\Theta|\textbf{X}}$:

• Yes, the estimator in (5.48) is the solution of the integral in (5.34), where $\theta$$θ$ and $X$$X$ are substituted by the respective vectors. The derivation of the estimator is however not as straightforward than for the scalar case due to the different rules for operations of vectors and matrices instead of scalars.

  If you refer to the last paragraph on slide $75$$75$, you find the case for $N=1$$N=1$. There, if you consider $1/{\sigma }_X^2$$1/σ^2_X$ to be equal with $C_X^{-1}$$C^{−1}_X$, you certainly see the strong similarity to the Eq. (5.48).

  Silde $75$$75$: For $N=1$$N=1$, we obtain

  $$\begin{array}{}& \begin{array}{c}{T}_{CM}=\frac{{\sigma }_{\mathrm{\Theta }}^2\sum _{i=1}^N{x}_i+m{\sigma }_{X|\mathrm{\Theta }=\theta }^2}{(N{\sigma }_{\mathrm{\Theta }}^2+{\sigma }_{X|\mathrm{\Theta }=\theta }^2)}=\frac{{\sigma }_{\mathrm{\Theta }}^{2}x+m{\sigma }_{X|\mathrm{\Theta }=\theta }^2}{{\sigma }_{\mathrm{\Theta }}^2+{\sigma }_{X|\mathrm{\Theta }=\theta }^2}\\ =m+\frac{{\sigma }_{\mathrm{\Theta }}^2}{{\sigma }_{\mathrm{\Theta }}^2+{\sigma }_{X|\mathrm{\Theta }=\theta }^2}(x-m)\end{array}\end{array}$$

• Given Jointly Gaussian Random Variables $X$$X$ and $Y$$Y$, the Conditional Mean Estimator $E[Y|X]$$E [Y |X ]$ is a Linear Function in $X$$X$ ($=\rho X$$=\rho X$), and the marginal distribution of $X$$X$ is also Gaussian independent of the correlation coefficient. This does not hold for arbitrarily jointly distributed random variables!