BLUE estimator

by

Why BLUE :

We have discussed Minimum Variance Unbiased Estimator (MVUE)   in one of the previous articles. Following points should be considered when applying MVUE to an estimation problem

  • MVUE is the optimal estimator
  • Finding a MVUE requires full knowledge of PDF (Probability Density Function) of the underlying process.
  • Even if the PDF is known, finding an MVUE is not guaranteed.
  • If PDF is unknown, it is impossible find an MVUE using techniques like Cramer Rao Lower Bound (CRLB)
  • In practice, knowledge of PDF of the underlying process is actually unknown.

Considering all the points above, the best possible solution is to resort to finding a sub-optimal estimator. When we resort to find a sub-optimal estimator

  • We may not be sure how much performance we have lost – Since we will not able to find the MVUE estimator for bench marking (due to non-availability of underlying PDF of the process).
  • We can live with it, if the variance of the sub-optimal estimator is well with in specification limits

Common Approach for finding sub-optimal Estimator:

  • Restrict the estimator to be linear in data
  • Find the linear estimator that is unbiased and has minimum variance
  • This leads to Best Linear Unbiased Estimator (BLUE)
  • To find a BLUE estimator, full knowledge of PDF is not needed. Just the first two moments (mean and variance) of the PDF is sufficient for finding the BLUE

Definition of BLUE:

Consider a data set x[n]= \{ x[0],x[1], \cdots ,x[N-1] \} whose parameterized PDF p(x ;\theta) depends on the unknown parameter \theta. As the BLUE restricts the estimator to be linear in data, the estimate of the parameter can be written as linear combination of data samples with some weights a_n

\hat{\theta} = \displaystyle{\sum_{n=0}^{N} a_n x[n] = \textbf{a}^T \textbf{x}}  \quad\quad \rightarrow (1)

Here \textbf{a} is a vector of constants whose value we seek to find in order to meet the design specifications. Thus, the entire estimation problem boils down to finding the vector of constants – \textbf{a} . The above equation may lead to multiple solutions for the vector \textbf{a} .  However, we need to choose those set of values of  \textbf{a} , that provides estimates that are unbiased and has minimum variance.

Thus seeking the set of values for  \textbf{a} for finding a BLUE estimator that provides minimum variance, must satisfy the following two constraints

  1. The estimator must be linear in data
  2. Estimate must be unbiased

Constraint 1: Linearity Constraint:

Linearity constraint was already given above. Just repeated here for convenience.

\hat{\theta} = \displaystyle{\sum_{n=0}^{N} a_n x[n] = \textbf{a}^T \textbf{x}}  \quad\quad (1)

Constraint 2: Constraint for unbiased estimates:

For the estimate to be considered unbiased, the expectation (mean) of the estimate must be equal to the true value of the estimate.

E[\hat{\theta}] = \theta \quad\quad (2)

Thus,

\displaystyle{\sum_{n=0}^{N} a_n E \left( x[n] \right) = \theta} \quad\quad (3)

Combining both the constraints  (1) and (2) or  (3),

E[\hat{\theta}] =\displaystyle{\sum_{n=0}^{N} a_n E \left( x[n] \right)  = \textbf{a}^T \textbf{x}  = \theta} \quad\quad (4) 

Now, the million dollor question is : “When can we meet both the constraints ? “. We can meet both the constraints only when the observation is linear. That is x[n] is of the form x[n]=s[n] \theta , where \theta is the unknown parameter that we wish to estimate.

Consider a data model, as shown below, where the observed samples are in linear form with respect to the parameter to be estimated.

x[n] = s[n] \theta + w[n]  \quad\quad (5)

Here , w[n] is zero mean process noise , whose PDF can take any form (Uniform, Gaussian, Colored etc., ). The mean of the above equation is given by

E(x[n]) = E(s[n] \theta) = s[n] \theta  \quad\quad(6)

Substuiting (6) in (4) ,

E[\hat{\theta}] = \displaystyle{\sum_{n=0}^{N} a_n E \left( x[n] \right) = \theta \sum_{n=0}^{N} a_n s[n] = \theta \textbf{a}^T \textbf{s} = \theta} \quad\quad (7)

Looking at the last set of equality,

 \theta \textbf{a}^T \textbf{s}  = \theta \quad\quad(8)

The above equality can be satisfied only if

\textbf{a}^T \textbf{s} =1  \quad\quad (9)

Given this condition is met, the next step is to minimize the variance of the estimate. Minimizing the variance of the estimate,

\begin{aligned} var(\hat{\theta})&=E\left [ \left (\sum_{n=0}^{N}a_n x[n] - E\left [\sum_{n=0}^{N}a_n x[n] \right ] \right )^2 \right ]\\ &=E\left [ \left ( \textbf{a}^T \textbf{x} - \textbf{a}^T E[\textbf{x}] \right )^2\right ]\\ &=E\left [ \left ( \textbf{a}^T \left [\textbf{x}- E(\textbf{x}) \right ] \right )^2\right ]\\ &=E\left [ \textbf{a}^T \left [\textbf{x}- E(\textbf{x}) \right ]\left [\textbf{x}- E(\textbf{x}) \right ]^T \textbf{a} \right ]\\ &=E\left [ \textbf{a}^T \textbf{C} \textbf{a} \right ]\\ &=\textbf{a}^T \textbf{C} \textbf{a} \end{aligned} \quad\quad (10)

Finding BLUE:

As discussed above, in order to find a BLUE estimator for a given set of data, two constraints – linearity & unbiased estimates – must be satisfied and the variance of the estimate should be minimum. Thus the goal is to minimize the variance of \hat{\theta} which is \textbf{a}^T \textbf{C}\textbf{a} subject to the constraint \textbf{a}^T \textbf{s} =1 . This is a typical Lagrangian Multiplier↗ problem, which can be considered as minimizing the following equation with respect to  \textbf{a} (Remember !!!  this is what we would like to find ).

J = \textbf{a}^T \textbf{C} \textbf{a} + \lambda(\textbf{a}^T \textbf{s} -1)  \quad\quad (11)

Minimizing J with respect to \textbf{a} is equivalent to setting the first derivative of J w.r.t \textbf{a}to zero.

\begin{aligned} \frac{\partial J}{\partial \textbf{a}} &= 2\textbf{C}\textbf{a} + \lambda \textbf{s}=0 \ & \Rightarrow \boxed {\textbf{a}=-\frac{\lambda}{2}\textbf{C}^{-1}\textbf{s}} \end{aligned} \quad\quad (12) 

Substituting (12) in (9)

 \textbf{a}^T \textbf{s} = -\frac{\lambda}{2}\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}=1 \Rightarrow  \boxed {-\frac{\lambda}{2}=\frac{1}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}}}  \quad\quad (13) 

Finally, from (12) and (13), the co-effs of the BLUE estimator (vector of constants that weights the data samples) is given by

\boxed{a = \frac{\textbf{C}^{-1}\textbf{s}}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}}} \quad\quad\quad (14) 

The BLUE estimate and the variance of the estimates are as follows

\boxed{ \hat{\theta}_{BLUE} =\textbf{a}^{T} \textbf{x} = \frac{\textbf{C}^{-1}\textbf{s} \textbf{x}}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}}}   \quad\quad\quad (15) 

\boxed {var(\hat{\theta})= \frac{1}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}} }  \quad\quad\quad (16) 

Rate this article: PoorBelow averageAverageGoodExcellent (37 votes, average: 4.08 out of 5)

Similar topics

[1]An Introduction to Estimation Theory
[2]Bias of an Estimator
[3]Minimum Variance Unbiased Estimators (MVUE)
[4]Maximum Likelihood Estimation
[5]Maximum Likelihood Decoding
[6]Probability and Random Process
[7]Likelihood Function and Maximum Likelihood Estimation (MLE)
[8]Score, Fisher Information and Estimator Sensitivity
[9]Introduction to Cramer Rao Lower Bound (CRLB)
[10]Cramer Rao Lower Bound for Scalar Parameter Estimation
[11]Applying Cramer Rao Lower Bound (CRLB) to find a Minimum Variance Unbiased Estimator (MVUE)
[12]Efficient Estimators and CRLB
[13]Cramer Rao Lower Bound for Phase Estimation
[14]Normalized CRLB - an alternate form of CRLB and its relation to estimator sensitivity
[15]Cramer Rao Lower Bound (CRLB) for Vector Parameter Estimation
[16]The Mean Square Error – Why do we use it for estimation problems
[17]How to estimate unknown parameters using Ordinary Least Squares (OLS)
[18]Essential Preliminary Matrix Algebra for Signal Processing
[19]Why Cholesky Decomposition ? A sample case:
[20]Tests for Positive Definiteness of a Matrix
[21]Solving a Triangular Matrix using Forward & Backward Substitution
[22]Cholesky Factorization - Matlab and Python
[23]LTI system models for random signals – AR, MA and ARMA models
[24]Comparing AR and ARMA model - minimization of squared error
[25]Yule Walker Estimation
[26]AutoCorrelation (Correlogram) and persistence – Time series analysis
[27]Linear Models - Least Squares Estimator (LSE)
[28]Best Linear Unbiased Estimator (BLUE)

Books by the author

Wireless Communication Systems in Matlab
Wireless Communication Systems in Matlab
Second Edition(PDF)
PoorBelow averageAverageGoodExcellent (180 votes, average: 3.62 out of 5)

Digital modulations using Python
Digital Modulations using Python
(PDF ebook)
PoorBelow averageAverageGoodExcellent (134 votes, average: 3.56 out of 5)

digital_modulations_using_matlab_book_cover
Digital Modulations using Matlab
(PDF ebook)
PoorBelow averageAverageGoodExcellent (136 votes, average: 3.63 out of 5)
Hand-picked Best books on Communication Engineering
Best books on Signal Processing

Post your valuable comments !!!