22: Statistics — Estimator Properties, Fisher Info, Cramér-Rao

📚 Նյութը

⚠️ Note

YouTube links in this section were auto-extracted. If you spot a mistake, please let me know!

Դասախոսություն

Գործնական

🏡 Տնային

1) Exponential Family & Sufficiency

01 Poisson Meets the Exponential Family

Let $X$ be a random variable following a Poisson distribution with parameter $\lambda > 0$, i.e., $P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \ldots$

1. Show that the Poisson distribution belongs to the exponential family by writing its PMF in the form \[f(x \mid \lambda) = h(x) \exp\!\big(\eta(\lambda)\, T(x) - A(\lambda)\big).\] Identify $h(x)$, $\eta(\lambda)$, $T(x)$, and $A(\lambda)$.
1. Using the exponential family form, what is the sufficient statistic for $\lambda$ based on an i.i.d. sample $X_1, \ldots, X_n$?

02 Slit Width Estimation

In an experiment, $n$ drops of solution are released uniformly through a slit onto a surface. We model the one-dimensional impact points $X_1, \ldots, X_n$ as i.i.d. $\mathrm{Uniform}(0, d)$, where the unknown slit width $d > 0$ is to be estimated.

1. Write down the joint density $f(\mathbf{x} \mid d)$ for the sample.
1. Using the Fisher–Neyman factorization theorem, show that $X_{(n)} = \max\{X_1, \ldots, X_n\}$ is sufficient for $d$.
1. Is $X_{(n)}$ unbiased for $d$? If not, find an unbiased estimator based on $X_{(n)}$.

Hint for (c): the CDF of $X_{(n)}$ is $F_{X_{(n)}}(x) = (x/d)^n$ for $0 \le x \le d$.

03 Normal Variance: Minimal Sufficiency

Let $X_1, \ldots, X_n \overset{\text{i.i.d.}}{\sim} N(\mu, \sigma^2)$ where $\sigma^2 > 0$ is unknown but $\mu$ is known.

1. Show that $T(\mathbf{X}) = \sum_{i=1}^{n}(X_i - \mu)^2$ is sufficient for $\sigma^2$ using the factorization theorem.
1. Using the likelihood ratio criterion, show that $T(\mathbf{X})$ is minimal sufficient for $\sigma^2$.

Recall: $T$ is minimal sufficient iff $T(\mathbf{x}) = T(\mathbf{y})$ $\Longleftrightarrow$ $\frac{f(\mathbf{x} \mid \sigma^2)}{f(\mathbf{y} \mid \sigma^2)}$ is free of $\sigma^2$.

04 Binomial Sufficiency and Estimating $\pi^2$

Let $\mathbf{X} = (X_1, \ldots, X_n)^\top$ with $X_i \overset{\text{i.i.d.}}{\sim} \mathrm{Bernoulli}(\pi)$, where $\pi \in (0, 1)$. Define $U(\mathbf{X}) = \sum_{i=1}^{n} X_i$.

1. Show that $U(\mathbf{X})/n$ is unbiased for $\pi$.
1. Show that $U(\mathbf{X})$ is minimal sufficient for $\pi$.
1. Now consider the estimator for $\pi^2$: \[V(\mathbf{X}) = \frac{U(\mathbf{X})\left[U(\mathbf{X}) - 1\right]}{n(n-1)}.\] Verify that $V(\mathbf{X})$ is unbiased for $\pi^2$.

Hint for (c): expand $\mathbb{E}[U(U-1)]$ using $\mathbb{E}[U] = n\pi$ and $\operatorname{Var}(U) = n\pi(1-\pi)$.

2) Fisher Information & Cram'{e}r–Rao

05 Fisher Information for the Exponential

Let $X_1, \ldots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$ with density $f(x \mid \lambda) = \lambda e^{-\lambda x}$ for $x \geq 0$.

1. Compute the score function $s(\lambda) = \frac{\partial}{\partial \lambda} \log f(X \mid \lambda)$.
1. Verify that $\mathbb{E}[s(\lambda)] = 0$.
1. Compute the Fisher information $I(\lambda) = \operatorname{Var}[s(\lambda)]$.
1. Verify your answer in (c) by computing $I(\lambda)$ via the second-derivative formula: $I(\lambda) = -\mathbb{E}\!\left[\frac{\partial^2}{\partial\lambda^2}\log f(X \mid \lambda)\right]$.
1. The MLE for $\lambda$ is $\hat{\lambda} = 1/\bar{X}$, which has $\operatorname{Var}(\hat{\lambda}) \approx \lambda^2/n$ for large $n$. Compare this with the Cram'{e}r–Rao lower bound. Is $\hat{\lambda}$ asymptotically efficient?

06 Cramér–Rao: When Can We Beat $1/n$?

Let $X_1, \ldots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Bernoulli}(p)$.

1. We know $I(p) = \frac{1}{p(1-p)}$. Write down the Cram'{e}r–Rao lower bound for any unbiased estimator of $p$.
1. The sample proportion $\hat{p} = \bar{X}$ has $\operatorname{Var}(\hat{p}) = \frac{p(1-p)}{n}$. Is $\hat{p}$ efficient?
1. Now consider estimating $g(p) = p^2$ instead of $p$. The Cram'{e}r–Rao bound for unbiased estimators of $g(\theta)$ is \[\operatorname{Var}(\hat{g}) \geq \frac{[g'(\theta)]^2}{n \cdot I(\theta)}.\] Compute the CR bound for unbiased estimators of $p^2$.
1. We showed in Problem 04(c) that $V(\mathbf{X}) = \frac{U(U-1)}{n(n-1)}$ is unbiased for $p^2$. Compute $\operatorname{Var}(V)$ (at least for large $n$). Does it achieve the CR bound?

3) Admissibility & Minimax

07 Admissibility: A Sketch Exercise

Suppose there exist exactly three estimators $T_1$, $T_2$, and $T_3$ for a parameter $\theta \in [0, 1]$.

1. Sketch an example of the MSE curves $\mathrm{MSE}(T_i, \theta)$ as functions of $\theta$ such that $T_1$ and $T_2$ are admissible, but $T_3$ is not admissible. Explain why your sketch works.
1. Now sketch (possibly different) risk functions for $T_1$, $T_2$, and $T_3$ such that $T_1$ is the minimax estimator. Must $T_1$ have the lowest MSE everywhere?

🎲 xx+37 (xx)

▶️ToDo
🔗Random link ToDo
🇦🇲🎶ToDo
🌐🎶ToDo
🤌Կարգին ToDo

--- title: "22: Statistics — Estimator Properties, Fisher Info, Cramér-Rao" format: html: css: homework-styles.css --- <script src="homework-scripts.js"></script> # 📚 Նյութը ::: {.callout-tip collapse="true"} ## ⚠️ Note YouTube links in this section were auto-extracted. If you spot a mistake, please let me know! ::: ## Դասախոսություն - [📺 Դասախոսություն — Estimators: Bias, MSE, Bias-Variance tradeoff](https://youtu.be/lAKPMjqQ6vc) - [📺 Դասախոսություն — Estimators: Consistency, sufficiency](https://youtu.be/Ye0ZsTDnPx4) - [📺 Դասախոսություն — Fisher information, Cramér-Rao, Minimax](https://youtu.be/DDdzHnQsyrA) - [🎞️ Սլայդեր — 03 stat](Lectures/stat/03_stat.pdf), [📝 Notes](Lectures/stat/03_stat_notes.pdf) - [🎞️ Սլայդեր — 04 stat](Lectures/stat/04_stat.pdf), [📝 Notes](Lectures/stat/04_stat_notes.pdf) ## Գործնական - [📺 Գործնական — Minimal sufficiency, Fisher-Neyman theorem](https://youtu.be/fxYxoHIz-P0) - [🛠️🗂️ Գործնականի PDF-ը](Homeworks/hw_24_stat_1_estimator_properties.pdf) # 🏡 Տնային --- ## 1) Exponential Family & Sufficiency ### 01 Poisson Meets the Exponential Family {data-difficulty="1"} Let $X$ be a random variable following a Poisson distribution with parameter $\lambda > 0$, i.e., $P(X = k) = \frac{\lambda^k e^{-\lambda}}{k!}$ for $k = 0, 1, 2, \ldots$ - a) Show that the Poisson distribution belongs to the exponential family by writing its PMF in the form $$f(x \mid \lambda) = h(x) \exp\!\big(\eta(\lambda)\, T(x) - A(\lambda)\big).$$ Identify $h(x)$, $\eta(\lambda)$, $T(x)$, and $A(\lambda)$. - b) Using the exponential family form, what is the sufficient statistic for $\lambda$ based on an i.i.d. sample $X_1, \ldots, X_n$? ### 02 Slit Width Estimation {data-difficulty="2"} In an experiment, $n$ drops of solution are released uniformly through a slit onto a surface. We model the one-dimensional impact points $X_1, \ldots, X_n$ as i.i.d. $\mathrm{Uniform}(0, d)$, where the unknown slit width $d > 0$ is to be estimated. - a) Write down the joint density $f(\mathbf{x} \mid d)$ for the sample. - b) Using the Fisher--Neyman factorization theorem, show that $X_{(n)} = \max\{X_1, \ldots, X_n\}$ is sufficient for $d$. - c) Is $X_{(n)}$ unbiased for $d$? If not, find an unbiased estimator based on $X_{(n)}$. *Hint for (c): the CDF of $X_{(n)}$ is $F_{X_{(n)}}(x) = (x/d)^n$ for $0 \le x \le d$.* ### 03 Normal Variance: Minimal Sufficiency {data-difficulty="2"} Let $X_1, \ldots, X_n \overset{\text{i.i.d.}}{\sim} N(\mu, \sigma^2)$ where $\sigma^2 > 0$ is unknown but $\mu$ is **known**. - a) Show that $T(\mathbf{X}) = \sum_{i=1}^{n}(X_i - \mu)^2$ is sufficient for $\sigma^2$ using the factorization theorem. - b) Using the likelihood ratio criterion, show that $T(\mathbf{X})$ is **minimal** sufficient for $\sigma^2$. *Recall: $T$ is minimal sufficient iff $T(\mathbf{x}) = T(\mathbf{y})$ $\Longleftrightarrow$ $\frac{f(\mathbf{x} \mid \sigma^2)}{f(\mathbf{y} \mid \sigma^2)}$ is free of $\sigma^2$.* ### 04 Binomial Sufficiency and Estimating $\pi^2$ {data-difficulty="2"} Let $\mathbf{X} = (X_1, \ldots, X_n)^\top$ with $X_i \overset{\text{i.i.d.}}{\sim} \mathrm{Bernoulli}(\pi)$, where $\pi \in (0, 1)$. Define $U(\mathbf{X}) = \sum_{i=1}^{n} X_i$. - a) Show that $U(\mathbf{X})/n$ is unbiased for $\pi$. - b) Show that $U(\mathbf{X})$ is **minimal** sufficient for $\pi$. - c) Now consider the estimator for $\pi^2$: $$V(\mathbf{X}) = \frac{U(\mathbf{X})\left[U(\mathbf{X}) - 1\right]}{n(n-1)}.$$ Verify that $V(\mathbf{X})$ is unbiased for $\pi^2$. *Hint for (c): expand $\mathbb{E}[U(U-1)]$ using $\mathbb{E}[U] = n\pi$ and $\operatorname{Var}(U) = n\pi(1-\pi)$.* --- ## 2) Fisher Information & Cram\'{e}r--Rao ### 05 Fisher Information for the Exponential {data-difficulty="2"} Let $X_1, \ldots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$ with density $f(x \mid \lambda) = \lambda e^{-\lambda x}$ for $x \geq 0$. - a) Compute the score function $s(\lambda) = \frac{\partial}{\partial \lambda} \log f(X \mid \lambda)$. - b) Verify that $\mathbb{E}[s(\lambda)] = 0$. - c) Compute the Fisher information $I(\lambda) = \operatorname{Var}[s(\lambda)]$. - d) Verify your answer in (c) by computing $I(\lambda)$ via the second-derivative formula: $I(\lambda) = -\mathbb{E}\!\left[\frac{\partial^2}{\partial\lambda^2}\log f(X \mid \lambda)\right]$. - e) The MLE for $\lambda$ is $\hat{\lambda} = 1/\bar{X}$, which has $\operatorname{Var}(\hat{\lambda}) \approx \lambda^2/n$ for large $n$. Compare this with the Cram\'{e}r--Rao lower bound. Is $\hat{\lambda}$ asymptotically efficient? ### 06 Cramér--Rao: When Can We Beat $1/n$? {data-difficulty="3"} Let $X_1, \ldots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Bernoulli}(p)$. - a) We know $I(p) = \frac{1}{p(1-p)}$. Write down the Cram\'{e}r--Rao lower bound for any unbiased estimator of $p$. - b) The sample proportion $\hat{p} = \bar{X}$ has $\operatorname{Var}(\hat{p}) = \frac{p(1-p)}{n}$. Is $\hat{p}$ efficient? - c) Now consider estimating $g(p) = p^2$ instead of $p$. The Cram\'{e}r--Rao bound for unbiased estimators of $g(\theta)$ is $$\operatorname{Var}(\hat{g}) \geq \frac{[g'(\theta)]^2}{n \cdot I(\theta)}.$$ Compute the CR bound for unbiased estimators of $p^2$. - d) We showed in Problem 04(c) that $V(\mathbf{X}) = \frac{U(U-1)}{n(n-1)}$ is unbiased for $p^2$. Compute $\operatorname{Var}(V)$ (at least for large $n$). Does it achieve the CR bound? --- ## 3) Admissibility & Minimax ### 07 Admissibility: A Sketch Exercise {data-difficulty="1"} Suppose there exist exactly three estimators $T_1$, $T_2$, and $T_3$ for a parameter $\theta \in [0, 1]$. - a) Sketch an example of the MSE curves $\mathrm{MSE}(T_i, \theta)$ as functions of $\theta$ such that $T_1$ and $T_2$ are **admissible**, but $T_3$ is **not** admissible. Explain why your sketch works. - b) Now sketch (possibly different) risk functions for $T_1$, $T_2$, and $T_3$ such that $T_1$ is the **minimax** estimator. Must $T_1$ have the lowest MSE everywhere? # 🎲 xx+37 (xx) - ▶️[ToDo]() - 🔗[Random link ToDo]() - 🇦🇲🎶[ToDo]() - 🌐🎶[ToDo]() - 🤌[Կարգին ToDo]() <a href="http://s01.flagcounter.com/more/1oO"><img src="https://s01.flagcounter.com/count2/1oO/bg_FFFFFF/txt_000000/border_CCCCCC/columns_2/maxflags_10/viewers_0/labels_0/pageviews_1/flags_0/percent_0/" alt="Flag Counter"></a>

22: Statistics — Estimator Properties, Fisher Info, Cramér-Rao

📚 Նյութը

Դասախոսություն

Գործնական

🏡 Տնային

1) Exponential Family & Sufficiency

01 Poisson Meets the Exponential Family

02 Slit Width Estimation

03 Normal Variance: Minimal Sufficiency

04 Binomial Sufficiency and Estimating \(\pi^2\)

2) Fisher Information & Cram'{e}r–Rao

05 Fisher Information for the Exponential

06 Cramér–Rao: When Can We Beat \(1/n\)?

3) Admissibility & Minimax

07 Admissibility: A Sketch Exercise

🎲 xx+37 (xx)