统计Ⅱ-笔记

这学期的课程内容，原本想要整理一下的，但笔记有点乱，另外这门课注重数理推导所以公式有点多；因此只整理习题课上的提纲，基本上覆盖了这学期的主要内容（主要是因为自己懒??）。

Estimation: para \(\theta\) estimator \(\hat\theta\)

• MLE
• LSE

Unbiased: \(E[\hat\theta]=\theta\)

Hyperthesis Testing

\(H_0: \theta\in\Theta_0\leftrightarrow H_1:\theta\in\Theta_0^C\)

e.g. normal dist. \(x_1...x_n\sim N(\theta,1)\), estimate the para

Confidence Interval

Regression

Model: \(Y=X\beta+\epsilon\)

• Y: response vector
• X: Data matrix, design matrix
• \(\theta\): unknown parameters
• \(\epsilon\): error\(\sim N(0,\sigma^2I_n)\)

Estimation

1. LSE: \(\hat\beta=(X^TX)^{-1}X^TY\overset{\triangle}{=}b\)
2. MLE
3. BLUE: best linear unbiased estimator (Gauss-Markov Theo)

Fitted Model of Y: \(\hat Y=Xb=X(X^TX)^{-1}X^TY\overset{\triangle}{=}HY\)

Residual: \(e=Y=\hat Y=(I-H)Y\)

Residual sum of squares: \(SSE=Y^T(I-H)Y\)

Estimator of \(\sigma^2\): \(\hat\sigma^2\overset{\triangle}{=}s^2={SSE\over n-k-1}\), unbiased

Distribution of \(b\) and \(s^2\):
\[ b\sim N(\beta, \sigma^2(X^TX)^{-1})\(n-k-1)s^2\sim\sigma^2 \chi^2(n-k-1)\{SSE\over \sigma^2}={(n-k-1)s^2\over\sigma^2}\sim\chi^2(n-k-1)\\]

Inference

• F-test for regression relation: \(H_0: \beta_1=...=\beta_k=0\)

\[ F={SSR/\sigma^2k\over SSE/\sigma^2(n-k-1)}={MSR\over MES}\sim F_{k,n-k-1} \]

Reject \(H_0\), when \(F>F_{k,n-k-1}(1-\alpha)\)

• Test for \(\beta_j\): \(H_0: \beta_j=0\)

\[ {b_i-\beta_j\over\sqrt{\sigma^2(X^TX)^{-1}_{j+1,j+1}}}\sim N(0,1) \]

Under \(H_0\)
\[ {b_i\over\sqrt{\sigma^2(X^TX)^{-1}_{j+1,j+1}}}\sim N(0,1) \]

Model Fitness

\[ R^2={SSR\over SST}\in[0,1]\R^2_\alpha=1-{SSE/(n-k-1)\over SST/(n-1)} \]

e.g.

given values \(x_0=(1,x_{01},...,x_{0k})^T\), \(x_0^T\beta\) CI
\[ {x_0^Tb-x_0^T\beta\over\sigma\sqrt{x_0^T(X^TX)^{-1}_{j+1,j+1}x_0}}\sim N(0,1)\{x_0^Tb-x_0^T\beta\over s \sqrt{x_0^T(X^TX)^{-1}_{j+1,j+1}x_0}}\sim t_{n-k-1}\\]
so the \(1-\alpha\) CI for \(x_0^T\beta\) is \([x_0^T\pm t_{n-k-1}(\alpha/1)s\sqrt{x_0^T(X^TX)^{-1}x_0}]\)

e.g.

CI for a new ovservation

e.g.

\(H_0: C\beta=0\), where \(C\in R^{d\times (k+1)}\)

The test statistic is
\[ F={SSE_R-SSE_F/d\over SSE_F/(n-k-1)}\sim F_{d,n-k-1} \]
if \(F>F_{d-,n-k-1}(\alpha)\), regect \(H_0\)

• \(SSE_F\): SSE under the full model

• \(SSE_R\): SSE under the reduced model: \(Y=X\beta+\epsilon, C\beta=0\)

\[ SSE_R-SSE_F=b^TC^T(C(X^TX)^{-1}C^T)^{-1}Cb \]

Model Seletion

\[ C_p={SSE\over s^2}-(n-2p+1)\AIC=n\log(SSE(p))+2p \]

Logistic Regression

Model: \(y_1,..y_n\) iid. binary observations \(y_i\)=1with prob. \(p_i\) and \(=0\) with prob. \(1-p_i\)
\[ logit(p_i)=\log{p_i\over 1-p_i}=\beta^Tx_i\p_i={\exp(\beta^T x_i)\over 1+\exp(\beta^T x_i)} \]
MLE for $?p_i$(as for \(\beta\)):
\[ l(\beta)=\sum y_i\log(p_i)+(1-y_i)\log(1-p_i)\\hat\beta=\arg\max_\beta l(\beta) \]

Generalized linear model

GLM:

• \(y_i\) : random component from a particular dist. in the exponential family

\[ g(EY)=X\beta \]

• \(X\beta\): linear predictor

• \(g\): link func.

Exponential family

\[ f_\lambda(x)=e^{\lambda x-r(\lambda)}f_0(x), \lambda\in\Lambda \]

Normal, Poisson, Binary, Gamma

e.g.: Gamma dist:
\[ f(x)={\beta^\alpha\over \Gamma(\alpha)}x^{\alpha-1}e^{-\beta x}, x>0\=\exp(\alpha \log(\beta)+(\alpha\log(x)-\beta x-\log\Gamma(x)))={x^{\alpha-1}\over\Gamma(\alpha)}e^{-\beta x +\alpha\log\beta} \]

Mean and variance

Moment generating func.
\[ \varphi(t)=Ee^{tX}\\varphi'(t)|_{t=0}=Ee^{tX}X|_{t=0}=EX\\varphi''(t)|_{t=0}=Ee^{tX}X^2|_{t=0}=EX^2\\]

\[ Var(X)=\varphi''(0)=\varphi'(0)^2 \]

For \(X\sim\) exponential family, \(EX=r'(\lambda), Var(X)=r''(\lambda)\)

• link func \(g(u)=u\) identity + normal response , GLM reduces to linear regression;
• link func \(g(u)=logit(u)\) + binary response, GLM reduces to logistic regression

A key idea in GLM is respresent \(\lambda_1,...,\lambda_n\) as a linear equation
\[ \lambda=X\beta\\lambda_i=x_i^T\beta \]
density func. of \(y\) is
\[ \prod f_{\lambda_i}(y_i)=\prod e^{\lambda_iy_i-r(\lambda_i)}f_0(y_i)\=\exp(\sum \lambda_i y_i-r(\lambda_i))\prod f_o(y_i) \]

\[ \sum \lambda_i y_i=\lambda^Ty=\beta^TX^Tz=\beta^Tz\\sum r(\lambda_i)=\sum r(x_i^T\beta) \]

\[ \prod f_{\lambda_i}(y_i)=\exp(\beta^Tz-\sum r(x_i^T \beta))\prod f_0(y_i)\=\exp(\beta^Tz-\varphi(\beta))f_0(y) \]

where \(\varphi(\beta)=\sum r(x_i^T\beta), f_0(y)=\prod f_0(y_i)\)

\[ \varphi'(\beta)=\sum r'(x_i^T\beta)x_i=\sum r'(\lambda_;)x_i=(x_1,...,x_n)(r'(\lambda_1),..,r'(\lambda_n))^T=X^T\mu(\beta) \]
loglikelihood:
\[ l_\beta(y)=\beta^Tz-\varphi(\beta)+\log f_0(y)=\beta^TX^Ty-\varphi(\beta)+c\l_\beta'(y)=X^Ty-X^T\mu(\beta) \]
then, MLE of \(\beta\), denoted by \(\hat\beta\) satisfies
\[ X^T(y-\mu(\hat\beta))=0 \]

ANOVA (one-way)

Model: \(y_{ij}=\mu_i+\epsilon_{ij}\), \(\epsilon_{ij}\overset{iid}{\sim}N(0,\sigma^2)\)

for \(i=1,...,r\) \(\sum_{i=1}^r n_i=n\)

Estimation

LSE, MLE for \(\mu_i\): \(\hat\mu_i=\bar y_{i.}\)

unbiaesd estimaor for \(\sigma^2\): MSE

Inference

Hyphysis: \(H_0: \mu_1=...=\mu_r\)

Test statistic:
\[ F={MSA\over MSE}\sim F_{r-1, n-r} \]
Reject \(H_0\), when \(F>F_{r-1,n-r}(\alpha)\)

Two-factor ANOVA

Model (sample level)

\(y_{ijk}=\mu_{ij}+\epsilon_{ijk}\), \(\epsilon_{ijk}\overset{iid}{\sim}N(0,\sigma^2)\)

Cell mean: \(y_{ijk}=\mu+\alpha_i+\beta_i+r_{ij}+\epsilon_{ijk}\)

Estimation

Inference (with interaction)

\(r_{ij}\ne0\)
\[ \sum_{ijk}(y_{ijk}-\bar y_{...})^2=\sum_{ijk}(y_{ijk}-\bar y_{ij.})^2+\sum_{ijk}(\bar y_{ij.}-\bar y_{...})^2\SST=S_e+S_{AB}\S_{AB}=SSA+SSB+SSAB\\]

\[ SSA=\sum_{ijk}(\bar y_{i..}-\bar y_{...})^2\SSB=\sum_{ijk}(\bar y_{.j.}-\bar y_{...})^2\SSAB=\sum_{ijk}(\bar y_{ij.}-\bar y_{i..}-\bar y_{.j.}+\bar y_{...})^2\\]

Revised: \(Y_{ijk}=\mu+\alpha_i+\beta_j+\epsilon_{ijk}\)

Test for interaction:
\[ F={MSAB\over MSE}\sim F_{(a-1)(b-1), ab(n-1)} \]

对于最后的双因子 ANOVA 再说明一点：上面给出了在有无交互作用下的方差分解及其自由度。对于交互作用/主效应的假设检验；或者均值/各均值的线性组合的 CI，他们的思路都是一样的。在这里只给出了对于交互作用的假设检验统计量；对于其他的各量来说，方法都是一样的：找出对于量的分布（如 A/B 的主效应为相应自由度的卡方分布，均值则为一定方差的正态分布），该分布中有为知参数\(\sigma^2\)，再借助 \(s^2=MSE=S_e/df\) 将其消去，得到相应的 t 分布或 F 分布，再根据问题要求（假设检验或 CI）进行求解。唯一需要注意的是，要看清两因子之间是否有交互作用，这决定了 \(MSE\) 的自由度。

统计Ⅱ-笔记

原文：https://www.cnblogs.com/easonshi/p/12158845.html