Central Limit Theorem

Math Probability

The Central Limit Theorem

Let us sum $n$ instances from an i.i.d (independent and identical distribution) with defined first and second moments (mean and variance). Center the distribution on $0$ and scale it by its standard deviation. As $n$ goes to infinity, the distribution of that variable goes toward

$$\frac{1}{\sqrt{2}\pi }{e}^{-\frac{x^2}{2}}$$

or the standard normal distribution

Mathematical Definition

Let Y be the mean of a sequence of n i.i.ds

$$Y=\frac{1}{n}\sum _{i=1}^n{X}_i$$

Let $\mu =E(X_i)$, the expected value of $X$, and $\sigma =\sqrt{Var(X)}$, the standard deviation of $X$

Calculate the expected value of Y, $E(Y)$, and the variance, $Var(Y)$:

$$\begin{gathered} E(Y) \\ =E(\frac{1}{n}\sum _{i=1}^n{X}_i) \\ =\frac{1}{n}\sum _{i=1}^{n}E(X_i) \\ =\frac{1}{n}\sum _{i=1}^n\mu \\ =\frac{n\mu }{n} \\ =\mu \end{gathered}$$ $$\begin{gathered} Var(Y) \\ =Var(\frac{1}{n}\sum _{i=1}^n{X}_i) \\ =\frac{1}{n^2}\sum _{i=1}^{n}Var(X_i) \\ =\frac{\sigma }{n} \end{gathered}$$

Let $Y^{\ast }$ be centered by $E(Y)$ and scaled by it’s standard deviation, $\sqrt{Var(Y)}$

$$\begin{gathered} Y^{\ast } \\ =\frac{Y-E(Y)}{\sqrt{Var(Y)}} \\ =\frac{Y-\mu }{\sqrt{\frac{{\sigma }^2}{n}}} \\ =\frac{\sqrt{n}(Y-\mu )}{\sigma } \\ =\frac{\sqrt{n}(\frac{1}{n}{\sum }_{i=0}^n{X}_i-\mu )}{\sigma } \\ =\frac{\frac{1}{\sqrt{n}}({\sum }_{i=0}^n{X}_i-\mu )}{\sigma } \end{gathered}$$

The CLT states

$$Y^{\ast }\stackrel{d}{\to }N(0,1)$$

Or $Y^{\ast }$ converges in distribution to the standard normal distribution with a mean of 0 and a standard deviation of 1

Proof

A Change in Variables

Let $S$ be the sum of our sequence of n i.i.ds

$$S=\sum _{i=1}^n{X}_i$$

Let’s calculate $E(S)$ and $Var(S)$

$$\begin{gathered} E(S) \\ =E(\sum _{i=1}^n{X}_i) \\ =\sum _{i=1}^{n}E(X_i) \\ =\sum _{i=1}^n\mu \\ =n\mu \end{gathered}$$ $$\begin{gathered} Var(S) \\ =Var(\sum _{i=1}^n{X}_i) \\ =\sum _{i=1}^{n}Var(X_i) \\ =\sum _{i=1}^n{\sigma }^2 \\ =n{\sigma }^2 \end{gathered}$$

Center $S$ by $E(S)$ and scale it by $\sqrt{Var(S)}$ for $S^{\ast }$

$$\begin{gathered} S^{\ast } \\ =\frac{S-E(S)}{\sqrt{Var(S)}} \\ =\frac{S-n\mu }{\sqrt{n{\sigma }^2}} \\ =\frac{S-n\mu }{\sqrt{n}\sigma } \\ =\frac{\frac{1}{\sqrt{n}}(S-n\mu )}{\sigma } \\ =\frac{\frac{1}{\sqrt{n}}({\sum }_{i=0}^n{X}_i-\mu )}{\sigma } \end{gathered}$$

From the above, $Y^{\ast }=S^{\ast }$. In the proof, we will use $S^{\ast }$, as it is easier to manipulate.

MGFs

An MGF is a function where

$$M_V(t)=E(e^{tV})$$

where $V$ is a random variable

(reminder for me to do another notion on this)

Properties of MGFs

Property 1:

If

$$C=A+B$$

Then

$$\begin{gathered} M_C(t) \\ =E(e^{tC}) \\ =E(e^{ta+tb}) \\ =E(e^{ta}{e}^{tb}) \\ =E(e^{ta})+E(e^{tb}) \\ =M_A(t)+M_B(t) \end{gathered}$$

Property 2:

$$M_V^{(r)}(0)=E(V^r)$$

The $r$ derivative of $M_V$ gives the $r$ moment of $V$

Property 3:

Let $A$ be a sequence of random variables with MGFs of $A_1$, $A_2$… $A_n$

If

$$M_{A_n}(t)\to M_B(t)$$

Then

$$A\stackrel{d}{\to }B$$

MGF of a Normal Distribution

Let a random variable derived from a standard normal distribution be Z

$$Z\sim N(0,1)$$ $$\begin{gathered} M_z(t) \\ =E(e^{xt}) \\ ={\int }_{-\infty }^{\infty }{e}^{xt}\frac{1}{\sqrt{2\pi }}{e}^{-\frac{x^2}{2}}dx \\ ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{tx-\frac{1}{2}{x}^2}dx \\ ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(x^2-2tx)}dx \\ ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(x^2-2tx+t)+\frac{1}{2}{t}^2}dx \\ ={\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(x-t{)}^2+\frac{1}{2}{t}^2}dx \\ =e^{\frac{1}{2}{t}^2}{\int }_{-\infty }^{\infty }\frac{1}{\sqrt{2\pi }}{e}^{-\frac{1}{2}(x-t{)}^2}dx \\ =e^{\frac{t^2}{2}} \end{gathered}$$

The Argument

To prove the CLT, we need to prove that $S^{\ast }$ converges to $N(0,1)$ as $n\to \infty$. Our approach will be to prove that the MGF of $N(0,1)$ converges to the distribution of $S^{\ast }$ as $n\to \infty$.

$$\begin{gathered} S^{\ast } \\ =\frac{S-E(S)}{\sqrt{Var(S)}} \\ =\frac{S-n\mu }{\sqrt{n{\sigma }^2}} \\ =\frac{{\sum }_{i=1}^n{X}_i-n\mu }{\sqrt{n}\sigma } \\ =\sum _{i=1}^n\frac{X_i-u}{\sqrt{n}\sigma } \end{gathered}$$

Start manipulating MGF of $S^{\ast }$:

$$\begin{gathered} M_{S^{\ast }}(t) \\ =E(e^{t{S}^{\ast }}) \\ =E(e^{t({\sum }_{i=1}^n\frac{X_i-u}{\sqrt{n}\sigma })}) \\ =E(e^{t(\frac{(X-\mu )}{\sqrt{n}\sigma })}{)}^n \\ =(M_{\frac{(X-\mu )}{\sqrt{n}\sigma }}(t){)}^n \\ =(M_{(X-\mu )}(\frac{t}{\sqrt{n}\sigma }{)}^n \end{gathered}$$

Expand out Taylor series for $(M_{(x-\mu )}(\frac{t}{\sqrt{n}\sigma }){)}^n$ (note $O(t^3)$ means order $t^3$ and above, and tends to zero as $n$ goes to $\infty$ ):

$$\begin{gathered} M_{(X-\mu )}(\frac{t}{\sqrt{n}\sigma }) \\ =(M_{(X-\mu )}(0))+(\frac{M_{(X-\mu )}\prime (0)}{1!})(\frac{t}{\sqrt{n}\sigma })+(\frac{M_{(X-\mu )}\prime \prime (0)}{2!})(\frac{t}{\sqrt{n}\sigma }{)}^2+(\frac{M_{(X-\mu )}\prime \prime \prime (0)}{1!})(\frac{t}{\sqrt{n}\sigma }{)}^3+... \\ =1+(\frac{t}{\sqrt{n}\sigma })E(X-\mu )+(\frac{t^2}{2n{\sigma }^2})E((X-\mu {)}^2)+(\frac{t^3}{6n^{\frac{3}{2}}{\sigma }^3})E((X-\mu {)}^3)+... \\ =1+(\frac{t}{\sqrt{n}\sigma })E(X-\mu )+(\frac{t^2}{2n{\sigma }^2})E((X-\mu {)}^2)+O(t^3) \\ \approx 1+(\frac{t}{\sqrt{n}\sigma })E(X-\mu )+(\frac{t^2}{2n{\sigma }^2})E((X-\mu {)}^2) \end{gathered}$$

Remember $E(X-\mu )=0$ and $E((X-\mu {)}^2)={\sigma }^2$

$$\begin{gathered} =1+(\frac{t}{\sqrt{n}\sigma })(0)+(\frac{t^2}{2n{\sigma }^2})({\sigma }^2) \\ =1+\frac{t^2}{2n} \end{gathered}$$

Solve for $M_{S^{\ast }}(t)$:

$$M_{S^{\ast }}(t)=(1+\frac{t^2}{2n}{)}^n$$

Solve $M_{S^{\ast }}(t)$ for ${\lim }_{n\to \infty }$:

$$\begin{gathered} \underset{n\to \infty }{\lim }{M}_{S^{\ast }}(t) \\ =\underset{n\to \infty }{\lim }(1+\frac{t^2}{2n}{)}^n \\ =\underset{n\to \infty }{\lim }(1+\frac{1}{(\frac{2n}{t^2})}{)}^{\frac{t^2}{2}(\frac{2n}{t^2})} \\ =e^{\frac{t^2}{2}} \end{gathered}$$

Since ${\lim }_{n\to \infty }{M}_{S^{\ast }}(t)\to M_Z(t)$, ${\lim }_{n\to \infty }{S}^{\ast }\stackrel{d}{\to }N(0,1)$. Therefore:

$$Y^{\ast }\stackrel{d}{\to }N(0,1)$$

proving the Central Limit Theorem

Summary of the Argument

$$\begin{gathered} Y^{\ast }=S^{\ast } \\ \underset{n\to \infty }{\lim }{M}_{S^{\ast }}(t)\to M_Z(t) \\ \underset{n\to \infty }{\lim }{S}^{\ast }\to N(0,1) \\ \underset{n\to \infty }{\lim }{Y}^{\ast }\to N(0,1) \end{gathered}$$