Sum of Random Variables
Lemma: The probability density function (PDF) of the sum of random variables is the
convolution of each random variable’s PDF.
Suppose the probability density function (PDF) for random variable \(X_1\) and
\(X_2\) is \(p_1 (x)\) and \(p_2 (x)\) respectively.
Prove: the PDF of random variable \(X = X_1 + X_2\) is \(p_1 * p_2\).
Proof:
Suppose \(p(x)\) is the PDF of \(X\):
\[ \begin{align}\begin{aligned}\begin{split}Pr (X \le t) & =
\int_{-\infty}^{t} p(x) dx
\\ & =
\iint_{x_1 + x_2 \le t} p_1 (x_1) p_2 (x_2) d x_1 d x_2\end{split}\\\begin{split}\\\end{split}\end{aligned}\end{align} \]
Let:
\[ \begin{align}\begin{aligned}\begin{split}\begin{cases}
u = x_1
\\
v = x_1 + x_2
\end{cases}\end{split}\\\begin{split}\\\end{split}\\\begin{split}\therefore
\begin{cases}
x_1 = u
\\
x_2 = v - u
\end{cases}\end{split}\end{aligned}\end{align} \]
\[\begin{split}\therefore
Pr (X \le t) & =
\iint_{x_1 + x_2 \le t} p_1 (x_1) p_2 (x_2) d x_1 d x_2
\\ & =
\int_{-\infty}^{\infty} \int_{-\infty}^{t} p_1 (u) p_2 (v - u) d v d u
\\ & =
\int_{-\infty}^{t} \int_{-\infty}^{\infty} p_2 (v - u) p_1 (u) d u d v
\\ & =
\int_{-\infty}^{t} (\int_{-\infty}^{\infty} p_2 (v - u) p_1 (u) d u) d v
\\ & =
\int_{-\infty}^{t} (p_1 * p_2)(v) d v
\\ & =
\int_{-\infty}^{t} (p_1 * p_2)(x) d x\end{split}\]
\[\therefore
p(x) = (p_1 * p_2) (x)
\space\]
\[\tag*{$\blacksquare$}\]
Central Limit Theorem
Random Variables \(X_1, X_2, \dots, X_n\) are independently and identically
distributed (i.i.d.) with their PDF equals \(p(x)\) and CDF equals \(P(x)\):
\[\begin{split}\int_{-\infty}^{+\infty} p(x) dx = 1
\\
\int_{-\infty}^{+\infty} x p(x) dx = 0
\\
\int_{-\infty}^{+\infty} x^2 p(x) dx = 1\end{split}\]
Let:
\[S_n = \frac{1}{\sqrt{n}} \sum_{i=1}^n X_i\]
Prove:
\[\lim_{n \to \infty} Pr (a \le S_n \le b) =
\frac{1}{\sqrt{2 \pi}} \int_a^b e^{-\frac{x^2}{2}} dx\]
Proof
Let \(S_n\)’s PDF equals \(p_n (x)\) and its CDF equals \(P_n (x)\).
The unproven conclusion is equivalent to:
\[\lim_{n \to \infty} p_n (x) = \frac{1}{\sqrt{2 \pi}} e^{- \frac{x^2}{2}}\]
Let the PDF and CDF of \(\sqrt{n} S_n = \sum_{i=1}^n X_i\) be \(f(x)\) and \(F(x)\)
respectively.
From the lemma we have:
\[f(x) = p^{(*n)} (x) = \overbrace{p * p * \dots * p}^{n} (x)\]
\[ \begin{align}\begin{aligned}\begin{split}\therefore
P_n (x) & =
Pr (S_n \le x)
\\ & =
Pr (\sqrt{n} S_n \le \sqrt{n} x)
\\ & =
F (\sqrt{n} x)\end{split}\\\begin{split}\\\end{split}\\\begin{split}\therefore
p_n (x) & =
\frac{d}{dx} P_n (x)
\\ & =
\frac{d}{dx} F (\sqrt{n} x)
\\ & =
f(\sqrt{n} x) \cdot \sqrt{n}
\\ & =
\sqrt{n} \cdot p^{(*n)} (\sqrt{n} x)\end{split}\\\begin{split}\\
\therefore
\mathcal{F} p_n (s) & =
\mathcal{F} (\sqrt{n} \cdot p^{(*n)} (\sqrt{n} x)) (s)
\\ & =
\sqrt{n} \cdot \mathcal{F} (p^{(*n)} (\sqrt{n} x)) (s)
\\ & =
\sqrt{n} \cdot \frac{1}{\sqrt{n}} \cdot \mathcal{F} p^{(*n)} (\frac{s}{\sqrt{n}})
\\ & =
\mathcal{F} p^{(*n)} (\frac{s}{\sqrt{n}})
\\ & =
(\mathcal{F} p)^n (\frac{s}{\sqrt{n}})\end{split}\end{aligned}\end{align} \]
\[\begin{split}\because
\mathcal{F} p (\frac{s}{\sqrt{n}}) & =
\int_{-\infty}^{+\infty} e^{-2 \pi i x \frac{s}{\sqrt{n}}} p(x) dx
\\ & =
\int_{-\infty}^{+\infty} (
1 - \frac{2 \pi i s x}{\sqrt{n}} + \frac{(2 \pi i s x)^2}{2 n} + R_3(x))
p(x) dx
\\ & =
\int_{-\infty}^{+\infty} p(x) dx -
\int_{-\infty}^{+\infty} \frac{2 \pi i s x}{\sqrt{n}} p(x) dx -
\int_{-\infty}^{+\infty} \frac{2 \pi^2 s^2 x^2}{n} p(x) dx +
\int_{-\infty}^{+\infty} R_3(x) p(x) dx
\\ & =
\int_{-\infty}^{+\infty} p(x) dx -
\frac{2 \pi i s}{\sqrt{n}} \int_{-\infty}^{+\infty} x p(x) dx -
\frac{2 \pi^2 s^2}{n} \int_{-\infty}^{+\infty} x^2 p(x) dx +
\int_{-\infty}^{+\infty} R_3(x) p(x) dx
\\ & =
1 - 0 - \frac{2 \pi^2 s^2}{n} + \int_{-\infty}^{+\infty} R_3(x) p(x) dx
\\ & \approx
1 - \frac{2 \pi^2 s^2}{n}\end{split}\]
\[\begin{split}\therefore
\lim_{n \to \infty} \mathcal{F} p_n (s) & =
\lim_{n \to \infty} (\mathcal{F} p)^n (\frac{s}{\sqrt{n}})
\\ & \approx
\lim_{n \to \infty} (1 - \frac{2 \pi^2 s^2}{n})^n
\\ & =
e^{-2 \pi^2 s^2}\end{split}\]
Let \(g(x)\) be the gaussian function \(e^{- \pi x^2}\). Obviously:
\[ \begin{align}\begin{aligned}\begin{split}\mathcal{F} (\frac{1}{\sqrt{2 \pi}} g(\frac{x}{\sqrt{2 \pi}})) & =
\frac{1}{\sqrt{2 \pi}} \sqrt{2 \pi} \mathcal{F} g ({\sqrt{2 \pi}} s)
\\ & =
g ({\sqrt{2 \pi}} s)
\\ & =
e^{-2 \pi^2 s^2}\end{split}\\\begin{split}\\\end{split}\\\begin{split}\therefore
\lim_{n \to \infty} p_n (x) & =
\mathcal{F}^{-1} (e^{-2 \pi^2 s^2}) (x)
\\ & =
\frac{1}{\sqrt{2 \pi}} g(\frac{x}{\sqrt{2 \pi}})
\\ & =
\frac{1}{\sqrt{2 \pi}} e^{- \frac{x^2}{2}}\end{split}\end{aligned}\end{align} \]
\[\tag*{$\blacksquare$}\]
Back to Fourier Transform.