.md .pdf

Contents

Properties of expectations

There are few very important properties that expectations satisfy. Let’s introduce them one by one. In what follows \(X\) is a random variable, either discrete or continuous.

Expectation Property 1: The expectation of a scalar times a random variable

Take any scalar \(\lambda\). Then:

\[ \mathbf{E}[\lambda X] = \lambda \mathbf{E}[X]. \]

The proof is super easy. It follows directly from the properties of integrals and summations.

Expectation Property 2: The expectation of a function of a random variable

Let \(f(x)\) be any function defined over the support of the random variable \(X\). Then:

\[ Z = f(X), \]

is a new random variable. It is just a random variable that is induced by transforming the values generated by \(X\) through the function \(f(x)\). Then, the expectation of \(Z\) is:

\[ \mathbf{E}[Z] \equiv \mathbf{E}[f(X)] = \int f(x) p(x) dx. \]

Note

We will not prove this property. The proof is non-trivial and requires expressing the PDF of \(Z=f(X)\) in terms of the PDF of \(X\). Because people tend to take this property for granted, it is known as the law of the unconscious statistician…

This formula is, of course, extremely convenient for calculating expectations of functions of random variables. Let’s see an example. Assume that \(X\) is a uniform:

\[ X \sim U([0,1]). \]

Then:

\[\begin{split} \begin{split} \mathbf{E}[X^2] &= \int x^2 p(x) dx\\ &= \int_0^1 x^2 dx\\ &= \frac{x^3}{3} |_0^1\\ &= \frac{1}{3}. \end{split} \end{split}\]

Similarly,

\[ \mathbf{E}[e^X] = \int_0^1 e^x dx = e. \]

And so on and so forth.

Warning

I have to say this because I have seen it many times… You may get the urge to write:

\[ \mathbf{E}[f(X)] = f\left(\mathbf{E}[X]\right).\;\text{(WRONG)}. \]

Please don’t. This formula does not hold unless \(f(x)\) is a linear function (i.e., \(f(x) = ax + b\)).

Things are way more complicated than this. For example, for convex functions you have that:

\[ f\left(\mathbf{E}[X]\right) \le \mathbf{E}[f(X)]\;(f\text{convex}). \]

For concave functions you have the opposite inequality. For arbitrary functions it could go in any direction.

Expectation Property 3: The expectation of a random variable plus a scalar

This property is a direct corollary of the second property, but it is worth spelling it out on its own. Take a scalar \(\lambda\). Then:

\[ \mathbf{E}[X + \lambda] = \mathbf{E}[X] + \lambda. \]

The property makes a lot of intuitive sense.

Let’s actually do the proof of this one. We have:

\[\begin{split} \begin{split} \mathbf{E}[X+\lambda] &= \int (x + \lambda)p(x)dx\;\text{(property 2)}\\ &= \int x p(x) dx + \lambda\int p(x) dx\;\text{(integral properties)}\\ &= \mathbf{E}[X] + \lambda\;\text{(PDF integrates to one)}. \end{split} \end{split}\]

previous

Simplifying our notation about expectations

next

Variance of a random variable