set.seed(1)
M <- 1e5
theta <- rbeta(M, 2, 5)
mean(theta) # Monte Carlo estimate[1] 0.2861808var(theta) / M # Monte Carlo variance[1] 2.56548e-074 Week 3 — Monte Carlo Integration and Simulation-Based Bayesian Inference
4.1 Overview
This week introduces Monte Carlo methods, which allow us to approximate Bayesian quantities when analytical solutions are unavailable.
We explore how random sampling can be used to estimate expectations, posterior summaries, and probabilities.
By the end of this week, students will understand how Monte Carlo simulation forms the foundation for modern Bayesian computation such as MCMC.
4.2 Learning Goals
By the end of Week 3, you should be able to:
- Explain the motivation for Monte Carlo methods in Bayesian inference.
- Approximate expectations, integrals, and posterior summaries using random sampling.
- Implement crude Monte Carlo and importance sampling in R.
- Assess the accuracy and variance of Monte Carlo estimators.
- Interpret Monte Carlo errors and convergence diagnostics.
4.3 Lecture 1: Motivation and Fundamentals of Monte Carlo
4.3.1 1.1 The Problem
Bayesian inference often requires evaluating integrals such as: \[ E[h(\theta) \mid y] = \int h(\theta)\, p(\theta \mid y)\, d\theta, \] which are rarely available in closed form.
4.3.2 1.2 Monte Carlo Idea
If we can sample \(\theta^{(1)}, \ldots, \theta^{(M)}\) from the posterior \(p(\theta \mid y)\),
then we can approximate the expectation by: \[
\hat{E}[h(\theta)] = \frac{1}{M} \sum_{m=1}^M h(\theta^{(m)}).
\] This is called the Monte Carlo estimator.
By the Law of Large Numbers, \(\hat{E}[h(\theta)] \to E[h(\theta)]\) as \(M \to \infty\). The Central Limit Theorem gives: \[ \sqrt{M}\,(\hat{E} - E[h(\theta)]) \approx N(0, \text{Var}[h(\theta)]). \]
4.3.3 1.3 Monte Carlo Error
We can estimate the simulation error by:
\[ \text{SE}(\hat{E}) \approx \sqrt{\frac{\text{Var}(h(\theta))}{M}}. \] Larger \(M\) gives more accurate approximations but increases computation time.
4.3.4 1.4 Simple Example
Compute \(E[\theta]\) for \(\theta \sim \text{Beta}(2,5)\) using Monte Carlo.