Useful Inequalities for Variational Inference

Variational Inference is a technique which consists in bounding the log-likelihood ln p(x) defined by a model with latent variables p(x,z)=p(x|z)p(z) through the introduction of  a variational distribution q(z|x) with same support as p(z):

latex-image-1

Often the expectations in the bound F(x) (aka, ELBO or Free Energy) cannot be solved analytically.

In some cases, we can make use of a few handful inequalities which I quickly summarize below.

Some of these inequalities introduce new variational parameters. Those should be optimized jointly with all the other parameters to minimize the ELBO.

Screen Shot 2015-12-12 at 6.38.05 AM

 

 

Advertisements

3 thoughts on “Useful Inequalities for Variational Inference

  1. Pingback: Short Notes on Variational Bounds with Rescaled Terms | Invariance

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s