Inverse problems convert indirect measurements into useful characterizations of the parameters of a physical system. Parameters are typically related to indirect measurements by a system of partial differential equations (PDEs), which are complicated and expensive to evaluate. Available indirect data are often limited, noisy, and subject to natural variation, while the unknown parameters of interest are often high dimensional, or infinite dimensional in principle. Solution of the inverse problem, along with prediction and uncertainty assessment, can be cast in a Bayesian setting and thus naturally tackled with Markov chain Monte Carlo (MCMC) methods. However, designing scalable and efficient sampling methods for high dimensional inverse problems that involve expensive PDE evaluations poses a significant challenge. In this talk, we will introduce a set of methods for identifying the intrinsic low dimensional structures in both the state space and the parameter space of inverse problems. By exploiting these low dimensional subspaces, scalable posterior approximations and reduced variance MCMC estimators can be obtained.