$$ \newcommand{\R}{\mathbb{R}} \newcommand{\SSS}{\mathbb{S}} \newcommand{\E}{\mathbb{E}} \newcommand{\C}{\mathbb{C}} \newcommand{\N}{\mathbb{N}} \newcommand{\Z}{\mathbb{Z}} \newcommand{\P}{\mathbb{P}} \newcommand{\Q}{\mathbb{Q}} \newcommand{\frakX}{\mathfrak{X}} \newcommand{\frakB}{\mathfrak{B}} \newcommand{\frakW}{\mathfrak{W}} \newcommand{\Id}{\mathrm{Id}} \newcommand{\dist}{\mathrm{dist}} \newcommand{\span}{\mathrm{Span}} \newcommand{\diag}[1]{\mathrm{diag}(#1)} \newcommand{\dim}[1]{\mathrm{dim} \, #1} \newcommand{\tr}[1]{\mathrm{tr}(#1)} \newcommand{\partder}[2]{\frac{\partial #1}{\partial #2}} \newcommand{\rank}[1]{\mathrm{rank} \, #1} \newcommand{\inner}[1]{\langle #1 \rangle} \newcommand{\innerbig}[1]{\left \langle #1 \right \rangle} \newcommand{\abs}[1]{\lvert #1 \rvert} \newcommand{\norm}[1]{\lVert #1 \rVert} \newcommand{\ker}[1]{\text{ker}(#1)} \newcommand{\supp}[1]{\text{supp}(#1)} \newcommand{\calA}{\mathcal{A}} \newcommand{\calB}{\mathcal{B}} \newcommand{\calC}{\mathcal{C}} \newcommand{\calD}{\mathcal{D}} \newcommand{\calE}{\mathcal{E}} \newcommand{\calF}{\mathcal{F}} \newcommand{\calG}{\mathcal{G}} \newcommand{\calH}{\mathcal{H}} \newcommand{\calI}{\mathcal{I}} \newcommand{\calJ}{\mathcal{J}} \newcommand{\calK}{\mathcal{K}} \newcommand{\calL}{\mathcal{L}} \newcommand{\calM}{\mathcal{M}} \newcommand{\calN}{\mathcal{N}} \newcommand{\calO}{\mathcal{O}} \newcommand{\calP}{\mathcal{P}} \newcommand{\calQ}{\mathcal{Q}} \newcommand{\calR}{\mathcal{R}} \newcommand{\calS}{\mathcal{S}} \newcommand{\calT}{\mathcal{T}} \newcommand{\calU}{\mathcal{U}} \newcommand{\calV}{\mathcal{V}} \newcommand{\calW}{\mathcal{W}} \newcommand{\calX}{\mathcal{X}} \newcommand{\calY}{\mathcal{Y}} \newcommand{\calZ}{\mathcal{Z}} \newcommand{\bh} \newcommand{\bf} \newcommand{\Pker}{P_{\ker{L}^{\perp}}} $$

Gaussian Field and SPDE

In this post, we will extend the previous discussion on Gaussian Field to Elliptical SPDE on Euclidean domain and compact Riemannian manifold with no boundaries. This turns out to be the natural way of defining general Gaussian field, which may have non-stationary second order structure, and/or its hyperparameters of the Covariance operator is dependent on spatial locations. Examples for Matérn class of kernels will be provided from Borovitskiy et al. 2020.

Posted by edenx on October 14, 2020

Gaussian Field and Covariance Operator

We will have a close look at Gaussian Process (Fields) in different views, with motivations from intuitive explanations given in Rasmussen & Williams 2003, and theoretical construction from Lototsky & Rozovsky 2017. This will serve as the preliminary setup for studying Elliptical SPDE driven by Gaussian white noise in the next post.

Posted by edenx on October 14, 2020

Green's function, RKHS and Regularisation

With the spectral perspective of RKHS introduced previously, we now look at a special and important category of reproducing kernels, which is the Green's functions to positive systems of differential equations. We will have examples on the eigenvalue problem for Dirichlet Laplace operator; and also on the Heat equation in Euclidean space. An intuitive generalisation is then provided. Finally, we will discuss informally the extension to discrete differential system and RKHS defined by Hermitian matrices.

Posted by edenx on October 7, 2020

Part 4: Covariance operator for reproducing kernels

Another important operator defined on a RKHS is Covariance operator, which is essential for doing kernel PCA, kernel CCA, and independence testing etc. We will see that under certain conditions, the centered Covariance operator is Hilbert-Schmidt, moreover, sharing the same spectrum as the HS Integral operator introduced in Part 3.

Posted by Eden on September 30, 2020

Part 3: Mercer's theorem and the spectral perspective of RKHS

In this article, we provide another formulation of RKHS in terms of the spectrum of the Integral operator corresponding to a reproducing kernel, which is given by Mercer's Theorem and the property of compact self-adjoint operators on separable Hilbert space.

Posted by Eden on September 28, 2020

Part 2: Characterising RKHS with linear operators on Hilbert space

In this article, we will see how an RKHS can be constructed by a linear operator on a Hilbert space, which has an important example linking probability and RKHS of stationary kernels, i.e. Bochner's Theorem. For the second part, we will see any complete orthogonal system (CONS) in a separable Hilbert space will give rise to a characterisation of RKHS.

Posted by Eden on September 24, 2020

Part 1: Reproducing Kernels and Construction of RKHS

This is the first post of a four-part series on fundamentals of RKHS, which mainly serves as a way to reorganise my understandings of RKHS (in Functional analysis) and kernel methods (in Machine learning) and try to connect the properties of RKHS to its applications in ML.

Posted by Eden on September 22, 2020