Learn Before
Concept

Kernel-Based Embeddings of Joint Distributions

Kernel embeddings for learning machines achieve strong performance through their representational power. To leverage this, Lopez-Paz et al. introduced kernel-based embeddings for feature construction in pairwise causal discovery. Starting from a dataset of empirical joint distributions S = { (x_{ij},y_{ij}){j=1}^{n_i} }{i=1}^n, a kernel mean embedding projects all these distributions into the same Reproducing Kernel Hilbert Space (RKHS) Hk\mathcal{H}_k. To obtain a homogeneous, low-dimensional embedding, random cosine-based embeddings are used to approximate empirical kernel mean embeddings: μk,m(PSj)=2CkSxij,yijSj(cos(wjxxij+wjyyij+bj))j=1mRm\mu_{k,m}(P_{S_j}) = \frac{2C_k}{|S|} \sum_{x_{ij},y_{ij} \in S_j} (\cos (w_j^x \ast x_{ij} + w_j^y \ast y_{ij} + b_j))_{j=1}^m \in \mathbb{R}^m where μk,m(PSj)\mu_{k,m}(P_{S_j}) is the kernel mean embedding of the empirical distribution PSjP_{S_j}, and mm defines the number of dimensions of the output space. The kernel parameters {wj,bj}j=1m\left\{w_j, b_j\right\}_{j=1}^m are sampled i.i.d. from N0,2×[0,2π]\mathbb{N}_{0,2} \times [0, 2\pi]. Finally, Ck=Lpk(w)dwC_k = \int_{\mathcal{L}} p_k(w)dw with p_k : mathbb{R}^d rightarrow mathbb{R} being the positive and integrable Fourier transform of the chosen kernel kk (equal to 1 in this case).

0

1

Updated 2026-06-14

Tags

Data Science