The core idea of parameter sharing is to use the same parameters in all the AGGREGATE and UPDATE functions in the GNN. Generally, this approach is most effective in GNNs with more than six layers, and it is often used in conjunction with gated update functions.

Parameter Sharing

In edge dropout, we randomly remove (or mask) edges in the adjacency matrix during training, with the intuition that this will make the GNN less prone to overfitting and more robust to noise in the adjacency matrix. This approach has been particularly successful in the application of GNNs to knowledge graphs, and it was an essential technique used in the original graph attention network (GAT) work.

Edge Dropout

In the context of GNNs, many of the standard regularization approaches are known to work well, including L2 regularization, dropout, and layer normalization. However, there are also regularization strategies that are somewhat specific to the GNN setting.

University of Michigan - Ann Arbor

GNNs are a general framework for defining deep neural networks on graph data. The goal of GNNs is to generate representations of nodes that actually depend on the structure of the graph, as well as any feature information available.

Graph Neural Networks (GNNs)

Hamilton, W.L. Graph Representation Learning. *Synthesis Lectures on Artificial Intelligence and Machine Learning*. 2020, 14(3), 1–159. https://doi.org/10.2200/S01045ED1V01Y202009AIM046

Graph Representation Learning by William Hamilton

One of the defining features of a GNN is that it uses a form of *neural message passing* in which vector messages are exchanged between nodes and updated using neural networks

Neural Message Passing

Regularization in GNN

GNNs had been independently developed from several distinct theoretical motivations and perspectives:
- Graph convolutions: the theory of graph signal processing, as a generalization of Euclidean convolutions to the non-Euclidean graph domain.
- Probabilistic graphical models: neural message passing approaches proposed by analogy to message passing algorithms for probabilistic inference in graphical models.
- Graph isomorphism: connection to the Weisfeiler-Lehman graph isomorphism
test.

Theoretical Motivations of GNNs

Original node-level message passing mechanism can be time-consuming.

Efficiency Concerns of GNNs

Pair wise node embedding loss function are deployed for applications utilizing graph relation prediction.

Loss function in GNNs for Relation Prediction

Pretraining a Graph Neural Network (GNN) is a method used to inject domain knowledge into a model prior to fine-tuning it on a specific downstream task. Finding effective pretraining strategies for graphs can be challenging; for instance, using a naive neighborhood reconstruction loss to predict missing edges often fails to improve downstream classification performance. Successful GNN pretraining typically relies on more advanced techniques, such as maximizing mutual information between local and global representations.

Pretraining a GNN

The basic GNN can be derived as a neural network variation of the existing Weisfieler-Lehman (WL) isomorphism algorithm

GNNs and Graph Isomorphism

1. Given two graphs, we assign each node an initial label in each graph. 
2. We iteratively assign a new label to each node in each graph by hashing the multi-set of the current labels within the node’s neighborhood, as well as the node’s current label.
3. We repeat Step 2 until the labels for all nodes in both graphs converge.
4. Finally, we construct  multi-sets summarizing all the node labels in each graph, and we declare G1 and G2 to be isomorphic if and only if the multi-sets for both graphs are identical.

The Weisfieler-Lehman(WL) Algorithm

Given an input graph $$G = (V,E)$$ and a set of node features **X** $$\in \mathbb{R}^{d*|V|}$$, our goal is to produce a node embedding $$z_u$$, for all nodes in the graph.

Learn Before

Related

Learn After