Attention mechanism is a type of neural network structure that allow neighbors of a nodes have different aggregation weight through training and node-neighbor interaction. For example, Graph Attention Network (GAT) let node neighbors learn different weights, then aggregate node neighbors’ feature to obtain center node feature using learned weights.

Improving GCN using Attention Mechanism

Previous methods improve GCN by utilizing node features. However, edge features can also be very useful. For example, Column Network (CLN) differentiated different type of edges, and trained different parameters for different types of edges, and average the all results as the eventual result. Diffusion-Convolutional Neural Networks (DCNN) converted each edge into one node that’s connected with the end nodes of that edge, and pass edge feature to that node feature.

Improving GCN using Edge Features

In the real world, graphs can be very large and make aggregation based GCN hard to handle. And many graphs follow power-law distribution, which makes the number of neighbors expand extremely quickly. To solve this, neighborhood sampling and layer-wise samplings have been proposed.

Improving GCN using Sampling Methods

In graph setting, inductive learning means nodes or graphs in testing set are totally unseen in training set. Inductive GCN can output a feature mapping function that’s not dependent on graphs and can be transferred to new graphs or nodes.

Improving GCN using Inductive Setting

Theoretical Analysis can be very helpful in understanding the effectiveness of GCN, and it includes node-focus tasks analysis, graph-focused tasks analysis, and general analysis.

Improving GCN using Theoretical Analysis

In deep neural networks, many researchers have noticed that the optimal depth is often very shallow due to the over-smoothing issue, i.e., nodes in deeper layers tend to have similar representations. To solve this issue, residue connection has been proposed and it allows deeper layers to attend to much shallower previous layers directly. This idea can be applied to GCN, such as in the GCN model proposed by Kipf and Welling, residue connection is used to allow for a deeper network.

Improving GCN using Residual Connections

1)	Attention Mechanism

2)	Residual and Jumping Connections

3)	Utilizing Edge Features

4)	Sampling Methods

5)	Inductive Setting

6)	Theoretical Analysis


University of Notre Dame

GCN uses convolution operation and readout operation to grasp node level and graph level patterns.

Graph Convolutional Networks (GCN)

Zhang, Z., Cui, P., & Zhu, W. (2020). Deep learning on graphs: A survey. IEEE Transactions on Knowledge and Data Engineering.

Deep Learning on Graphs: A Survey

Convolution operations can be divided into two types:

1)	Spectral convolution

2)	Spatial convolution

3)	Mixture of spectral convolution and special convolution


Types of Convolution Operation

Possible Improvements of GCN

When a graph-level representation is needed to tackle graph-level tasks, node-level representations can be aggregated into a single graph-level representation. This aggregation process is referred to as a readout operation.

Learn Before

Related

Learn After