It receives input from some other nodes, or from an external source and computes an output. Each input has an associated weight (w), which is assigned on the basis of its relative importance to other inputs. Below is an example of a neuron for logistic regression.

University of Michigan - Ann Arbor

A neural network is typically structured with an input layer, one or more hidden layers, and an output layer. Its design is historically inspired by biological neurons: dendrites are modeled as inputs $$x_i$$, synapses as synaptic weights $$w_i$$, and the nucleus as an aggregator that computes a weighted sum $$y = \sum_i x_i w_i + b$$, potentially followed by a nonlinear activation function $$\sigma(y)$$. The resulting signal is then transmitted to the output, analogous to a biological axon. A positive weight reflects an excitatory connection, while a negative value signifies an inhibitory connection.

Neural Network Representation

Singular neuron of NN

A computational graph is a directed dependency graph where nodes represent mathematical operations or variables, mapping the functional dependency from inputs to outputs. When evaluating a vector-valued function, this graph is traversed in a forward direction. Conversely, to compute gradients automatically, the graph is traced in a backwards direction from output to input, which involves multiplying matrices of partial derivatives along the paths.

Computational Graph

Linear regression can be conceptualized as a single-layer, fully connected neural network. In this representation, each given input feature $$x_1, \ldots, x_d$$ corresponds to an input neuron. Since the goal is to predict a single numerical value, all input neurons are directly connected to a single computed output neuron $$o_1$$. The total number of inputs $$d$$ is referred to as the feature dimensionality in the input layer.

Learn Before

Related