A team wants ideas for a new neural network architecture. Which sources are most useful at the start?

Google

Two models can behave very differently on the same task: one architecture may fit the training data well but generalize poorly, while another may do the opposite. Because of that, changing the architecture can move both training error and dev/test error, not just one of them. Compared with adding more data or enlarging an existing model, trying a new architecture is usually harder to predict.

Changing Architecture Can Shift Both Error Sources

How changing model structure can affect error sources

Changing a model’s architecture usually has the same predictable effect on performance as simply increasing the amount of training data or the model size.

Architecture Changes Are as Predictable as Scaling

Architecture Changes Are Often _____

Ways to Improve a Neural Network

Trying Unusual Model Designs

Question:
Explain why changing a model’s design can sometimes be a better strategy than simply making the model larger or collecting more examples.

Sample answer:
Changing the structure of a model can produce a large improvement because it may correct both underfitting and overfitting at the same time. That makes it a potentially strong way to fit the task more closely. The downside is that this kind of change is usually difficult to choose well and harder to implement than basic scaling steps. It is also less certain: adding training data or increasing capacity tends to be more straightforward and easier to predict, while design changes can lead to very different outcomes.

Key points:
- May improve both bias and variance together
- Usually harder to select and apply correctly
- Outcomes are less predictable than scaling data or model size

Rubric:
A complete response should mention the possible gain of improving both bias and variance, and also note the practical difficulty and uncertainty of architecture changes compared with simpler scaling methods.

Why Architecture Changes Can Be Powerful and Risky

Case context:
You are building a model to spot manufacturing defects in conveyor-belt images. The current network is underfitting, but when you made the network much larger, the validation error got worse. You are now considering a different network architecture rather than only adjusting the size of the current one.

Question:
What should you expect about this approach, and where might you look for ideas for new architectures?

Sample answer:
A new architecture may help with both underfitting and overfitting at the same time. However, it is usually harder to predict the result of than a simple size change, and it can be more difficult to design and implement. Good places to look for ideas include recent research papers and open-source code repositories such as GitHub.

Key points:
- Can influence both bias and variance
- Less predictable than a simple size adjustment
- Harder to discover and implement
- Search papers and open-source code for ideas

Rubric:
The learner must mention the possible effect on both error sources, the difficulty and unpredictability of the change, and sources for architecture ideas.

Choosing a New Neural Network Design

Question:
Why might a machine learning engineer decide to significantly redesign a model architecture even when the change is hard to implement?

Sample answer:
An engineer may accept the difficulty because an architectural redesign can be one of the few changes that improves both bias and variance at the same time, making the model better suited to the task.

Key points:
- Can reduce both bias and variance together
- Can make the model better matched to the problem

Rubric:
Must state that architecture changes can simultaneously affect or reduce both bias and variance.

Learn Before

Related