Google

Different model architectures can have different amounts of bias and variance for a given problem, and modifying the architecture to better suit the problem can affect both bias and variance. Trying new architectures is less predictable than simply increasing model size or adding data.

Modifying Model Architecture Can Affect Both Bias and Variance

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Impact of Modifying Model Architecture

Predictability of Architecture Changes

Difficulty of Architecture Modifications _____

Comparing Approaches to Bias and Variance

Exploring New Architectures

Question:
Discuss the potential benefits and drawbacks of modifying a model's architecture compared to simpler methods like increasing model size or adding data.

Sample answer:
Modifying a model's architecture can be highly beneficial because it has the potential to simultaneously affect and reduce both bias and variance, making the model more suitable for a specific problem. However, the drawbacks are significant. Identifying and implementing the right architecture changes is harder than simpler methods. Furthermore, the results of trying new architectures are much less predictable than the straightforward formula of just adding more data or increasing the size of an existing model.

Key points:
- Can affect both bias and variance
- Harder to identify and implement
- Results are less predictable than adding data or increasing model size

Rubric:
The response should identify the benefit (affecting both bias/variance) and the drawbacks (harder to implement, less predictable).

Trade-offs in Changing Architecture

Case context:
You are working on a computer vision task using a standard neural network. Your model is suffering from high bias, but your previous attempts to simply increase model size led to massive variance. You are considering completely changing the neural network architecture.

Question:
Based on the text, what should you expect regarding the predictability and difficulty of this approach, and where could you look for new architectures?

Sample answer:
Changing the architecture can potentially address both the bias and variance issues simultaneously. However, you should expect this approach to be harder to identify and implement, and the results will be less predictable than just increasing the network size. You should look to academic literature and open-source implementations on GitHub for inspiration on innovative architectures.

Key points:
- Can affect both bias and variance simultaneously
- Less predictable than simple formulas
- Harder to identify and implement
- Use academic literature or open-source GitHub implementations for inspiration

Rubric:
The learner must note the potential to affect both bias/variance, the unpredictability/difficulty of the approach, and suggest sources like literature or GitHub.

Struggling with Bias and Variance in Vision Models

Question:
Why might a machine learning practitioner choose to make major changes to a system's architecture despite it being difficult to implement?

Sample answer:
A practitioner might change the system's architecture because it is one of the few methods that can simultaneously reduce both bias and variance, making the model more suitable for the specific problem.

Key points:
- Simultaneously reduce bias and variance
- Make model more suitable for the problem

Rubric:
Must state that architectural changes can impact or reduce both bias and variance simultaneously.

Learn Before

Related