Based on the scenario, analyze the most likely deficiency in the 'label' vectors associated with each word in the sentence that would cause this specific failure.

Google

In an attention mechanism, the Key vector (K) is associated with each item in the input sequence. It serves as a label or identifier that is compared with the Query vector. The similarity between a Query and a Key determines the attention score, or weight, assigned to the corresponding input item.

Key Vector

A language model is processing the sentence: 'The delivery driver carefully parked the large van.' To understand the context of the word 'parked', the model generates a 'focus' vector for it. This 'focus' vector is then compared against a 'label' vector for every word in the sentence to calculate relevance scores. What is the primary function of the 'label' vector associated with the word 'van' in this process?

Diagnosing Attention Mechanism Failure

In a system designed to focus on relevant parts of an input sentence, each word is assigned a 'label' vector that acts as its identifier. A 'context' vector, representing the current point of focus, is then compared against every word's 'label' vector to calculate a relevance score. If, due to a system error, every word in the sentence 'The quick brown fox jumps' was assigned an identical 'label' vector, what would be the consequence for the calculated relevance scores? Explain your reasoning.

Learn Before

Related