How Embeddings Represent Information

Consider 2 sentences: “Sit beside the river bank” and “Deposit money at the bank.” In our classroom tokenizer, bank is 1 unit with ID 417 in both sentences. The model needs to use its surroundings to distinguish the situations. How can the same identifier lead to a representation that depends on the sentence?

The last lesson established what an identifier does: it selects an entry in a vocabulary. The next step is an embedding lookup. An embedding is an ordered list of learned numerical values, called a vector. The lookup uses the token ID to select a row

from the model’s embedding table. That row supplies numbers the model can transform through calculation. Suppose the bank row in a tiny teaching table contains 0.2, negative 0.4, and 0.7. ID 417 selects that 3-coordinate vector. Another selected row can

contain a different vector. The row numbers identify entries; the coordinates are the representation used in subsequent computation. These supplied numbers help us follow the lookup. They are not a measurement of a production model or a claim that 1 coordinate literally means river or money.

2 dimensions of the table matter. Vocabulary size determines how many token entries it can contain. Embedding dimension determines the number of coordinates in each row. With 50,000 entries and 1,000 coordinates per entry, a simple table holds 50,000,000 scalar values.

Multiplying the row count by the coordinates per row gives that result. This counts 1 matrix. The full model contains other learned transformations, so this calculation does not give its total parameter count. Keep the bank row and the checkpoint fixed while we change the surrounding sentence.

The initial token embedding can stay the same, while the model’s layers combine information under the architecture and produce context-dependent representations. An intermediate representation is called a hidden state. At the bank position, a later hidden state can depend on the surrounding river words or payment words.

The model is computing for the supplied sequence; this example has not performed a training update. The sequence must also preserve order. “The dog chased the cat” and “The cat chased the dog” contain the same familiar words but describe different relationships. Position information helps the model

distinguish where each unit occurs. Depending on the architecture, position can be represented or applied in different ways. When we follow the layers, we need the token representations, their order, and the rules governing which positions can interact. A dated architecture example makes that concrete.

Devlin and colleagues introduced BERT in October 2018 and published the paper at NAACL in June 2019. Its input representation adds 3 kinds of vector: the token’s learned representation, a vector marking which input text segment it belongs to, and its position. Its bidirectional encoder uses

context from both sides. The input construction and the later hidden states therefore perform different jobs. We will compare this encoder’s information boundary with causal generation later. Adding position information does not give a coordinate an automatic human label. In a trained model, useful patterns can

be distributed across many coordinates and transformations. A particular coordinate might participate in several relationships. To argue that it consistently represents a named concept would require a separate investigation. For this lesson, follow which object is a vocabulary identity, which is an input vector, and which

is the state produced after contextual computation. There is another representation boundary worth keeping clear. A retrieval system may turn a whole sentence or document into a vector for comparing it with a query. That representation needs an appropriate training objective and a way to

combine information across the input. An internal token state in a language model is produced for a different role. Its presence alone does not establish a useful document-similarity system. A retrieval test should check the intended corpus and query task. Imagine searching a collection containing a

river-bank description and a bank-loan document. A question about erosion calls for the river document; a question about loan interest calls for the financial document. The retrieval application needs a representation and comparison rule suited to those selections. Inside a language model, we may instead inspect the state at 1 bank

token while it processes a supplied sentence. The objects differ in what they represent and how the application will use them. Calling both an embedding does not specify those choices. Pause for a checkpoint. In our supplied example, ID 417 selects the same initial

bank row for 2 different sentences. Must its final hidden state also be identical? Separately, what does a 50,000 by 1,000 embedding table contain? Explain what each number counts before continuing. The initial lookup can be identical while the later state changes with

context and position under the architecture. The table has 50,000 token entries, each with 1,000 coordinates, for 50,000,000 scalar values. That is the size of the defined embedding matrix. It leaves other model parameters and runtime state outside the calculation. For a separate practice task,

a simple embedding table has 30,000 entries with 512 coordinates each. Calculate its scalar-entry count. A colleague then says token ID 418 must be semantically closer to 417 than to ID 900. Explain what evidence would be needed for a similarity claim.

Pause and use the blank worksheet. The table contains 15,360,000 scalar values. The neighboring identifier values establish no semantic distance. To study similarity, identify the actual learned representations, the comparison rule, and the relevant task, then test the relationship.

The IDs alone supply none of those measurements. We can now follow bank from an identifier into a learned row, then through order-sensitive contextual computation into a hidden state. We can also ask whether a vector represents 1 token inside a model or a whole item for retrieval.

The next question concerns what survives the request: which numerical values belong to the saved model, and which values were computed while it processed this input?