Dense Models And Mixture Of Experts Explained

A deployment proposal advertises a model with a large total parameter count and a much smaller active count. The team budgets memory from the smaller number. Will the model actually fit? Follow a token through a router, the selected parameter blocks, the weights that remain

accessible, and the devices serving a workload. This will separate reduced computation for a token from the resources needed to operate the complete model. A dense model uses its relevant dense parameter blocks for each processed token under the architecture. A sparse mixture of experts, usually shortened

to MoE, provides multiple parameter blocks and selects a subset for a token. Here expert names a computational component. It does not mean a human professional or a separately verified specialist in medicine, law or coding. Use a supplied architectural sketch with 8 feedforward experts at a layer.

A router takes the current token representation and selects experts 2 and 6 for this token. Each selected block transforms that representation, and their outputs are combined under the routing rule. The next token, or the same sequence at another layer,

can have a different selection. Our numbered route is illustrative, not an observed routing trace. Mixtral's January 2024 paper provides a historical concrete architecture: 2 of 8 feedforward experts are selected for each token at each relevant layer, and their outputs are combined. It reports roughly 47 billion

total parameters and 13 billion active parameters during inference under its convention. These figures describe that model. They are not calculated by multiplying the model's product-name numbers or by assuming every parameter belongs to an expert block. Keep the feedforward routing distinct from output-token decoding.

The router selects internal computation for the current representation. It does not choose the next word from the vocabulary. A decoder's later output projection and selection rule still have their own roles. Some other parts of the transformer are shared rather than being independently

chosen from these 8 feedforward experts. Total parameters count the relevant parameter collection. Active parameters count those used for a token under the reporting convention. Selecting 2 expert blocks can reduce the expert computation performed for that token compared with executing every block. The unselected blocks still belong

to the model and may be selected by other tokens. They do not disappear from the deployment because this particular token did not use them. For a supplied memory illustration, suppose a complete model has 60 billion parameters stored at 2 bytes each.

The raw weight storage is 120 billion bytes, or 120 gigabytes in decimal units. If its active count is 12 billion, multiplying that smaller count by 2 gives 24 gigabytes for that selected parameter amount. It does not establish the weight storage for the whole model. This example is hypothetical and

excludes runtime state and storage overhead. A deployment must make the relevant weights accessible. They might reside across multiple devices, or a designed system might move weights between storage and accelerators. Those arrangements have resource and latency costs. Add the temporary key/value state

from the previous lesson, activations, buffers and other runtime requirements before judging whether a service fits its hardware. Active count alone supplies none of those complete measurements. Expert parallelism places different experts on different devices. Tokens routed to an expert must

reach the device that hosts it, and results must return to the continuing computation. In a supplied workload sketch, many tokens choose expert 2 while another device's expert receives few. The busy device can become a bottleneck while capacity elsewhere sits underused. Communication and load balancing now matter

alongside the arithmetic performed inside each expert. The word expert also invites a semantic shortcut. Mixtral's authors examined routing across data subsets and did not observe obvious topic-based assignments in that limited analysis. To claim that a particular component is the biology expert, define a

behavioral question and inspect evidence rather than assign a profession by its number. For the deployment decision, compare task quality, memory, communication, throughput, latency and operating complexity under the actual workload and configuration. Pause at the 60-billion-total, 12-billion-active memory example with 2 bytes per parameter. Calculate raw storage for all

weights and for the active parameter amount in decimal gigabytes. Which quantity answers the whole-weight question, and what additional information is needed to decide whether the service fits? The full parameter collection requires 120 gigabytes of raw weight storage at the supplied precision.

The active amount corresponds to 24 gigabytes, which is not a whole-model memory estimate. Inspect how all needed weights are stored or moved, and add cache, activations, buffers and overhead. The model may be distributed, but that requires an actual deployment design rather than making the missing weights vanish.

Try a new routing case. 2 devices host different expert blocks. A batch sends most of its tokens to an expert on device A, while device B receives little work. A colleague says a lower active parameter count guarantees faster service than a dense alternative.

Identify the likely bottleneck and the measurements needed to test the claim without inventing a speed result. Uneven routing can overload device A while device B remains underused. Transfers to and from the expert can add communication cost. Compare the actual workload's task quality,

memory, throughput and latency with the dense alternative on specified hardware and batch settings. A lower active count can reduce some computation, but the routing, hardware and runtime measurements determine this service comparison. No measured winner is supplied here. The proposal's smaller active count describes a part of the computational work,

while the total collection and runtime design determine wider deployment needs. You can now explain the route, retained weights and possible workload bottleneck rather than read a parameter headline as a hardware plan. Next we will consider another way computation can change: spending more

work during an answer to propose, check or revise a result.