When one straight line is too simple
Our first delivery-cost rule drew one straight line. Suppose cost rises gently on clear roads but sharply after congestion begins. A larger network might represent a pattern like that. What is a neural network actually doing, and how does training change its many values?
This is Module one, Lesson twelve: Neural networks and nonlinear transformations. By the end, you should be able to describe a layer in ordinary arithmetic, show what a simple nonlinear activation does, distinguish architecture from parameters, and separate backpropagation from the optimizer's update.
We'll build from the weighted sums of Lesson eleven, add one nonlinear rule, then trace how a training error can guide changes through several layers.
Describe the operations in a layer
A neural network combines transformations arranged in layers. A simple layer receives input values, calculates weighted combinations, adds offsets and applies a selected transformation. The weights and offsets are parameters that can be learned. They influence how information moves through the network.
Keep the historical analogy in scope
The name neural network comes from a historical analogy. This calculation is not a literal miniature brain. It is a structured collection of numerical operations whose behavior depends on its architecture, parameters, data and training objective.
Use intermediate representations
Picture the delivery shop supplying distance and expected travel time. An early layer could combine them into intermediate values. A later layer uses those values to estimate cost. The exact meanings of the intermediate values need not be obvious or stable enough to give each one a human label. They are representations useful to further computation.
Why nonlinear structure matters
If we only stack suitable straight-line, or linear, transformations without relevant nonlinear structure, we do not automatically gain the ability to represent every curved relationship. Nonlinearities help the network model richer patterns.
Apply ReLU to actual values
One simple activation is called ReLU, short for rectified linear unit. Its rule is easy to say: a negative input becomes zero; a positive input stays as it is. Feed it negative three and it returns zero. Feed it four and it returns four. At zero, it returns zero.
Understand boundaries without invented hidden-unit meanings
The rule is simple, but inserting it between weighted transformations changes their combined behavior. A network can respond differently on different sides of a boundary. In a teaching sketch of congestion, one intermediate calculation might become active only after travel conditions cross a point. A real model's learned pattern could be more involved; we should not assume a hidden unit is literally a human-readable traffic detector.
Separate architecture from parameters
Architecture is the arrangement of the computation: how many layers, how they connect and which operations are used. Parameters are values learned within that structure. Choosing more layers is an architecture decision; changing a weight from one value to another during training is a parameter update.
Match arrangements to data structure
Different tasks invite different arrangements. Images, sequences and graphs have different structures, so networks often organize their computations differently. Deep learning refers to networks with multiple layers and learned representations. More depth can expand what can be expressed, but it also affects training and operating resources.
Require reliability evidence
Size is not evidence of understanding or reliability. A large network can still fail under a new condition, reflect flawed data or pursue an objective poorly connected to the task. We need generalization evidence and useful outcomes, as in Lessons six and seven.
Trace output to loss
During training, the network produces an output and the objective calculates loss. That loss depends on calculations throughout the layers. To adjust an early weight, the procedure needs to know how changing it could affect the output and therefore the loss.
Pass gradient information backward
Backpropagation efficiently calculates relevant gradients through the chain of computations. It passes change information backward from the loss toward earlier parameters. The chain rule from calculus underlies the calculation. For this course, keep the operator meaning: backpropagation supplies information about how parameters relate to the chosen loss.
Apply the optimizer update
The optimizer then uses those gradients, and sometimes additional stored state, to choose parameter updates. Backpropagation and optimizer are related stages, not two names for the same action. One computes change information through the network; the other applies an update rule.
Account for computation and stored values
This process can be expensive. More layers and parameters add operations and stored values during training. Memory for activations and optimizer state can matter as well as memory for the learned parameters themselves.
Compare flexible and simpler models
A small neural network may represent a nonlinear pricing relationship better than one straight line. But imagine the shop has only a tiny, noisy dataset. A flexible network may fit accidental details and perform poorly on new deliveries. The simpler line may be easier to validate, explain and operate.
Use held-out cases and operating consequences
Compare the models on relevant held-out cases and on the decisions they support. Consider operating cost, review needs, interpretability and consequences. If a more complex model improves only a metric that does not change useful decisions, the extra resource demand may not be justified.
Connect computation to useful outcomes
Architecture and parameter count tell us what computation is possible. Data quality, training, evaluation and workflow determine whether that computation helps.
Check your understanding
What does backpropagation supply, and what does the optimizer do with it?
Backpropagation calculates relevant gradients through the network's computations. The optimizer uses gradients and possibly additional state to select parameter updates. It is like our one-parameter lesson scaled to a connected system, though real networks require many more calculations.
What does ReLU return for negative three and positive four?
It returns zero for negative three and four for positive four. That nonlinear rule helps a sequence of transformations represent relationships beyond one simple straight line.
Connect the training stages
A neural network stacks transformations; a layer combines inputs with learned values and applies an operation. Nonlinear activations such as ReLU change what the composition can represent. Architecture is the structure, while parameters are the values learned inside it. Backpropagation calculates gradients; an optimizer uses them to update parameters.
Continue to learning signals
Lesson thirteen asks a broader question: what kind of learning signal is available? We'll distinguish learning from explicit targets, finding structure, constructing targets from data, learning from rewards, and learning from demonstrations.