Start with one neuron, expand it into Dense layers, then train a small network on a curved spiral dataset
Brief Intro
A neural network is a model that learns from examples. Instead of writing every rule by hand, we give it data, compare its prediction with the correct answer, and let it adjust its internal numbers until the predictions improve.
The idea is loosely inspired by biological neurons: many small units receive signals, combine them, and pass a result forward. In machine learning, those signals are numbers, and the things being adjusted are weights and biases.
This post builds that idea from the smallest useful piece: one neuron. Once that one piece is clear, every later section only answers one question: how do we make the same idea handle more inputs, more samples, harder shapes, and actual learning?
Companion notebook: src/0.neural_net/0. neural_network_from_scratch.ipynb. The notebook uses pure NumPy, no PyTorch, no autograd.
One Neuron
The smallest neural network unit receives inputs, multiplies them by weights, adds a bias, and produces one output. This is the smallest place where the main ingredients appear: input, weight, bias, prediction, and loss.
With one input, the neuron is just a line:
output = weight * input + bias x = 3 w = 0.8 b = 2 output = w * x + b print(output) # 4.4 - Input: the value the model receives.
- Weight: controls how strongly the input affects the output.
- Bias: shifts the output up or down.
- Output: the prediction from this tiny model.
The weight changes the slope of the line. A larger positive weight makes the output climb faster as the input grows. A negative weight flips the slope downward. The bias shifts the whole line up or down without changing the slope.

A weight changes the slope of the line; a bias shifts the line without changing its slope.
With more than one input feature, one neuron still produces one output. In vector form, the neuron takes a dot product between the input vector and the weight vector, then adds one bias:
output = dot(inputs, weights) + bias Expanded, that same dot product is just a weighted sum:
output = sum(inputs * weights) + bias inputs = [1, 2, 3] weights = [0.2, 0.8, -0.5] bias = 2 output = ( inputs[0] * weights[0] + inputs[1] * weights[1] + inputs[2] * weights[2] + bias ) print(output) # 2.3 Shape view for this one-neuron case:
inputs [D] weights [D] output scalar Drawn as a small computation graph:

Each input is first scaled by its own weight. The neuron then adds those scaled values plus one bias to produce one output.
The model is useful only if we can say how wrong it was. For a simple regression target, squared error is enough:
target = 3.0 loss = (output - target) ** 2 print(loss) # 0.49 Bonus: Why squared error?
The raw error can be positive or negative:
error = output - target If output = 2.3 and target = 3.0, the error is -0.7. If output = 3.7, the error is 0.7. Both predictions are equally wrong, but the signs are different. Squaring removes the sign:
(-0.7) ** 2 = 0.49 ( 0.7) ** 2 = 0.49 Why not use absolute error?
loss = abs(output - target) Absolute error also removes the sign, but it has a sharp corner at 0, so the derivative is not defined exactly at the target. Away from 0, its gradient is always either -1 or 1.
That means the gradient push has the same strength whether the model is very far from the target or already very close. Squared error changes that:
loss = error ** 2 dLoss/dError = 2 * error When the error is large, the gradient is large. When the error gets small, the gradient also gets small. This gives training a natural braking effect near the target.
Why not use a fourth power?
loss = error ** 4 dLoss/dError = 4 * error ** 3 Now the loss becomes too sensitive. A small error like 0.5 gives a tiny gradient, while a large error like 10 gives a gradient of 4000. Squared error is a useful middle ground for this first regression example: it removes the sign, gives smooth gradients, and makes larger mistakes matter more without being as extreme as higher powers.
For classification, we will switch to cross-entropy later. Squared error is only here because it is the simplest loss for understanding one numeric prediction.
The output is the prediction. The loss is the penalty. Training is the process of changing the weights and bias so the loss gets smaller.
The important distinction is that inputs come from data, while weights and biases belong to the model. During training, the input values stay fixed for a given sample; the model changes its weights and biases to make better outputs.
This hand-written neuron is useful because every number is visible. It is also obviously not how we want to write a real model, so the next step is to replace the repeated arithmetic with array operations.
Arrays, Matrices, and Tensors
The previous neuron had three inputs and three weights. A real model may have millions, so the first upgrade is not conceptual; it is notation. The same weighted sum can be written as a dot product between an input vector and a weight vector.
inputs = np.array([1, 2, 3]) weights = np.array([0.2, 0.8, -0.5]) bias = 2 output = inputs @ weights + bias Shape view:
inputs [D] weights [D] output scalar The dot product does exactly what the hand-written neuron did:
inputs @ weights = inputs[0] * weights[0] + inputs[1] * weights[1] + inputs[2] * weights[2] To produce multiple outputs, put many weight vectors side by side in a matrix:
x [D] W [D, H] b [H] out [H] = x @ W + b To process many samples at once, stack the samples into a batch:
X [B, D] # B samples, D features per sample W [D, H] # D input features, H output features b [1, H] # broadcast across the batch out [B, H] = X @ W + b This is the shape convention used through the rest of the post:
Rows are samples. Columns are features. Matrix multiplication has one rule that matters here: the inner dimensions must match.
X @ W [B, D] @ [D, H] -> [B, H] The D dimensions disappear into dot products. The remaining dimensions become the output shape: one row per sample, one column per output feature.
In deep learning code, "tensor" usually means a multidimensional array. A vector is a 1D tensor, a matrix is a 2D tensor, and a batch of images might be a 4D tensor like [B, H, W, C]. Arrays also need to be rectangular along each dimension: every row in a matrix must have the same number of columns, otherwise it is just a ragged Python list and not a useful numeric array.
Once the shapes are written this way, the next abstraction is almost forced: package X @ W + b into a reusable layer.
Dense Layer
A Dense layer is the array version of many neurons sitting side by side. Every neuron receives the same input vector, but each neuron owns a different weight column and a different bias. The layer wraps X @ W + b, takes a batch of vectors, and projects each vector into a new feature space.
For the first pass, ignore gradients and look only at the forward computation.
class Dense: def __init__(self, n_in, n_out, rng, init="xavier"): if init == "he": scale = np.sqrt(2.0 / n_in) elif init == "xavier": scale = np.sqrt(1.0 / n_in) else: raise ValueError(f"unknown init: {init}") self.W = rng.normal(0.0, scale, size=(n_in, n_out)) self.b = np.zeros((1, n_out)) def forward(self, x, training=True): return x @ self.W + self.b For B = 32, D = 2, H = 64:
x [32, 2] W [2, 64] b [1, 64] out [32, 64] 
Caption: a Dense layer is just matrix multiplication plus a bias broadcast across the batch.
-
n_inis the number of input features. -
n_outis the number of output features. -
Wstores one column per output feature. -
bis broadcast across all rows in the batch.
The weight matrix shape is chosen so the forward pass does not need a transpose:
X [B, D] @ W [D, H] -> out [B, H] Some tutorials store weights as [H, D] and call X @ W.T. Both layouts represent the same math. This post stores weights as [D, H] because the forward pass reads directly as X @ W + b.
Weights start as small random numbers so neurons do not all behave identically. Biases usually start at zero because the weights already break symmetry; the bias can learn its offset later.
One Dense layer is already enough to build a linear classifier. Using the Sequential wrapper we define later, a [B, 2] input can map to 3 class scores:
linear_model = Sequential([ Dense(2, 3, np.random.default_rng(100), init="xavier"), ]) X [B, 2] W [2, 3] b [1, 3] logits [B, 3] = X @ W + b The three output numbers are called logits. They are raw class scores, not probabilities. The predicted class is the index of the largest logit:
preds = logits.argmax(axis=1) That gives us the simplest classifier worth testing. Before adding hidden layers, we should see exactly where this linear model breaks.
The First Target: Spiral Data
The Dense layer can produce class scores, but a toy classifier is only interesting if the data exposes its limits. Linear data can be fit by a straight line or a few straight decision regions. Spiral data is still easy to draw, but not easy for straight lines. Each sample is a 2D point, and the label is one of three spiral arms.
def make_spiral(samples_per_class=120, n_classes=3, noise=0.20, seed=42): rng = np.random.default_rng(seed) X = np.zeros((samples_per_class * n_classes, 2), dtype=np.float64) y = np.zeros(samples_per_class * n_classes, dtype=np.int64) for class_id in range(n_classes): ix = slice(class_id * samples_per_class, (class_id + 1) * samples_per_class) r = np.linspace(0.0, 1.0, samples_per_class) t = np.linspace(class_id * 4.0, (class_id + 1) * 4.0, samples_per_class) t = t + rng.normal(0.0, noise, samples_per_class) X[ix] = np.c_[r * np.sin(t), r * np.cos(t)] y[ix] = class_id return X, y The shapes are small:
X [360, 2] # 360 points, each point has x1 and x2 y [360] # integer labels: 0, 1, 2 logits [360, 3] # one score per class Read the plot literally: each dot is one training example, and the color is the label. The word "spiral" only describes the shape of the toy dataset. We use it because a straight line cannot separate the classes, but the data is still small enough to debug.
Each sample has two features:
sample = [x1, x2] target = class_id The colors in the plot are only for us. The model does not see red, green, or blue. It only sees two coordinates and a target label like 0, 1, or 2.
The one-layer linear model exposes the first failure mode:
Linear model: X [B, 2] -> Dense(2, 3) -> logits [B, 3] With the notebook seed, the linear baseline reaches 48.6% validation accuracy (54.5% train accuracy). It learns three broad regions split by straight lines, but the spiral needs a boundary that bends.
The important point: the linear baseline does not fail because the optimizer is bad. It fails because one Dense layer can only draw linear boundaries. A hidden layer gives the model more room, but only if we add something that prevents stacked Dense layers from collapsing back into one linear map.
ReLU: Why Dense Layers Need an Activation
The spiral failure suggests adding a hidden layer. But adding another Dense layer by itself does not solve the problem, because two linear transformations in a row still collapse into one linear transformation.
y = (X @ W1 + b1) @ W2 + b2 = X @ (W1 @ W2) + (b1 @ W2 + b2) = X @ W' + b' Without a nonlinearity, stacking Dense layers only creates a bigger linear model. ReLU breaks that collapse with one small function:
class ReLU: def forward(self, x, training=True): self.mask = x > 0 return np.maximum(0.0, x) def backward(self, dout): return dout * self.mask def parameters(self): return [] ReLU(x) = max(0, x) - Forward: positive values pass through, negative values become
0. - Backward: gradients pass only through positions that were positive during the forward pass.
- Effect: many ReLU units split the input space into small linear regions that can bend around the spiral.
The step function is the simplest "fire or do not fire" activation, but it throws away too much information: every positive input becomes the same 1, and every negative input becomes the same 0. ReLU keeps the useful part of that idea without collapsing all positive values together.
step(-0.1) = 0 step(100.0) = 1 ReLU(-0.1) = 0 ReLU(100.0) = 100.0 
Step only says active or inactive. ReLU keeps the positive magnitude, which gives later layers more useful signal.
That difference matters for learning. A ReLU neuron has an activation region: inputs on one side of its threshold are silent, and inputs on the other side pass through with their size preserved. Many such regions can approximate a curved boundary with piecewise-linear segments.
Our main MLP only adds one hidden Dense layer and one ReLU:
model = Sequential([ Dense(2, 64, np.random.default_rng(101), init="he"), ReLU(), Dense(64, 3, np.random.default_rng(102), init="xavier"), ]) X [B, 2] Dense1 [B, 64] ReLU [B, 64] Dense2 [B, 3] logits [B, 3] The first Dense layer creates 64 hidden features. ReLU gates those features. The second Dense layer mixes the gated features into 3 class scores.
ReLU gives the model the shape it was missing. The next problem is different: the model now returns three raw scores per point, so we need a clean way to read those scores as class predictions and training loss.
Softmax: From Logits to Class Probabilities
The ReLU MLP ends with logits: raw class scores. For prediction, we only need the largest logit:
def accuracy(model, X, y): logits = model.forward(X, training=False) preds = logits.argmax(axis=1) return (preds == y).mean() Softmax is useful when we want class probabilities and a classification loss. It does two things:
- Exponentiate logits so every value becomes positive.
- Normalize each row so the values sum to 1.
shifted = logits - logits.max(axis=1, keepdims=True) exp_scores = np.exp(shifted) probs = exp_scores / exp_scores.sum(axis=1, keepdims=True) logits [B, C] probs [B, C] y [B] Each row of probs sums to 1. The largest probability points to the predicted class, but the probabilities also tell us how confident the model was.
logits row: [4.8, 1.2, 2.4] softmax row: [0.90, 0.02, 0.08] prediction: class 0 
Softmax keeps the winning class the same, but converts raw scores into probabilities.
The exponential is monotonic: a larger logit still becomes a larger exponentiated value. That means softmax does not change which class wins; it changes raw scores into comparable confidence values.
Cross-entropy penalizes the model when the correct class gets low probability:
correct_probs = probs[np.arange(len(y)), y] loss = -np.log(correct_probs + 1e-12).mean() For sparse labels, y contains class indices:
probs: sample 0 -> [0.70, 0.10, 0.20] sample 1 -> [0.10, 0.50, 0.40] sample 2 -> [0.02, 0.90, 0.08] y = [0, 1, 1] correct_probs = [0.70, 0.50, 0.90] losses = -log(correct_probs) Cross-entropy only needs the probability assigned to the correct class. If the correct class gets probability near 1, the loss is near 0. If it gets probability near 0, the loss becomes very large.
The subtraction in stable softmax is a numerical trick:
softmax([1000, 1001, 1002]) == softmax([-2, -1, 0]) Softmax contains exp, so large logits can overflow. Subtracting the per-sample max keeps the numbers safe without changing the probabilities.
In the loss, we also clip or add a tiny epsilon around probabilities before log. The reason is practical: -log(0) is infinite, and one infinite value can ruin the whole training step.
Now we have a scalar loss. The loss tells us how wrong the model is; backpropagation tells us how each weight should move to reduce it.
Backpropagation
The forward pass produced logits, and cross-entropy turned those logits into one loss number. Training needs the reverse path: how did each parameter affect that loss? A gradient is that sensitivity: if this weight moves a little, how does the loss move?
Start with a one-variable function:
y = f(x) The derivative tells us the slope at a point. For a line, the slope is constant. For a curve, the slope depends on where we measure it. In training, the "curve" is the loss as a function of every weight and bias in the model.
Neural networks have many parameters, so we need partial derivatives:
dLoss/dW1 how much W1 affected the loss dLoss/db1 how much b1 affected the loss dLoss/dX how much the layer input affected the loss Backpropagation is chain rule bookkeeping. The model is a chain of functions:
X -> Dense1 -> ReLU -> Dense2 -> SoftmaxCrossEntropy -> loss graph LR X["X<br/>[B, 2]"] --> D1["Dense1<br/>[B, 64]"] D1 --> R["ReLU<br/>[B, 64]"] R --> D2["Dense2<br/>logits [B, 3]"] D2 --> L["SoftmaxCrossEntropy<br/>loss scalar"] L -. "dlogits [B, 3]" .-> D2B["Dense2.backward"] D2B -. "dhidden [B, 64]" .-> RB["ReLU.backward"] RB -. "dhidden [B, 64]" .-> D1B["Dense1.backward"] style X fill:#DBEAFE,stroke:#3B82F6,color:#111827 style D1 fill:#DCFCE7,stroke:#10B981,color:#111827 style R fill:#DCFCE7,stroke:#10B981,color:#111827 style D2 fill:#DCFCE7,stroke:#10B981,color:#111827 style L fill:#FEE2E2,stroke:#EF4444,color:#111827 style D2B fill:#FEF3C7,stroke:#F59E0B,color:#111827 style RB fill:#FEF3C7,stroke:#F59E0B,color:#111827 style D1B fill:#FEF3C7,stroke:#F59E0B,color:#111827 Caption: forward values move left to right. Gradients start at the loss and move backward through the same layers in reverse order.
The gradient starts at the loss and moves backward through that chain. Each layer receives an incoming gradient, multiplies it by its local derivative, stores gradients for its own parameters, then returns the gradient for the previous layer.
That gives each layer two concrete jobs:
1. Compute gradients for its own parameters. 2. Return the gradient for the layer before it. In code, trainable arrays are easier to manage if each one stores both its value and gradient:
class Parameter: """Trainable array plus its gradient.""" def __init__(self, value): self.value = np.asarray(value, dtype=np.float64) self.grad = np.zeros_like(self.value) def zero_grad(self): self.grad.fill(0.0) Now the Dense layer can cache its input during the forward pass and reuse it during the backward pass:
class Dense: def __init__(self, n_in, n_out, rng, init="he"): if init == "he": scale = np.sqrt(2.0 / n_in) elif init == "xavier": scale = np.sqrt(1.0 / n_in) else: raise ValueError(f"unknown init: {init}") self.W = Parameter(rng.normal(0.0, scale, size=(n_in, n_out))) self.b = Parameter(np.zeros((1, n_out))) def forward(self, x, training=True): self.x = x return x @ self.W.value + self.b.value def backward(self, dout): self.W.grad[..Source:https://dev.to/backpropolis/neural-network-from-scratch-from-one-neuron-to-a-tiny-mlp-in-numpy-38ghRelated posts
Databricks is a Leader and positioned highest in execution in the 2026 Gartner® Magic Quadrant™ for Data Lakehouse Platforms
Tabular foundation models on Databricks AI Runtime
Introducing the Databricks AI Index: The state of enterprise AI tokenomics

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient



