From Lab to Code #3: Building an Interactive Robot Cell with Python and Three.js

In an automated production line, a robotic arm might look like the most interesting part of the system.

But moving the arm is only one part of the problem.

How does the robot know when an object has arrived? How does it decide where to place it? And how do the conveyor, sensors, gripper, and controller coordinate their actions?

More importantly, what happens when a sensor is disconnected or connected to the wrong input?

These questions involve several fundamental concepts in robotics and industrial automation: kinematics, finite state machines, sensor feedback, and digital I/O communication.

For the third project in my From Lab to Code series, I decided to explore these concepts by building a small interactive robot-cell simulation.

The result is Robot Cell Playground, a Python-controlled robotic sorting cell with a 3D interface built using Flask, HTML, CSS, JavaScript, and Three.js.

The interesting part is not just watching a robot sort colored boxes. Users can modify the digital connections between components, introduce faults, and observe how those changes affect the entire system.

The Idea: A Small Automated Sorting Cell

The simulation represents a simple production scenario.

A conveyor transports colored boxes toward a robotic arm. Once a box reaches the pickup position, the controller uses simulated sensors to identify it and determine its destination.

The robot then picks up the box, transfers it to the correct location, and returns to its initial position.

There are three possible destinations: red, green, and blue.

Under normal conditions, each box should reach the destination corresponding to its color.

To make this possible, the application combines several components:

• A three-degree-of-freedom robotic arm.
• A conveyor and object generator.
• Idealized position and color sensors.
• A finite-state-machine controller.
• Configurable digital input/output connections.
• A 3D visualization of the complete cell.

The project uses a simplified geometric model rather than a full industrial physics simulator.

This was a deliberate decision. I wanted to focus on understanding how the individual components interact without introducing the complexity of a complete robotics framework.

1. Robot Kinematics: From Coordinates to Joint Angles

Before controlling a robot, we need a mathematical representation of its movement.

In robotics, kinematics describes positions, orientations, and motion without considering the forces that produce them.

Two important problems are forward kinematics and inverse kinematics.

Forward Kinematics

Forward kinematics answers a straightforward question:

If we know the robot's joint angles, where is its end effector?

For this project, I implemented a simplified robotic arm with three degrees of freedom:

1. Base rotation (yaw).
2. Shoulder rotation.
3. Elbow rotation.

The arm consists of two links moving in a vertical plane, while the base rotates that plane around the vertical axis.

Let the upper-arm length be L1, the forearm length be L2, and the joint angles be q1, q2, and q3.

The horizontal distance from the base is:

r = L1*cos(q2) + L2*cos(q2 + q3) 

The vertical position is:

z = shoulder_height + L1*sin(q2) + L2*sin(q2 + q3) 

Using the base rotation, we obtain the horizontal coordinates:

x = base_x + r*cos(q1) y = base_y + r*sin(q1) 

The corresponding Python implementation is relatively compact:

def forward_kinematics(angles): yaw, shoulder, elbow = angles r = ( UPPER_ARM * math.cos(shoulder) + FOREARM * math.cos(shoulder + elbow) ) z = ( SHOULDER_HEIGHT + UPPER_ARM * math.sin(shoulder) + FOREARM * math.sin(shoulder + elbow) ) return ( BASE_X + r * math.cos(yaw), BASE_Y + r * math.sin(yaw), z, ) 

This allows the simulation to calculate the wrist position from the current joint configuration.

Inverse Kinematics

The opposite problem is more useful when commanding the robot to move somewhere.

Suppose a box has arrived at a known pickup coordinate.

We know where the robot should move, but we do not yet know the joint angles required to reach that position.

This is the inverse kinematics problem.

For the two-link arm, we can calculate the elbow angle using the law of cosines.

First, we determine the horizontal distance r and height h relative to the shoulder:

r = sqrt((x - base_x)^2 + (y - base_y)^2) h = z - shoulder_height 

Then:

cos(q3) = (r^2 + h^2 - L1^2 - L2^2) / (2*L1*L2) q3 = -acos(cos(q3)) 

Once the elbow angle is known, the shoulder angle can be calculated:

q2 = atan2(h, r) - atan2(L2*sin(q3), L1 + L2*cos(q3)) 

Finally:

q1 = atan2(y - base_y, x - base_x) 

The implementation selects one elbow configuration and checks whether the requested target lies within the robot's geometric workspace.

A useful test is to apply inverse kinematics to a target and then reconstruct the position using forward kinematics.

target = (0.72, 1.05, 0.92) angles = inverse_kinematics(target) reconstructed = forward_kinematics(angles) 

The reconstructed coordinates should match the target within floating-point precision.

This gives us a simple mathematical consistency check.

Of course, reaching a coordinate geometrically does not mean the movement is physically safe. Real robots must also consider collisions, mechanical joint limits, actuator dynamics, and other constraints.

For this project, the geometric model is enough to explore the relationship between joint angles and target positions.

2. Finite State Machines: Teaching the Robot What to Do Next

Kinematics tells us how to reach a position.

But it does not tell us when the robot should move there.

A sorting cell must coordinate multiple operations in a specific order.

The robot should not attempt to pick up a box before it arrives. It should not transfer the box before activating the gripper. And it should not generate a new box while the previous sorting operation is still in progress.

For this, I implemented a finite state machine (FSM).

An FSM represents a system using a finite number of states and transitions between them.

In Robot Cell Playground, the normal operating sequence is:

IDLE | v CONVEYING | v DETECTED | v APPROACH | v PICKING | v LIFTING | v TRANSFER | v PLACING | v RETRACTING | v RETURNING | +------> IDLE 

Each state has a specific responsibility.

For example, CONVEYING advances the box toward the pickup point, while DETECTED confirms that the necessary sensor information has been received.

The movement states handle approaching, picking, lifting, transferring, and placing the box.

A separate FAULT state handles certain failed operations.

Implementing the State Machine

The simulation is controlled by the RobotCell class in Python.

The controller maintains the current phase, box position, robot angles, sensor information, wiring configuration, and sorting statistics.

A particularly interesting part is the conveyor logic:

if self.phase == "CONVEYING": if self._inputs()["Belt.OnOff"] and self.box is not None: self.box["x"] = min( PICKUP_X, self.box["x"] + 1.20 * dt ) inputs = self._inputs() if ( inputs["Robot.ItemAvailable"] and item_present(self.box) ): matched = [ c for c in COLORS if inputs[f"Robot.Color{c.capitalize()}"] ] if len(matched) == 1: self.detected_color = matched[0] self.phase = "DETECTED" 

Notice that the conveyor does not move unconditionally.

It moves only when the corresponding digital input is active.

Similarly, the controller proceeds to the detection state only when an object is available and exactly one color input is active.

These conditions become particularly important when we start modifying the wiring.

Making the Motion Smooth

For the robot's joint movements, I used a cubic smoothstep interpolation:

s(t) = 3t² - 2t³ 

where t varies from 0 to 1 during a movement phase.

Each joint angle is interpolated as:

q(t) = q_start + s(t)*(q_target - q_start) 

This produces smoother visual transitions than immediately jumping between configurations.

It is not a physically constrained trajectory planner, but it provides a simple way to animate the robot while keeping the motion calculations inside the Python controller.

The important distinction is that kinematics determines where the robot should move, while the state machine determines what the robot should do next.

3. Digital I/O: Connecting the Components

This became one of the most interesting parts of the project.

An automated robot cell consists of multiple components that need to communicate.

The conveyor needs to receive a start command. The robot needs to know when a box arrives. The gripper must receive an activation signal, and the controller needs information from the color sensor.

In this simulation, these interactions are represented using digital input/output signals.

A digital signal has two possible logical values:

LOW = 0 HIGH = 1 

A signal travels from an output to an input.

For example:

Robot.StartStop ------> Belt.OnOff 

Here, the robot controller produces a command, and the conveyor receives it.

The same principle applies to the simulated gripper:

Robot.Grasp ------> Gripper.Vacuum 

Sensor communication flows in the other direction:

PositionSensor.Detect | v Robot.ItemAvailable 

In this case, the sensor provides the output, while the robot controller receives it.

Making the Connections Editable

Instead of hardcoding every connection directly into the controller, I implemented an independent digital I/O routing system.

The application stores a mapping between device inputs and their connected outputs.

For example:

DEFAULT_CONNECTIONS = { "Source.Trigger": "Robot.Generate", "Belt.OnOff": "Robot.StartStop", "Gripper.Vacuum": "Robot.Grasp", "Robot.ItemAvailable": "PositionSensor.Detect", "Robot.ColorRed": "ColorSensor.Red", "Robot.ColorGreen": "ColorSensor.Green", "Robot.ColorBlue": "ColorSensor.Blue", } 

This is a shortened version of the default configuration. The complete system also includes signals for the three sorting destinations.

The key is that the controller does not directly read the physical source signals.

Instead, it receives values through the current connection mapping.

The IOBoard class resolves these connections using:

def resolve(self, outputs: dict[str, bool]) -> dict[str, bool]: return { p.id: bool( outputs.get(self.routes.get(p.id, ""), False) ) for p in INPUTS } 

For each input, the function finds the connected output and retrieves its logical value.

If the input has no valid connection, its value becomes False.

This introduces an important distinction:

A controller output can be HIGH while the corresponding device input remains LOW.

For example, the robot might send a command to start the conveyor, but the conveyor will not respond if the connection is missing.

The Interactive Wiring Lab

To make this behavior visible, I added an interactive wiring panel to the application.

The panel contains ten output ports and ten input ports.

Users can connect outputs to inputs, disconnect existing connections, restore the default configuration, and inspect the live digital signal levels.

The connections are visualized as lines between ports.

HIGH and LOW signals are displayed separately from the connection status, so an inactive signal is not confused with a disconnected wire.

The Python controller also provides diagnostic messages explaining which signals are missing during particular operations.

This is a simplified logical routing model, not a complete PLC programming environment.

However, it gives users a way to explore a central principle of automation:

The behavior of a system depends not only on the commands being generated, but also on how those commands and sensor signals are connected.

4. Fault Injection: What Happens When the Wiring Is Wrong?

Once the connections became editable, I wanted to explore another question:

What happens if we deliberately connect something incorrectly?

This turns the simulation into a small fault-injection environment.

Instead of only observing the correct sorting sequence, we can investigate why the system stops working or produces incorrect results.

Experiment 1: Disconnecting the Conveyor

Under normal conditions:

Robot.StartStop ------> Belt.OnOff 

When the controller enters the CONVEYING state, it activates its StartStop output.

Now imagine disconnecting that connection.

The output still becomes HIGH, but the conveyor input remains LOW.

The result is straightforward: the box stays at the beginning of the conveyor.

The controller remains in the conveying phase because the box never reaches the pickup position.

The interesting part is that the controller still issues the correct command.

The problem is the missing connection between the controller and the device.

Experiment 2: Disconnecting the Position Sensor

Normally, the position sensor sends a detection signal to the robot:

PositionSensor.Detect | v Robot.ItemAvailable 

Now disconnect this signal.

The conveyor still transports the box toward the pickup point.

But when the box arrives, the robot controller does not receive the corresponding detection signal.

As a result, the controller cannot proceed to the pickup operation.

The robot waits even though the box has physically reached the correct location in the simulated scene.

This demonstrates how missing feedback can prevent an otherwise functioning system from progressing.

Experiment 3: Disconnecting the Gripper

The gripper is activated through:

Robot.Grasp ------> Gripper.Vacuum 

If this connection is removed, the robot can still approach the box.

However, when it attempts to pick up the object, the gripper does not receive the activation signal.

In this implementation, the controller recognizes the missing signal and enters the FAULT state.

The simulation must be reset before another sorting cycle can begin.

This is a simplified fault-handling mechanism, but it clearly illustrates the dependency between control commands and actuator inputs.

Experiment 4: Swapping the Color Sensor Connections

This was my favorite experiment.

Imagine that the simulated color sensor correctly identifies a red box.

The sensor's red output becomes HIGH.

Now suppose the red and blue sensor connections are swapped:

ColorSensor.Red | v Robot.ColorBlue ColorSensor.Blue | v Robot.ColorRed 

The actual box is still red.

The color sensor still detects red.

But the controller receives that signal through its blue input.

Consequently, it identifies the object as blue and selects the blue destination.

The robot then picks up the red box and transfers it into the blue bin.

The application records this as a mis-sort.

This is particularly interesting because the robot's movement can be geometrically correct while its overall decision is wrong.

The kinematics are still functioning. The state machine is still progressing. The robot reaches the destination selected by the controller.

The error originates from the information entering the controller through an incorrect connection.

Comparing the Fault Scenarios

These scenarios are also covered by automated tests.

For example, one integration test verifies that a red box incorrectly routed to the blue destination increments the mis-sort counter:

self.assertEqual(cell.missorted, 1) self.assertEqual(cell.pick_count, 0) self.assertEqual(cell.counts["red"], 0) self.assertEqual(cell.bin_contents["blue"]["red"], 1) 

Instead of merely checking whether the application displays an error, the test verifies that the simulated sorting behavior actually changed.

For me, this is one of the most valuable parts of the project.

A small wiring error can affect the outcome of an entire automation process, even when the underlying movement algorithms are functioning correctly.

5. Visualizing the System with Three.js

After implementing the Python controller, I wanted to make its behavior visible and interactive.

For the frontend, I used HTML, CSS, JavaScript, and Three.js.

The backend uses Flask to expose the simulation state to the browser.

One design decision was particularly important: Python remains responsible for the robot's behavior.

The frontend does not independently decide when the robot should move, when a box has been detected, or which destination should be selected.

Instead, it renders the state calculated by Python.

The architecture looks like this:

Browser Interface | | HTTP / JSON v Flask API | v Python Controller | +-- Kinematics +-- State Machine +-- Sensors +-- Digital I/O Routing | v Simulation State | v Three.js Visualization 

You can also inspect the system architecture diagram included in the repository.

The browser periodically sends simulation-step requests to Flask.

Python advances the controller and returns the updated state, including joint positions, box coordinates, sensor values, connection states, and sorting statistics.

Three.js uses this information to update the scene.

The user can observe the robotic arm, conveyor, colored boxes, and sorting destinations in 3D.

The interface also allows camera rotation, zooming, simulation-speed adjustments, and selection of the next generated box color.

Below the 3D scene, the Wiring Lab displays the digital connections and their live signal levels.

An event stream provides additional information about state transitions, detection events, successful sorting operations, and faults.

This separation between simulation and visualization has another advantage: the Python control logic can be tested independently of the browser.

It also makes the system easier to understand. The 3D scene shows what happens, while the controller determines why it happens.

6. Reproducible Results and Testing

An interactive simulation is useful for exploring individual scenarios.

However, I also wanted to verify that the underlying controller behaves consistently without manually operating the interface.

For this reason, I implemented a deterministic demonstration using a fixed random seed.

The simulation generates and sorts twelve colored boxes using the default connections.

The recorded result is:

The random seed for this run was 13.

These values represent simulated outcomes, not measurements from a physical production line.

The elapsed time is measured in simulation time, so it should not be interpreted as an industrial throughput benchmark.

Beyond this demonstration, the project includes unit and integration tests covering kinematics, sensor behavior, sorting cycles, input validation, digital wiring, and fault scenarios.

An important test checks that forward and inverse kinematics produce consistent coordinates.

Other tests verify that missing connections prevent operations from progressing, invalid connections are rejected, and swapped color inputs produce an actual mis-sort.

These tests do not make the application hardware-ready, but they help ensure that the simplified model follows its intended rules.

7. What This Simulation Does Not Model

Robot Cell Playground is an educational simulation, not a physics-accurate industrial robot controller.

The robotic arm has three geometric degrees of freedom and uses analytical kinematics.

The system does not model actuator forces, motor torque, realistic gripper contact, collisions, detailed joint constraints, or obstacle avoidance.

The sensors are idealized. The color sensor returns a discrete color signal instead of processing real camera data.

Similarly, the digital I/O connections model logical Boolean values, without electrical characteristics, signal propagation delays, or PLC scan cycles.

The robot's motion uses smooth joint-angle interpolation rather than a complete industrial trajectory planner.

These limitations are intentional.

The objective is to make specific engineering concepts easier to observe and experiment with, not to reproduce the full complexity of a real manufacturing environment.

What I Learned

This project helped me connect several engineering concepts that are often studied independently.

Robotic movement is only one part of automation.

Forward and inverse kinematics provide a mathematical way to position the robot, but they do not determine when a conveyor should stop, whether a box is available, or which destination the robot should select.

That requires a control strategy.

Finite state machines make the behavior easier to understand.

Breaking the sorting process into distinct states creates clear responsibilities and transition conditions.

It also makes debugging more manageable because the controller's current state provides information about what it is trying to accomplish.

Digital connections are part of the system's behavior.

One of the biggest takeaways came from the Wiring Lab.

A controller may execute its programmed logic correctly and still produce the wrong result if its inputs contain incorrect information.

The swapped-color experiment demonstrates this directly: the robot follows its controller's decision, but the decision itself is based on an incorrectly routed signal.

Fault injection makes hidden dependencies visible.

When everything works, the importance of individual sensor and actuator connections is easy to overlook.

Disconnecting them reveals which operations depend on which signals.

Finally, separating the Python simulation from the Three.js visualization made the project easier to test and reason about.

The controller is responsible for the logic, while the frontend provides an interactive way to observe the results.

Final Thoughts

With Robot Cell Playground, I wanted to explore how relatively simple mathematical and programming concepts can work together to form an automated system.

The project began with a robotic arm and a conveyor, but the more interesting questions appeared when I started considering communication between components.

What happens when the controller receives no signal? What if it receives the wrong one? Can the robot complete a movement correctly but still make an incorrect sorting decision?

By combining kinematics, finite state machines, editable digital I/O connections, and fault injection, the simulation provides a small environment for investigating these questions.

The project is intentionally limited in scope. It is not designed to replace industrial robotics software or operate physical machinery.

Instead, it follows the idea behind From Lab to Code:

Build something small, understand the engineering behind it, make the behavior observable, and share the implementation.

The complete source code, technical documentation, setup instructions, and reproducible experiments are available in the GitHub repository:

Explore Robot Cell Playground on GitHub

The implementation and visualizations are original and independently developed. No university handouts, CIROS assets, proprietary robot models, or restricted instructional materials are redistributed.

Thanks for reading!