This post is for engineers who build or evaluate camera-based machine monitoring, computer vision specialists among them, and for production and maintenance managers who are considering a camera for machines with no PLC or data connection. A camera aimed at a machine can record when it runs and when it stands idle, with nothing wired into the machine. Our post on machine monitoring without a PLC compares the camera with a current sensor and a stack light.

Here we explain how a camera tracks the moving parts of a machine: how each part is defined without training data and followed in the image, how movement is told from noise, what the camera cannot see and how a tracker is tested. Engineers get the method and its limits; managers get what to ask a supplier and how monitoring is set up, from three stills of the machine to running time, idle time and a log of stoppages for each shift.

The example throughout is a machine we monitor, with three heads that travel along rails. For each head, 15 times a second, the software works out where the head is and whether it is moving. When a movement ends, it records how far the head travelled, how fast, in which direction and for how long. Over a shift, these records add up to running time, idle time and the number of strokes per hour.

From the camera image to shift figures
How the camera image becomes shift figures. On the right, the check that the box is still on the part.

Why motion detection is not enough for machine monitoring

The simplest camera method is motion detection. The software keeps a picture of the scene at rest and marks the pixels that differ from it in each new frame. Security cameras use it to start a recording. For a machine, it answers a different question from the one monitoring asks: it shows that something changed in an area of the image, not which part moved, how far or in which direction.

Two parts in one area give one signal, and so do a part and the machine frame around it. A change of light, a shadow or a forklift driving past the machine also changes pixels, and the detector counts it as motion.

Machine monitoring needs a measurement for each part: its position in every frame, and from that the start, end and length of each movement. That means following the part itself rather than the pixels that change around it. The rest of this post describes how.

Defining a part without training data

The usual way to find an object in an image is a detector, trained on thousands of labelled pictures of the same kind of object. No such pictures exist for the parts of one machine, and every machine is different. Instead, a person defines each part once, on one frame from the camera: they draw a box around the part and pick at least five points on it. Nothing is trained, so the method needs no pictures of other machines of the same kind.

The points have to sit where the surface has detail in two directions, such as the corner of a bracket or the head of a bolt. On flat paint the image looks the same wherever a point moves, and along a straight edge it looks the same when the point slides along the edge, so neither can be followed. The marking tool therefore snaps each click to a nearby spot with detail in two directions. On our machine, each head carries between 8 and 15 points.

Three heads defined on one frame
A frame from the camera above the machine, with the box and points of each head as they were defined. Below, the head in the dashed frame, enlarged.

In principle, two points fix the position and the angle of a rigid part. On a camera image, though, the points on one part lie close together, so an error of a fraction of a pixel in one of them shows up as a large error in the angle. The tracker therefore measures the angle only while at least four points survive, and asks for five or more, so that one hidden point does not stop the measurement.

The frame on which the points are picked is kept as the part's first reference view: what the part looks like, as defined by the person who marked it. More views of the part, at another time of day or in another place on its rails, can be added. The tracker checks against them that the box is still on the part.

The definition covers only what the camera sees. A part with no corners, such as a smooth painted cover, gives the points nothing to hold on to.

Following the part from frame to frame

Between two frames, each point is followed by optical flow: the software looks for the small shift that makes the pixels around the point in the new frame best match the pixels around it in the previous one. The method used, Lucas-Kanade, assumes that neighbouring pixels move together, which holds for a small patch on a rigid part. It finds only small shifts, so it works first on a reduced copy of the image, where a large move becomes a small one, and then refines the result on the full image.

Optical flow says where the texture under a point went, not whether the match is real. Each point is therefore also followed backwards, from the new frame to the old one. If it does not come back to within half a pixel of where it started, the point is left out for that frame.

The surviving points are then fitted together as one rigid body: the shift and turn that best carry them from where they were picked to where they are now. A point that disagrees with the rest, because it slipped onto the machine frame behind the part or was hidden, is put back where the others say it belongs. The box is attached to this fit. It keeps its drawn size, turns with the part, and its centre is the position that is measured.

The three heads tracked
Video in the original post.
Five stretches from the camera above the machine, recorded on two days between 08:04 and 17:00, shown five times faster than real time, as the tracker followed the heads. A green box is moving, an orange box is at rest, and the line behind a box is its recent path.

Fitting all the points to the frame where they were picked keeps errors from adding up. Moving the box by each frame's average shift instead lets a small error in every frame accumulate, until the box and its own points end up tens of pixels apart.

Following points cannot tell what they sit on. So in every frame the tracker also compares the image under the box with the part's reference views. If nothing matches them, it searches the image for the part; after five seconds without a match, it flags the part as unconfirmed and moves the box to where the search finds the part.

Telling movement from noise

Even a part that stands still does not stay on exactly the same pixel. Image noise, changes of light and video compression shift its measured position by a few hundredths of a pixel from frame to frame. A simple rule, moving when the position changes by more than a set amount in one frame, has to choose: set high, it misses a part that creeps slowly; set low, it fires on noise.

The tracker instead adds up the part's movement frame by frame and compares the sum with the noise. Noise points one way, then another, so its sum grows slowly, with the square root of the number of frames. A real movement points the same way in every frame, so its sum grows in step with the number of frames. The part counts as moving when the sum passes four times the spread that noise alone would give it over that many frames. A slow creep passes this limit after enough frames; noise does not. The noise level itself is measured all the time, from how much the part's own points disagree with each other.

The part is at rest again when its movement over the last two seconds stays within what noise alone would give, so two seconds of stillness end a movement.

Movement added up against the noise limit
Head 3 over seven seconds, measured. The vertical axis is how far the head has moved in total since the sum last started again, in pixels, on a scale where each grid line is ten times the one below. At rest, the sum stays under the noise limit and starts again every two seconds. When the head starts a stroke, the sum passes the limit at once.

One more rule guards against very low noise. When the noise is very low, the limit is very low too, and a nudge of a fraction of a pixel can open a movement. Points that sit on a static patch of the machine can then log dozens of tiny movements every half hour.

A movement counts only once the part has travelled 2 pixels of the analysed image at its furthest point, so that a stroke that goes out and comes back still counts. That removes most of the jitter and nothing a person would call a movement: the heads' real strokes are 60 to 100 pixels.

The figures are in pixels of the image. How many millimetres a pixel is on the machine depends on the camera's distance and lens, and is not measured here.

What the camera does not show

The camera sees only what is in the picture, and only as clearly as the picture allows. These are the limits we have met:

  • A part that moves further between two frames than the flow can follow is lost for a moment and has to be found again by the search. Machine parts are usually slow enough; thrown or swung objects are not.
  • While something stands in front of a part, the camera cannot tell whether the part moved.
  • A second part that looks the same, such as an identical head on the same rail, can be taken for the part.
  • In poor light, at dusk or at night, parts are harder to recognise, and a look-alike is easier to mistake for them.
  • The measurements are in pixels. Turning them into millimetres needs the camera's distance and lens, as in our machine vision lens calculator.
  • When a machine stops for a reason outside the picture, such as a busy crane or missing material, the camera records the stop but not its cause.

How to check a tracker

A tracker cannot be judged by its own output, because its own check can pass while the box sits in the wrong place. It has to be compared with something it did not produce.

We compare ours with a reference that does not depend on it. A second, simpler method follows each part on every frame, using none of the tracker's rules, and leaves out any frame where it is unsure. A person then checks only a few frames per part by eye, chosen where mistakes are most likely: at rest, in the largest steps and where movements start. Nobody has to mark every frame by hand. A plain measure of where pixels changed inside each head's area, also worked out without the tracker, shows when a box reports rest while the head is moving.

Three questions are worth asking about any camera-based monitoring:

  • Does the system say when it is unsure, or does it always give a number?
  • Was it checked against a reference it did not produce, on machines like yours?
  • What does it not see, and how will you know when that happens?

Three camera approaches compared

There are three common ways to follow a machine with a camera: motion detection, as built into most video recorders; a detector trained on labelled pictures, with a tracker that links its boxes from frame to frame; and a part tracker like ours, as described in this post.

No. Feature Motion detection Trained detector and tracker Our part tracker
1 Which part moved, and how far No; it shows only that pixels changed in an area Yes, for the kinds of object it was trained on Yes, for each part named on the machine
2 Training images None needed Many labelled pictures of each kind of object None; one frame, a box and at least five points per part
3 Slow, small movements Missed, or mixed up with noise Hard to tell from the box's own jitter Added up and compared with the measured noise
4 Says when it is unsure No Gives a confidence score for each box Flags a part as unconfirmed after five seconds without a match
5 Computer needed Usually built into the camera or recorder Usually one with a graphics card An ordinary computer

Motion detection is enough to see that something moved. To measure which part moved, how far and when it stopped, on a machine nobody has trained a detector for, each part has to be defined and followed on its own.

From tracks to shift figures

Each movement the tracker records has a start, an end and a length. Added up over a shift, they give each part's running time, its idle time and its strokes per hour. Each stop becomes an entry in the log of stoppages, with its time, its duration and a video clip of the machine during the stop. Availability and OEE (overall equipment effectiveness) follow from there, as in our OEE calculator.

Setting up machine monitoring with a camera
How machine monitoring with a camera is set up, in five steps. The shift bar in step 5 is an illustration.

The place to start is one machine, the bottleneck, because it decides how much the plant produces. The camera on it shows where its time goes and which stop to fix first. What a fix is worth depends on the cost of an hour at that machine; our post on 13 minutes a shift works it out for an example.

For machines without a data interface, our machine monitoring provides running and idle time for each shift and a log of stoppages with time, duration and a video clip.

Sources

  1. Shi, J., Tomasi, C. (1993). Good Features to Track. Technical Report TR 93-1399, Cornell University. https://users.cs.duke.edu/~tomasi/papers/shi/TR_93-1399_Cornell.pdf
  2. OpenCV (2026). Optical Flow. OpenCV 4.13.0 documentation, tutorials. https://docs.opencv.org/4.x/d4/dee/tutorial_optical_flow.html