The Problem
The robot classifies what is inside a sealed cup entirely by touch. Because the lid blocks vision, the arm picks up the cup, swirls it, and identifies the contents by measuring the resulting joint-torque dynamics.
Live dashboard: swirl, predict, and teach new categories in real time.
Classifying Without a Force-Torque Sensor
External force-torque sensors add cost and complexity. The system classifies contents from the arm's built-in joint-torque telemetry alone.
What Makes Classification Hard
- Baseline drift: Temperature changes and subtle pose shifts cause baseline torques to wander over time. Because static offsets are noisy and unreliable, only dynamic motion yields a clean differential signal.
- No static payload signal: At this pose, resting hold torques are identical across all contents. Every discriminative cue comes from how the mass shifts during the swirl.
How it works
Core Intuition
Mapping direct force and torque measurements to an object class (liquid, solid, empty) is straightforward. In the absence of an external force-torque sensor, however, the system exploits a latent signal: variations in the payload held by the end-effector induce measurable differences in joint torques throughout a trajectory.
Consider an arm moving in a single dimension: increasing the payload's weight directly increases the torque required to execute that motion. Feels expands on this principle — classifying the hidden contents of a sealed cup strictly through joint-torque dynamics during a fixed motion primitive.
Sensing With Built-in Hardware
Every servo arm naturally reports joint loads through its motor controllers. Analyzing this existing telemetry channel, the pipeline classifies materials from built-in hardware. An external 6-axis force-torque sensor would add $5k+, wrist integration, and calibration.
Deterministic Swirl Motion
The robot executes a single, repeatable sequence (pick up → swirl → place) to ensure every dynamic recording is strictly comparable. Torque telemetry arrives at 4–5 Hz via cloud polling — roughly 10 samples per swirl — so the motion is designed to produce strong low-frequency momentum and drag signals that survive the sparse sampling.
The motion itself: pick up → swirl → place, filmed at the hackathon.
recording_20261029_190400_water_075pct.csv (35.3 s).
Top: TCP path per stage (faint = full path, blue = stage). Bottom: TCP
height with stages shaded. Boundaries from the CSV phase column.System Architecture
atlas.predict_lda decides — Atlas $vectorSearch supplies
the neighbors and the plot. Bottom lane: Mode 2 and teach_csv.py write
taught signatures into the signatures collection.10-Dimensional Physical Signature
The pipeline translates the raw torque time series directly into a 10-D vector, mapping every extracted feature to a distinct, observable physical property.
| Group | Features | What they capture |
|---|---|---|
| Static lift | mass_j2, mass_j3*, lift_dyn_j3 | Gravity torque offsets = mass estimate; inertial cue from the lift |
| Swirl dynamics | swirl_var_j2, swirl_var_j3, swirl_ptp_j3, phase_lag_s, rattle_e | Slosh energy; shifting-mass swing; TCP–torque phase lag = viscous damping; rattle energy (observable band only) |
| Grip & excitation | grip_comp, swirl_speed* | Gripper compliance under load; TCP swirl speed for excitation normalization |
* Zero-weighted in the embedding — the static offsets sit below baseline noise, and swirl speed measures the fixed macro's drive rather than the contents. The matcher works in 7 dimensions.
The extractor scales and L2-normalizes these features before matching. By zero-weighting static offsets, the matcher evaluates active dynamics across 7 informative dimensions.
Teaching and the Library
The system learns new items immediately: one stored embedding updates the library, and the next query searches it.
Teach the items the demo needs via the dashboard's Mode 2 class dropdown.
Record 2–3 repetitions per item;
repeats let the matcher average out run-to-run noise. Acceptance: the
SWIRL phase is present and the baseline flags are clean —
the tool refuses the rest.
The library holds the taught signatures — each a 10-D embedding with its label, material, category, and session tag, stored in MongoDB Atlas. Teaching writes a signature; the next prediction searches it.
Two Ways to Decide
Vector search over hand features. The query embedding
goes to Atlas $vectorSearch, which returns the nearest taught
signatures by cosine similarity. This path powers the vector-space plot and
supplies the neighbor list on the decision card. On the held-out set it
reaches 24% at the category level.
Shrinkage LDA on the same features. atlas.predict_lda
standardizes the 10-D library, fits a shrinkage-LDA classifier in
milliseconds, and predicts — refit on every call, so a teach takes effect
immediately. This is the live decider. On the held-out set: 52% category,
43% material. Binary empty/nonempty detection reaches 86–89% across all
methods — empty is reliably separable.
A shrinkage-LDA classifier refits the library at every prediction, reporting the measured accuracy across three levels: general category, specific material, and closest taught item.
Learned Embeddings
A bake-off compared two unsupervised alternatives against the hand features: PCA-10 on raw torque-delta trajectories (43% material, matching hand features) and Chronos foundation-model embeddings (56% on swirl-only data, the best measured). Both stay in evaluation — Chronos needs its own 2304-D index and torch on the machine.
Measured Results
106 labeled swirl recordings, collected in two days; classifiers trained on held-out data. Empty-vs-nonempty detection reached 86–89% accuracy — the robot reliably knows when the cup is empty. Three-way classification (liquid / solid / empty) reached ~52%, roughly double the pure vector-retrieval baseline. These are small-data results, not production numbers.