Analyze a run

tools/run_analysis/ reads a recorded run’s bag and explains it. It compares what each node commanded with how the car moved, measured independently of odometry by matching each LiDAR scan against the one before. The tools are in tools/run_analysis/GitHub. Code excerpts below are from its scan_motion.py and actuation.py, shortened. The commands use car 1’s run folder, ~/racerbot-ws/data/<id>/racerbot_runs/<ts>; on car 2 it is ~/racerbot-ws/data/<id>/racerbot_runs/<ts> (see Record a run).

Analyze a run

cd ~/racerbot-ws
python3 tools/run_analysis/analyze_run.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts>

The first run reads the bag, which needs ROS sourced, and takes a minute or two for a long recording, mostly scan matching. The result is cached in analysis/run.npz, so later runs take seconds and do not need ROS. The report is printed and saved as analysis/report.md, with a plot per LB hold.

How the car’s motion is measured

/odom cannot be the reference for how the car moved: its speed is wheel speed, which reads high when the tyres slip, and its yaw rate is computed from the steering command, so it agrees with the command by construction. The analyzer measures motion from the LiDAR instead, by matching each scan against the scan two before it with ICP, iterative closest point.

Each scan is a cloud of points around the car. Between two scans the car moved by some rotation $\Delta \theta$ and shift $(\Delta x,\Delta y)$, so the walls in the newer scan are the walls in the older one, moved the other way. ICP finds that motion by repeating three steps:

1. Pair the points. Move the newer scan by the current guess, and pair each of its points with the nearest point in the older scan. Pairs further apart than a shrinking threshold, 0.5 m at first and never below 0.15 m, are dropped as not the same wall.

2. Find the best rigid motion for those pairs. Subtract each cloud’s centroid. For centred points ${\mathbf{p}}_i$ and their partners ${\mathbf{q}}_i$, the rotation that lines them up best, in the least-squares sense, is the angle of the summed cross products against the summed dot products:

   $$\Delta \theta =atan2(\sum _i(p_{x,i}{q}_{y,i}-p_{y,i}{q}_{x,i}),\sum _i(p_{x,i}{q}_{x,i}+p_{y,i}{q}_{y,i}))$$

   and the shift is whatever then moves one centroid onto the other.

3. Update the guess and repeat, until the correction is below $10^{-5}$ or 30 rounds have run.

p, q = moved[inlier], reference[idx[inlier]]
pm, qm = p.mean(axis=0), q.mean(axis=0)
h = (p - pm).T @ (q - qm)
dtheta = math.atan2(h[0, 1] - h[1, 0], h[0, 0] + h[1, 1])
rot = np.array([[c2, -s2], [s2, c2]])
shift = qm - rot @ pm

h is the 2×2 matrix of summed products, so h[0, 1] - h[1, 0] is the summed cross product and h[0, 0] + h[1, 1] the summed dot product. Dividing the motion by the time between the two scans gives the forward speed and the yaw rate. On run 20261019-200703 the median distance between matched points was 6 mm. In a long corridor with no features, the walls look the same after a shift along the corridor, so ICP can under-read speed there.

How lag and gain are measured

Lag. If the car follows a command after a delay $\tau$, then the measured signal $m(t)$ looks like the command shifted later, $c(t-\tau )$. The analyzer tries each candidate delay, shifts the command by it, and keeps the delay with the smallest root-mean-square difference:

Gain. Once aligned, the measured value is modelled as a fixed fraction $g$ of the command, $m\approx gc$. The least-squares $g$ minimizes $\sum (m_k-g{c}_k{)}^2$. Setting its derivative with respect to $g$ to zero gives

A curvature gain of 0.95 means the car turned 95% as sharply as it was asked to.

def best_lag(t, measured, cmd_t, cmd, lags, valid=None):
    best = (float("nan"), float("inf"))
    for lag in lags:
        c = np.interp(t - lag, cmd_t, cmd)
        err = measured - c
        rms = float(np.sqrt(np.mean(err ** 2)))
        if rms < best[1]:
            best = (float(lag), rms)
    return best

def _gain(measured, commanded):
    den = float(np.sum(commanded ** 2))
    return float(np.sum(measured * commanded) / den) if den > 0 else float("nan")

Read the report

Each period of LB held gets four parts.

1. What the car did: time, distance, mean and peak speed, and how many ticks commanded zero speed.
2. How the car responded: each line compares a command with what the car did.
3. Replay: how closely the simulator’s copy of gap_follow reproduces what the car published, from the same scans.
4. Stops: one line per group of stop ticks, naming the layer that stopped the car.

For a time-to-collision stop, the report also gives the beam that tripped it, how far that obstacle was from the car’s path and from the body, and how far the body’s path was clear against the stopping distance. A stop whose obstacle sits outside the body with the path clear well past the stopping distance is a false alarm. On run 20261019-200703, all 97 stop ticks were attributed this way, which led to the narrower TTC corridor gap_follow uses now.

Replay with a changed parameter

--set name=value replays the same scans through gap_follow with one parameter changed. Pass --yaml with the configuration the car ran, saved from Git, whenever gap_follow.yaml has changed since:

git show <commit>:src/gap_follow/config/gap_follow.yaml > /tmp/gap_follow_run.yaml
python3 tools/run_analysis/analyze_run.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts> \
  --yaml /tmp/gap_follow_run.yaml --set ttc_path_margin=0.05

A steering error near 0.001 rad in the replay section means the replay matches the car. The report ends with the stop count with and without the change. On run 20261019-200703, the replay reproduced 96 of the 97 recorded stop ticks, and none occurred with ttc_path_margin=0.05.

The replay is open loop: it uses the scans the real car saw. It can count which stops a change would remove, but not where the changed car would have driven next. To judge whether a change is safe, run the closed-loop simulator.

Compare turn-rate sources

For a run recorded with the IMU side running, the default for map_course on both cars:

python3 tools/run_analysis/imu_report.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts>

It compares the turn rate from /odom (computed from the steering command), both gyros, /odom/imu and scan matching, measures steering lag and turning gain from the gyro, and reports how much heading correction SLAM applied on top of each odometry. Odometry that needs less correction gives SLAM a better start.

Check how SLAM built the map

slam_report.py reads a recorded mapping run and reports how slam_toolbox behaved: how far the odometry’s heading drifted from SLAM’s, how often and how far SLAM corrected the map, each lap’s quality and whether scans were dropped. It is in slam_report.pyGitHub. map_course runs it after a mapping run with analysis:=true (see Map a course). To run it by hand on a stopped car, from the workspace root:

source /opt/ros/jazzy/setup.bash
source install/setup.bash
python3 tools/run_analysis/slam_report.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts>

The first read of a bag needs ROS sourced, and its inputs are cached in analysis/slam_report_inputs.npz so later reads do not. The report is printed and written to analysis/slam_report/ as slam_report.txt, summary.json and PNG plots. It counts loop closures from launch.log in the run folder, or from the log given with --log <path>. Without a log the count is reported as unknown. --refresh rereads the bag, and --no-plots skips the PNGs.

To compare two runs side by side, with overlay plots written to a third folder and nothing written into either run:

python3 tools/run_analysis/slam_report.py --compare <run A> <run B> --out <output folder>

Each lap gets exactly one verdict. Laps before the last requested lap are not judged. A judged lap is failed when the supervisor logs a failure, or failed (accepted at the ceiling) when the run reached its lap limit and kept it anyway. It is passed only when a later log line confirms its racing profile was written, and unknown without that evidence.

On car 1’s run 20261026-194913, mapped on gyro odometry, the final heading difference was −16.44° between /odom/imu and SLAM, against 102.61° for /odom. The report also found map consistency of 34.1%, three corrections over 1.5 m and 32 scan drops. Laps 3 and 4 failed the 15° heading check at 29.5° and 29.2°, and the run saved no map.

Replay SLAM exactly

slam_replay.py runs slam_toolbox again on a recorded bag, with the car parked, and writes the result into a new run folder. Its --exact mode reproduces the car’s own SLAM poses one for one, so a SLAM setting can be tested against a known baseline without driving again.

The replay refuses to start while a VESC, LiDAR or joystick driver process is running, including the simulator’s joystick node. Every process it starts runs on ROS domain 77 with discovery limited to the same computer, so nothing it publishes reaches the car’s graph. From the workspace root, with ROS and the workspace sourced as above:

python3 tools/run_analysis/slam_replay.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts> \
  --exact --odom odom_imu --label exact

Use --odom odom_imu for a run mapped on gyro odometry and --odom odom for one mapped on wheel odometry. The replay prints its output folder, ~/racerbot-ws/data/<id>/racerbot_runs/<ts>-replay-<odom>-<label>/, which holds a replay bag, logs, the map when it saved and replay.json. Check status and pose_sequence_error in replay.json: the error gives the largest position and heading difference between the recorded and replayed poses. An existing output folder is refused, so give each attempt a new --label, or put results elsewhere with --out-root <folder>. The source run is only read.

How the exact replay matches the car

SLAM does not process every scan: it admits a scan only after enough time or travel since the last one, and the timing on the car decides which scans pass. The exact replay therefore takes the list of scans the car processed from the stamps of the recorded /pose messages, and feeds those scans to synchronous SLAM one at a time, waiting for each matching pose before sending the next.

• Admission. The four admission gates (throttle_scans, minimum_time_interval, minimum_travel_distance, minimum_travel_heading) are opened, throttle_scans to 1 and the other three to zero, so every planned scan is processed. Every other scan-matching, map and solver setting is the recording’s own, from its parameter snapshot.
• Startup. slam_toolbox discards the second to fourth scans it receives at startup, so three recorded scans are sent as sacrifices before the second planned scan. If one of them produces a pose, the replay fails and names its stamp.
• Transforms. The replay publishes the car’s recorded /tf messages in their original order, with every transform whose parent is map removed so the car’s old corrections cannot replace the replay’s new ones. TF then interpolates and composes the odometry chain (odom_imu → odom → base_link → laser on a gyro run) the same way it did on the car.

A missing pose is logged after 10 s and the replay continues. More than ten in a row ends the replay as failed, naming the stamps.

On run 20261026-194913, 3,826 scans and 315 SLAM poses, the exact replay matched all 315 poses with no missing or extra ones. The largest differences were 1.17 × 10⁻¹³ m and 3.05 × 10⁻¹³°. An earlier version that interpolated the already-composed /odom/imu pose instead of the recorded transform tree matched every stamp but drifted to 2.25 m and 3.63° by pose 59, so replaying the car’s own TF tree is what makes the result exact.

If SLAM’s map-saving service fails three times, the replay writes map.pgm and map.yaml from the last /map in its own bag, and replay.json records map_saved: "from_bag".

Other replay modes

IMU analysis tools

Three more tools measure the IMUs and the estimator. They were written on car 2. Each reads a recorded run, needs ROS sourced for the first read of a bag, caches its arrays in the run’s analysis/ folder, and prints every result next to the configuration value it would replace.

python3 tools/run_analysis/estimator_replay.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts> --set accel_source=vesc

estimator_replay.py is open loop, like the gap_follow replay above: it scores the estimate, not what a controller using it would have done. For a run recorded without a copy of the IMU calibration in its metadata/files/, pass the car’s imu/car.yaml with --car-yaml.

Fit the simulator to the car

fit_sim.py plays a run’s recorded commands into the simulated car and searches for the actuation settings that move it most like the real one:

python3 tools/run_analysis/fit_sim.py ~/racerbot-ws/data/<id>/racerbot_runs/<ts> \
  --grid speed_delay_ticks=1,2,4 --grid speed_loop_brake_gain=2,4,8 --grid steering_delay_ticks=1,3,5

It prints the best settings first, with the speed and yaw-rate error each produced. On run 20261019-200703 the fit cut the speed error from 0.46 to 0.24 m/s and the yaw-rate error from 0.25 to 0.15 rad/s. Write accepted values into tools/f1tenth_sim/sim_fidelity/calibration.py with the run named as their source, then re-run the simulator sweeps.

Add a map as a simulator track

track_import.py turns any occupancy map into a simulator track: it fills unexplored space with wall, finds the loop’s centre line, and picks the straightest stretch as the start.

cp ~/racerbot-ws/data/<id>/racerbot_auto/<ts>/map.pgm tools/f1tenth_sim/maps/sources/hall.pgm
sed 's/image: map.pgm/image: hall.pgm/' ~/racerbot-ws/data/<id>/racerbot_auto/<ts>/map.yaml > tools/f1tenth_sim/maps/sources/hall.yaml
python3 tools/f1tenth_sim/track_import.py tools/f1tenth_sim/maps/sources/hall.yaml \
  --name hall --source "slam_toolbox map, run <ts>" --plot /tmp/hall.png

It prints the track’s lap length and minimum width, and /tmp/hall.png shows the centre line. Every simulator tool then accepts --track hall.

Limitations

• Scan matching under-reads speed in long corridors with no features.
• The replay is open loop, as above.
• One fit is valid for one floor surface. Fit again on a different surface.
• The SLAM report compares SLAM against its own final map and poses, which are estimates, so a high map consistency is not independent proof that the map is accurate.
• The exact SLAM replay reproduces a run only when every scan SLAM processed was recorded. It refuses before starting if a pose has no matching scan.