Neural racer

neural_racer_node drives with a small neural network trained in simulation. It reads the LiDAR scan, the car’s speed and the IMU, and outputs a steering angle and a speed once per scan, about 40 times a second. It is experimental: it runs in shadow mode by default, and no model has driven the car yet. The aim is a policy that races well on the first lap of a course it has never seen, uses that lap’s map from the second lap on, and handles other cars.

The node is in src/neural_racer/GitHub and the training code in tools/neural_racer/GitHub. Both cars run it. It needs a healthy IMU, and drives only while /imu/status reads ok and the IMU data is fresh. Car 2’s BNO085 mount yaw is not measured yet, so the network has been run only on car 1. Code excerpts below are from observation.py, policy_numpy.py and safety_wrapper.py, with docstrings and input checks removed.

Topics

Subscribes to

Publishes

One tick

1. Check the driver and the inputs. LB must be held, with /joy newer than 0.5 s. The scan must be newer than 75 ms, odometry newer than 0.1 s, the IMU newer than 0.1 s with its status ok, and in map mode the pose newer than 0.2 s. Any failure gives a zero command.
2. Build the observation: turn the scans, speeds, IMU samples and past commands into fixed-size arrays of numbers.
3. Run the network to get a steering angle, a speed, and a forecast of the car’s path.
4. Wrap it: a fixed sequence of checks that can only lower the command.
5. Publish the command, then the intent.

The observation

A network takes a fixed number of numbers, each on roughly the same scale. The observation turns the raw inputs into that form. Every scale is a constant written in observation.py, never a running average, so the car’s copy and the training code’s copy cannot drift apart.

The scan. The LiDAR gives 1081 ranges, 270° at 0.25°. The node groups them into 270 bins of four beams and keeps the minimum valid range in each bin, so a thin chair leg or the corner of another car cannot vanish into an average. Each range is then put on a log scale between 0.05 m and 10 m:

which maps 0.05 m to 0 and 10 m to 1. A log scale spends the network’s resolution where it matters: the difference between 0.3 m and 0.6 m is as large as the difference between 4 m and 8 m, because both are a doubling. A bin with no valid beam reads 10 m with a separate validity flag of 0, so “no echo” and “far away” stay distinguishable.

def pool_scan(ranges, range_min, range_max):
    r = np.asarray(ranges, dtype=np.float64)
    valid = beam_validity(r, range_min, range_max)
    usable = np.where(valid, r, np.inf)[:N_BINS * POOL].reshape(N_BINS, POOL)
    nearest = usable.min(axis=1)
    bin_valid = np.isfinite(nearest)
    nearest = np.where(bin_valid, nearest, R_MAX)
    return encode_range(nearest).astype(np.float32), bin_valid.astype(np.float32)

reshape(N_BINS, POOL) lays the 1080 beams out as 270 rows of 4, so min(axis=1) takes the minimum of each row at once. Beam 1081 is dropped.

The scan block holds the last three scans plus the newest scan’s validity flags, four rows of 270. Three scans in a row let the network see motion: how fast things are approaching, not only where they are.

Speeds are divided by 5 m/s, yaw rates by 3 rad/s and accelerations by 9.81 m/s², so each lands near the range −1 to 1.

The network

The network runs in NumPy on the Jetson’s CPU, with no PyTorch or GPU. It has four parts.

Convolutions over the scan. A 1D convolution slides a small set of weights along the 270 bins and computes a weighted sum at each position. The same weights are used everywhere, so a pattern such as a gap edge is detected the same way wherever it appears. Four convolutions with stride 2 each halve the length, building up from edges to larger shapes. The code computes each one as a single matrix multiplication: it lays out every window of the input as one row of a matrix, then multiplies by the weights.

def __call__(self, x):
    xp = np.pad(x, ((0, 0), (self.padding, self.padding)))
    win = sliding_window_view(xp, self.k, axis=1)[:, ::self.stride, :]
    cols = np.ascontiguousarray(win.transpose(1, 0, 2)).reshape(win.shape[1], -1)
    return (cols @ self.w2 + self.b).T

Fusion. The scan features, the ego features (two dense layers) and, in map mode, the preview and race features are joined into one vector. Layer normalization rescales it to mean 0 and variance 1, so no block dominates by size. In reactive mode the map blocks are multiplied by 0.

Memory. A gated recurrent unit (GRU) with 128 numbers of state carries information from one tick to the next. Each tick, two gates decide how much of the old state to keep and how much to replace with the new input:

${\mathbf{z}}_t$ is the update gate, between 0 and 1 for each of the 128 entries, ${\mathbf{n}}_t$ is the candidate new state, and $\odot$ multiplies entry by entry. Where ${\mathbf{z}}_t$ is near 1 the old state is kept, so the network can remember, for example, that a wall it saw a second ago is now beside it.

Outputs. A small head turns the state into two numbers, $a_0$ and $a_1$, and squashes each into its allowed range:

$\tanh$ lies between −1 and 1, so the steering can never exceed 0.33 rad, and $\sigma$ between 0 and 1, so the speed lies between 0 and the cap. The limits hold by construction, whatever the weights. A second head forecasts the car’s own path for the next 1.5 s, six points 0.25 s apart, and the positions of up to four opponents. Neither forecast drives the car. The path forecast goes to /drive_intent, so the dashboard shows where the network expects to go, and in map mode the wrapper uses it for the map path cap.

r = _sigmoid(gi[:H] + gh[:H])
zg = _sigmoid(gi[H:2 * H] + gh[H:2 * H])
n = np.tanh(gi[2 * H:] + r * gh[2 * H:])
self.h = ((1.0 - zg) * n + zg * self.h).astype(np.float32)
t = _silu(self._lin('head.fc0', self.h))
a = self._lin('head.act', t)
...
steering=float(obs.STEER_LIMIT * np.tanh(a[0])),
speed=float(v_cap * _sigmoid(a[1])),

One tick, observation, network and wrapper together, took 3.90 ms median and 4.03 ms at the 99th percentile on car 1’s Jetson, 16% of the 25 ms between scans.

The safety wrapper

A deterministic wrapper runs around the network every tick. Each step can only make the command safer, never faster. The deadman and freshness checks are in step 1 above. The rest, in order:

1. Scan sanity. Stop if the beam count is wrong or more than 30% of beams are invalid (max_invalid_fraction).
2. Output sanity. Stop if the network produced a non-finite number.
3. Clip the steering to max_steering_angle and the speed to max_speed (1.0 m/s).
4. Steering rate. Limit the change to 2.0 rad/s.
5. Braking cap. gap_follow’s own braking-distance formula, imported from gap_logic, not copied, applied to the smallest body clearance in any direction. See Follow the gap for its derivation.
6. Cornering cap. $\sqrt{a_{\text{lat}}/\kappa }$ with $\kappa =\tan \delta /L$ and $a_{\text{lat}}$ = 2.0 m/s².
7. Map path cap. In map mode, the braking cap again, to the first wall along the network’s forecast path.

The final speed is the lowest of the network’s speed and every cap. The wrapper records which one won, and whether anything other than the network shaped the command.

caps = {'network': speed}
max_step = c.max_steering_rate * min(dt, c.command_slew_max_dt)
steering = min(max(steering, prev_steering - max_step), prev_steering + max_step)
boundary = self.boundary(angles)
clearance = gap_logic.minimum_footprint_clearance(ranges, valid, boundary)
caps['braking'] = gap_logic.braking_speed_limit(
    clearance, c.reserve_distance, c.max_braking_decel, c.max_speed,
    c.braking_latency_sec)
kappa = abs(math.tan(steering)) / c.wheelbase
if kappa > 1e-9:
    caps['curvature'] = math.sqrt(c.max_lateral_accel / kappa)
binding = min(caps, key=caps.get)
final_speed = caps[binding]

Shadow mode

By default the node publishes only on /neural_racer/shadow_drive, never on /drive. It refuses a shadow topic that reaches the motor, and then publishes nothing at all. gap_follow drives the car while the network’s commands are recorded beside it for comparison.

Terminal 1:

ros2 launch f1tenth_stack bringup_launch.py

Terminal 2, the IMU from imu_bridge. Keep the car still for about 3 s after it starts, for the gyro bias and the gravity check.

ros2 launch imu_bridge vehicle_state_launch.py

It is working when ros2 topic echo /imu/status --once shows "state": "ok". A fault names its reason.

Terminal 3:

ros2 launch gap_follow gap_follow_launch.py

Terminal 4, the network:

ros2 launch neural_racer neural_racer_launch.py model_hash:=<hash>

It is working when the log reads SHADOW -> /neural_racer/shadow_drive and ros2 topic hz /neural_racer/shadow_drive reads about 40 Hz while LB is held. A REFUSING TO DRIVE: line names the reason when it is not: an unapproved model, a camera that is turned on, a max_speed outside 0 to 6.37 m/s, or an unknown mode.

Approve a model

The node loads only models listed in ~/.ros/neural_racer/models/manifest.yaml. approve_model adds one after checking its hash, its observation format, and that the NumPy forward pass matches PyTorch to within 1e-5.

ros2 run neural_racer approve_model ~/.ros/neural_racer/incoming/<hash> --shadow-only

It prints approved <hash> for shadow mode only; golden check passed, or REFUSED: with the reason. A model may drive /drive only once its selection and referee reports both pass.

Training

Training runs on the FIR cluster or a CUDA workstation, never on the car. A batched simulator in PyTorch runs about 16,000 races at once and matches the calibrated CPU simulator to within 2 cm and 0.5° over 10 s of driving. A teacher is trained first, distilled into the student network the car runs, then improved in a league against archived copies of itself and scripted opponents. Releases are chosen by racing the exported policy against gap_follow and pure_pursuit on held-out tracks that training never sees. The stages and the cluster scripts are in tools/neural_racer/README.mdGitHub.

Test order

1. Static check of its topics, with no driver stack running.
2. Shadow mode beside gap_follow.
3. Wheels off the ground, with shadow:=false.
4. The floor at max_speed:=1.0, raised 0.5 m/s at a time after each clean run.

Tests

The package’s tests run without ROS, apart from the node test:

python3 -m pytest src/neural_racer/test/ -v --ignore=src/neural_racer/test/test_neural_racer_node.py

They check, among other things, that the NumPy network matches a plain reference implementation step by step, that its outputs stay inside their limits for any input, that the wrapper never raises the network’s speed, and that its braking cap matches gap_logic. The match against PyTorch is checked per model, by approve_model. See Run the tests.

Limitations

• No model has driven the car yet.
• Car 2 cannot run it until it has a BNO085 and its IMU calibration, because imu_bridge will not start without them.
• Map mode has no source for its reference path yet: nothing publishes /auto_map_race/race_path.
• The camera input slot exists but is disabled. camera_enabled: true is refused.