It wasn't one. This part is what happened the moment I tried to scale that exact same fix past the point where it had only ever been tested.

The 300k Run: Same Symptom, Different Clothes

The obvious next question after Part 5 was whether the fix actually held, or whether it was a 100k-step-scale artifact that would fall apart the second I gave it more room to run. Same architecture, dueling critic, nav_info concatenated into the actor's input, auto-tuned alpha through the Lagrangian entropy trick, gamma still compressed to 0.90, just scaled up to 300,000 steps.

It didn't hold cleanly. To be fair, the actual Part 5 fix stuck, steering wasn't frozen at -0.0102 anymore. But a new pattern showed up in its place, and it was oddly specific. Steering would climb smoothly from near-zero, hold there for a stretch, plateau, then decay back down, a shape closer to a slow sine wave than anything resembling "read the road, correct toward center."

I checked this across six different scenario indices, different maps, different layouts entirely, and the same ramp-then-decay shape kept showing up in close variants across all of them. That consistency was the first real clue that whatever was driving this wasn't something specific to a given road. It was structural.

Which meant the framing from Part 4 needed correcting. Nav scaling and more training steps had solved bias-vector lock-in. That was real, and the table proved it. But bias-vector lock-in and actually reading the road are two different bars, and I'd only cleared the first one.

Hypothesis: Borrowing MetaDrive's Own Expert

My first read on the ramp was that the critic could tell actions apart now, probe_critic_at_turn.py was showing smooth, sensibly shaped Q-surfaces instead of the flat tabletop from Part 3, but maybe it still hadn't seen enough correctly-labeled turning behavior to actually generalize "nav_command says left, so steer left" purely from self-play. Early self-play is mostly incompetent exploration. If a critic trained almost entirely on bad turns is the thing shaping the actor, that's a version of the classic spurious-correlation trap, the model latches onto whatever correlate happens to be lying around, elapsed time, distance since reset, instead of the actual causal signal it's supposed to be using.

I considered scripting a rule-based driver by hand for straight sections, turns, and junctions separately, and decided against it almost immediately. MetaDrive already ships a working expert autopilot, env.agent.expert_takeover = True, IDM-style, competent at lanes, turns, and traffic. Using it meant zero new controller logic to write or debug. Just let it drive, and record.

Two Bugs Before the Expert Would Actually Drive

Getting that flag to do anything useful took longer than it should have, and both problems were found by actually reading output, not by guessing.

First attempt, expert_takeover = True on its own did nothing. info["action"] kept handing back my own placeholder [0.0, 0.0], unchanged, step after step. Turned out manual_control has to be set to True in the env config for the takeover flag to actually engage at all. Without it, the flag is silently inert, no error, no warning, it just quietly doesn't do anything.

Second bug, once the car was genuinely under ManualControlPolicy, the real action being executed wasn't sitting under the "action" key like I'd assumed. It was under "raw_action", a tuple of np.float32 values. Only found that by dumping the entire info dict and reading every key in it instead of guessing which one was likely to hold the answer, which is a lesson I seem to relearn every few phases of this project.

The Collection Pipeline

collect_demonstrations.py: reset with manual_control=True, num_scenarios=200 for variety, force expert_takeover=True after every reset, then step through full episodes until terminated or truncated fires. Pull raw_action at each step and store the full transition, (obs_grid, obs_num_others, obs_nav_info, action, reward, next_obs_grid, next_obs_num_others, next_obs_nav_info, terminated, truncated), shaped exactly the way the real replay buffer expects it. All of it serialized out to demonstrations.pkl.

Fifty episodes came out to 37,503 transitions total, verified after the fact rather than assumed.

train-v4.py is identical to v3 except for one addition, a load_demonstrations() call right after the replay buffer gets instantiated, run once before the main loop starts, iterating the pickled transitions and pushing each one in through the exact same .add() call the live loop uses later. One deliberate simplification here: demo transitions aren't protected from being overwritten once the circular buffer cycles past capacity during training. I thought about building a reserved region for them and decided against it for this first pass, getting an honest read on whether seeding helped at all mattered more right now than optimizing how it was seeded.

Ran it first at 100k steps as a validation pass, then extended the same setup out to 500k to see if it held at a longer horizon.

The Result: Shorter Episodes, Same Ramp

Demo-seeding didn't fix anything. Worse, by one metric it made things measurably worse. Episodes in the demo-seeded run were ending after 25 to 49 steps of out_of_road, compared to 85-plus steps in the equivalent v3 run with zero demo data. And the ramp-then-decay steering shape was still right there, now showing up at a strikingly consistent point, a sharp correction landing around step 20 to 21, run after run, regardless of which map it was on.

That consistency was the actually interesting part. It smelled exactly like the Husky-versus-wolf problem, a model technically solving the training objective by keying off a correlate that happens to be available, instead of the causal signal it's actually supposed to be reading.

Two Checks to Rule Out the Demo Data Itself

Rather than keep training and hoping the next run looked different, I ran two direct, falsifiable checks on the demonstration data.

Label balance. Tallied steering sign across all 37,503 transitions. 51.4% left, 48.6% right, mean steering sitting at -0.008. Essentially balanced. Ruled out demo-label imbalance as the cause outright.

Temporal clustering. If "meaningful turn" moments, anything with |steering| > 0.1, clustered tightly around a specific step-count-since-reset across episodes, that would be direct evidence of a temporal confound baked into the demo data itself, most demo episodes just happening to hit their first real turn around a similar point. Measured mean step-since-reset at those moments: 257.99, standard deviation 142.03. That's a wide spread, not a cluster. Ruled that out too.

Both hypotheses about the demonstration data were cleanly falsified by actual measurement. Good process. Wrong location.

The Actual Tell

Went back and compared a v3 run, no demo data at all, 300k steps, directly against the demo-seeded 500k run, steering value by steering value. Both showed the same shape, a climb from near-zero up to a plateau around 0.13 to 0.15, a brief hold, then a smooth decay. The absolute step where it happened differed, v3 plateaued around step 55 to 56, the demo-seeded run around step 20, but the shape itself, climb, hold, decay, was structurally identical in both.

That's the decisive result. Demonstration seeding is neither the cause of the ramp nor the fix for it. It shows up with zero demo data and it shows up with demo data. Whatever's producing it lives somewhere in the base training setup, not in what's feeding the replay buffer.

Current Hypothesis: The Comfort Penalty Might Be the Real Author

Which sent me back to something I'd added two parts ago and hadn't really interrogated since. The comfort penalty from Part 7:

R_comfort = −0.1 × (a_steer,t − a_steer,t−1)² − 0.1 × (a_throt,t − a_throt,t−1)²

That term penalizes how fast the action changes, full stop, independent of whether the change was actually the correct thing to do given the road. An agent training under that penalty has a direct incentive to move smoothly and gradually between action values regardless of target. A slow climb, a plateau, a slow decay is close to exactly the shape you'd predict from a policy that's learned "large jumps are expensive" without a strong enough countervailing signal about where it should actually be steering toward. If the comfort penalty's gradient pressure is currently outweighing the still-forming road-reading signal coming from the critic, "smooth arcs regardless of geometry" is a locally attractive, easy-to-learn strategy to fall into. It's basically the same baseline-dominance problem from Phase I and II, just relocated from the value-magnitude domain into the temporal one.

The Test in Progress

train-v5.py is identical to v4, dueling critic, nav_info, demo-seeding, auto-alpha, gamma at 0.90, with exactly one change: comfort_penalty forced to 0.0, not just shrunk, fully zeroed out, specifically to keep this a clean single-variable test. Scaled back down to 100k steps, validation scale, so the answer comes back fast enough to actually inform the next move instead of costing another ten-hour wait for one data point.

The falsifiable prediction: if the comfort penalty is actually driving the ramp shape, removing it should produce steering that looks more like sharp, direct corrections tied to observable road geometry, probably noisier and more jerky moment to moment, since nothing's discouraging abrupt changes anymore, but responsive in a way the smooth arcs never were. If the ramp shape survives with the penalty at zero, that rules the comfort penalty out too, and the search moves somewhere else next, most likely the actor-critic learning rate balance, or the possibility that the CNN's spatial features still aren't discriminative enough between "road curves left" and "road curves right" at the resolution this whole pipeline is running at.

Run was still going as of writing this. Don't have the answer yet.

Table, Continued From Part 7

The pattern across this whole arc is starting to feel less like a series of separate bugs and more like one repeating shape. Every fix has been finding whatever shortcut is currently cheapest for the optimizer to fall into, a flat critic, a frozen bias vector, a temporal confound that turned out not to exist, and now possibly a smoothness penalty that's winning against a signal that hasn't caught up yet, and removing that one specific shortcut before checking what's underneath it. No conclusion yet on this one. That's genuinely where it stands.

The Car That Glitched

First symptom: the vehicle started accelerating, then immediately started glitching. Steering violently hard-right, then hard-left, on consecutive steps. Throttle swinging between full power and hard braking, with nothing smooth in between.

The cause traced back to how SAC's actor actually works. It's stochastic by design, outputting a mean and standard deviation rather than a single fixed number, and during training that's exactly right, you sample from N(μ, σ) specifically to keep exploring. The mismatch was that I'd left evaluation running the same way, deterministic=False, which meant every single step during evaluation was still drawing an independent random sample from that distribution. Two back-to-back steps could land on wildly different points of the same distribution purely by chance, and the car experienced that as violent, second-to-second whiplash. The fix was narrow and specific: set deterministic=True at evaluation time, so inference runs strictly along the policy mean, a_t = μ(s_t), and the randomness that's genuinely useful during training stops leaking into how the car actually drives once training is over.

Smooth Isn't the Same as Comfortable

Even with deterministic evaluation fixing the second-to-second chatter, the driving was still jerky over longer stretches. The reward function I'd built was shaped entirely around progress and safety, and it had nothing at all to say about comfort. Which meant a trajectory that swerved violently at high speed scored exactly the same as a trajectory that stayed smooth the whole way through, as long as both ended up making equivalent progress.

So I added a quadratic action-smoothness penalty directly into the environment loop:

R_comfort = −0.1 × (a_steer,t − a_steer,t−1)² − 0.1 × (a_throt,t − a_throt,t−1)²

Caching the previous step's action and comparing it against the current one, then folding that penalty into the reward before it ever reaches the replay buffer. Simple, but it meant the agent was now being scored not just on where it ended up, but on how abruptly it got there.

Two Bugs at the Wrapper Boundary

A couple of structural issues showed up right where my custom wrapper met the underlying environment.

First, an AttributeError: 'FrameStackWrapper' object has no attribute 'action_space'. The wrapper I'd built was acting as a complete barrier, hiding everything about the raw environment underneath it, including attributes other parts of the training code needed direct access to. Fix was straightforward once I saw it: query the inner, unwrapped environment directly, raw_env.action_space.sample(), instead of expecting the wrapper to transparently forward every attribute on the object it was wrapping.

Second, a missing positional argument, 'truncated', crashing the replay buffer. Older Gym APIs merged termination and timeout into one shared done flag. Modern Gymnasium APIs split that into two separate signals on purpose, terminated for a genuine catastrophic end, a crash, where the expected future return really is zero, and truncated for hitting a step-count timer while otherwise driving fine, where the expected future return very much isn't zero. I'd already reasoned through why that distinction mattered back in Part 2, but this was where it actually broke the code until I decoupled the two signals properly and stored them independently.

The Last Problem: Steering Stuck at -0.0102

With the structural issues cleared out, the car was actually accelerating well, hitting speeds around 4.35 m/s. But its steering had locked onto almost exactly -0.0102, consistently, across different scenarios, and the car was slowly drifting off the left edge of the track because of it.

The root cause was the asymmetric setup itself, the same design that had fixed the earlier "cheat sheet" problem back in Part 1 was now creating a new one. The critic, the teacher, was processing the latent vector plus the sixteen-dimensional num_others array, which meant it had direct access to exact lateral error and heading deviation. With that information sitting right there, it could easily compute a properly curved value surface and meaningful steering gradients. The actor, the student, was only ever seeing the detached visual latent, nothing else. And early in training, at only around 15,000 steps in, that visual CNN was still close to uninitialized static noise, producing features with effectively zero meaningful variance relative to actual lane boundaries. With no useful signal coming from its own input, the actor's forward pass ended up dominated entirely by the bias vectors sitting in its output layers, vectors that happened to have initialized somewhere near -0.0102. And because the critic's steering gradient was still comparatively weak relative to everything else going on, the comfort penalty from earlier in this part locked the behavior in place further: the safest way to avoid a change penalty was to just leave the steering exactly where the bias vector had put it, and focus entirely on accelerating instead.

Two ways to fix this presented themselves, and they pointed in opposite directions.

Option one: give the actor privileged telemetry too. Just concatenate num_others onto the actor's input directly, matching what the critic already sees. This worked essentially instantly, steering sensitivity came back and the car started centering itself in the lane right away. But it broke the entire premise of the project. The whole point was an end-to-end visual policy, a car that drives off what it can actually perceive, not one that queries privileged simulator coordinates it would never have access to outside a simulation. Taking this option would have meant winning the metric and losing the actual research question.

Option two: stay pure end-to-end, and fix the underlying imbalance instead. This is what I went with. Two things made it work. First, scaling down the navigation input relative to the visual latent before concatenation:

x_actor,t = [z_visual,t ‖ (n_nav,t × 0.1)]

Scaling the navigation coordinates down by a factor of 0.1 kept them from mathematically overwhelming the still-forming visual representation, giving the CNN room to actually start contributing instead of getting drowned out by a numerically larger but less informative signal. Second, simply extending training out to 100,000 steps. That gave the critic enough temporal volume to keep backpropagating TD errors through the visual encoder long enough for the CNN to actually learn to map raw road geometry into structured spatial features, rather than staying stuck as noise. Once that happened, the actor finally broke free of its bias-vector default, its steering signs aligned with the geometry of the road, and it started gating its own navigation behavior, only actually steering when the lane ahead genuinely opened up for it.

The Whole Arc, Side by Side

Pulling every phase of this into one table, since seeing it laid out end to end says something the individual debugging stories don't quite capture on their own:

Every row of that table is a different flavor of the same underlying lesson, really: a critic that can't feel the action signal will always find some cheaper baseline to hide behind, and every fix in this whole arc, dueling streams, horizon compression, leaky gradients, comfort penalties, nav scaling, was really just different ways of forcing the critic to stop being able to take the easy way out.

Defining the Problem

First instinct with something like this is to assume it's broken. It wasn't. The agent wasn't stuck or malfunctioning, it had actually calculated, correctly, that moving was worse than standing still. Zero throttle and zero braking minimized its losses better than anything else it had tried.

That's a strange thing to sit with. The car wasn't failing to learn. It had learned something, just not the thing I wanted.

Hypothesis One: A Risk-Averse Optimizer

Looking at it through the Q-function, the thing SAC is actually trying to maximize is expected return. So I engineered a reward topology where driving forward gives a continuous stream of small positive rewards, efficiency and progress, while crashing or going out of bounds triggers a massive negative penalty.

Run the numbers on that. If the agent drives forward for ten steps and then crashes: (10 steps × 0.1 driving reward) − 50 crash penalty = −49 total return. If the agent instead slams the brakes and sits completely still for those same ten steps: (10 steps × 0 reward) + 0 = 0 total return.

Zero beats negative forty-nine. Every time. The agent wasn't broken, it was doing exactly what the math told it to do.

Diagnosing Policy Collapse

My first read on this was that the policy had collapsed, in the SAC sense. In continuous action spaces, the policy isn't a lookup table, it's an explicit distribution, predicting the parameters of a Gaussian at every state rather than a single fixed action.

Here's how I thought the collapse might have happened. At step 0, the network initializes with random weights, which means a wide standard deviation. Wide standard deviation means highly random actions, which inevitably means the car crashes into something almost immediately. The critic sees that catastrophe and sends back massive negative gradients. And the actor's mathematical objective, in response, is to eliminate the randomness that caused it, since less randomness means fewer catastrophic actions. If the architecture clamps the log standard deviation at some minimum, say -20, the exponential of -20 is a vanishingly small number, and the Gaussian effectively collapses into a single sharp spike, a Dirac delta function in all but name. At that point entropy regularization has basically nothing left to push against. The critic has overpowered the actor, and the actor has decided that absorbing the entropy penalty is cheaper than ever risking the crash penalty again.

It's a clean theory. I went looking for evidence of it in the training logs.

Step 990 → out of road, reward drifting between 0.000 and 0.051. Step 1640 → out of road. Step 3180 → out of road, reward sitting at 0.001 or 0.000. Step 5140 → 0.050, but 0.00 for most of the run. Step 6148 → out of road. Step 8604 → out of road.

Same pattern, over and over. So I tried the obvious fixes. Lowered the crash penalty from 50 down to 15. Raised alpha, the entropy coefficient, up to 0.3, to force more exploration back into the policy. Nothing changed. The agent still wasn't moving. Which meant the actual problem wasn't where I thought it was, and it was time to go looking in the training code directly instead of just adjusting hyperparameters and hoping.

The Diagnostic That Ruled Out My Own Theory

I wrote a small diagnostic to actually check whether the policy had collapsed, rather than just assuming it had. The logic: if the standard deviation values are genuinely tiny, somewhere around 1e-6 or lower, that confirms real collapse, the actor paralyzed by fear of the environment and shutting off all stochastic exploration entirely. If instead the standard deviation is still meaningfully large, above roughly 0.1, then collapse isn't the story, and something else architectural is going on, a dying CNN latent vector, or a critic outputting flat gradients.

Even after reducing alpha down to 0.05, the actor's standard deviation sat at roughly 0.87. Nowhere near a collapse. Which meant my entire first theory, however clean it looked on paper, was wrong.

A Detour Through SAC's Automatic Temperature Tuning

Before abandoning the collapse theory completely, I wanted to rule out one more thing: whether alpha itself was being tuned correctly. SAC has a mechanism for this, automatic temperature adjustment, which treats entropy as a constrained optimization problem rather than a fixed hyperparameter. The framing is roughly: maximize expected return subject to policy entropy staying above some minimum target, H(π_θ(s)) ≥ H_target. Using dual formulation, that constraint introduces a learnable Lagrange multiplier:

L(α) = E[−α(log π_θ(a|s) + H_target)]

Worked through both directions of that. If the policy is too random, entropy exceeding target, say H_target = −2.0 and the term inside the parenthesis comes out to log π_θ(a|s) + H_target = −2.5 + 2.0 = −0.5, negative. Plug that back in: L(α) = −α × (−0.5) = +0.5α, which drives alpha down toward zero, pulling exploration back in. If the policy is collapsing instead, entropy falling below target, the sign flips, L(α) = −α × (+1.8) = −1.8α, and alpha climbs upward, pushing exploration back up. Target entropy itself is typically chosen as H_target = −dim(A) = −2.0 for a two-dimensional continuous action space, steering and throttle.

None of it mattered here. Even with alpha auto-tuned down to 0.06, the standard deviation stayed stubbornly around 0.87. The temperature mechanism was doing exactly what it was supposed to do. It just wasn't the thing keeping the car frozen.

The Real Culprit: An Uninformative Critic

If the standard deviation wasn't collapsing, then the actor wasn't afraid of exploring. Which meant the problem had to be sitting somewhere else in the loop entirely, in what the critic was actually telling it.

Here's the calculation that cracked it open. Take two nearly identical actions: a1 = [steer: 0, throttle: 0.5], a2 = [steer: 0, throttle: 0.52]. One step later, the vehicle's speed under a1 might land at 10.00 km/h, and under a2 at 10.02 km/h. That's a velocity difference of Δv = 0.00005. Practically nothing.

Which means, for almost the entire space of reasonable driving actions, the reward signal barely changes at all between one action and its near neighbor. The critic only ever sees a sharp, high-variance signal at two specific, discontinuous events: releasing the throttle entirely and triggering the stationary penalty, or crashing outright and eating an immediate penalty. Everywhere in between, for any action sequence that keeps the car safely on the road, the critic's training targets end up nearly identical to each other, and the value landscape it learns ends up exceptionally smooth and correspondingly flat.

That flatness isn't a bug. It's the mathematically correct shape of the actual task. There's a genuinely high-dimensional manifold of "acceptable" driving behaviors, gentle variations in steering and throttle that all lead to roughly the same long-term outcome, and a flat value landscape across that manifold is exactly what an accurate critic should produce. It has no measurable impact on long-term expected return whether you steer 0.01 degrees left or 0.01 degrees right, so an accurate critic correctly reports that they're worth the same.

I tested this directly rather than just trusting the theory. Sampling the critics' Q-value estimates across a batch of similar safe-driving states:

Critic 1 → range [4.0858, 4.0968], max-min delta ΔQ = 0.0110, standard deviation ≈ 0.0030. Critic 2 → range [4.0838, 4.0951], max-min delta ΔQ = 0.0113.

Confirmed the uninformative-critic hypothesis directly. Geometrically, the Q-surface really was close to a perfectly flat table. In that state, it genuinely doesn't matter what the actor does, crashing, braking, steering, speeding, they're all landing at close to mathematically identical value as far as the critic can tell.

The Actor-Critic Paradox

This is where the actual freeze comes from, and it's a kind of paradox once you see it laid out. When the critic's landscape is steep, sharp feedback, it violently pulls the actor's mean toward the peak of that landscape and rapidly forces the standard deviation down, since there's a clear best action worth committing to. But when the landscape is flat, which is exactly the situation here, the critic is effectively telling the actor "left or right, faster or slower, doesn't matter, they're all equally good." There's no gradient sharp enough to pull the actor's mean anywhere in particular, and there's nothing pulling it toward "drive forward" specifically either. The safe, do-nothing basin sits right there in that same flat region, easy to fall into and with nothing steep enough nearby to push the agent back out of it.

To get past that deadlock, I needed a way to actually see what the critic was telling the actor at the level of a single gradient, not just infer it from standard deviation values and reward logs.

Building a Probe

I froze a single representative starting observation, the car sitting at the track's start line, and queried the critic's gradient with respect to action directly, ∇_aQ(s,a), evaluated at the zero-action baseline. The result:

Evaluated State Q-Value       : 4.0913
Analytical Gradient (dQ/da)   : Steering: 0.00087807 | Throttle/Brake: 0.00461295

Both gradients were microscopically tiny, on the order of 10⁻⁴ and 10⁻³. And that mattered a lot more than it might sound like, because of exactly how SAC updates its actor. The actor's weights get updated by propagating gradients through the critic, using the reparameterization trick:

∇_θL_actor ≈ E[α∇_θ log π_θ(a|s) − ∇_aQ_φ(s,a) · ∇_θf_θ(ε;s)]

With ∇_aQ_φ(s,a) sitting essentially flat, that entire exploitation term went dead. Backpropagation through the critic was, functionally, severed. The actor's weights weren't receiving zero direction because of some bug, they were receiving zero direction because there was genuinely nothing there to receive. Whatever movement the actor made was coming entirely from the entropy regularization term, not from anything the critic actually knew about driving.

Why the Critic's Landscape Collapsed in the First Place

Two things were stacking on top of each other here. First, a dimensionality mismatch: the input state space was huge, a 2,048-dimensional visual latent plus a 16-dimensional neighbor-tracking vector, 2,064 dimensions total, feeding into a control output that was just two numbers, steering and throttle. Second, and more subtly, a signal-to-noise problem. Under a discount factor of γ = 0.99, the expected cumulative discounted return integrates progress rewards over roughly a hundred-step horizon: V(s) ≈ r_progress / (1 − γ) = 0.04 / 0.01 ≈ 4.0. That balloons the baseline expected return up to around 4.09. Meanwhile, the actual step-to-step difference in advantage between, say, a lane-centering steer and a mild swerve, was on the order of 0.01. A signal-to-noise ratio of roughly 400 to 1.

Given that ratio, the Adam optimizer did exactly what any gradient-based optimizer would do: found the path of least resistance. It achieved a 99.9 percent reduction in loss simply by learning to reconstruct that dominant 4.09 baseline, and never bothered engaging with the comparatively tiny action-dependent signal buried underneath it. The critic hadn't failed to learn, it had learned something real, Q(s,a) ≈ V(s), a state-value function wearing a state-action function's name.

Attempt One: A Dueling Critic

The obvious structural fix was to stop asking one network to do both jobs. Split the critic into a state-value stream and an action-advantage stream, Q(s,a) = V_ψ(s) + A_χ(s,a). Since V_ψ(s) only ever sees the state, it's mathematically forced to absorb that 4.09 baseline on its own, freeing the advantage stream to focus purely on the local, action-dependent variation that actually mattered.

I ran it expecting a breakthrough. Instead I got a landscape flatter than before I'd started:

Critic 1 Q-value Range       : [4.1446, 4.1446]
Critic 1 Max-Min Delta       : 0.0000
Analytical Gradient (dQ/da)  : Steering: 0.00000000 | Throttle/Brake: 0.00000000

Gradients at exact, literal zero. The reason came down to something discrete dueling architectures get for free and continuous ones don't: an identifiability constraint. Discrete dueling DQN normalizes its advantage stream by subtracting the mean advantage across the finite set of possible actions, A(s,a) − (1/|A|)Σ A(s,a'). With a continuous, infinite action space, a ∈ [-1,1]², there's no finite set to sum over, no way to apply that same normalization. Without it, the optimizer found a new shortcut instead. Early in training, while the policy was still noisy, any nonzero output from the advantage stream only added variance and increased prediction error. Minimizing MSE meant driving the advantage stream's weights all the way down to zero and leaving them there. The critic hadn't just stayed flat, it had actively learned to switch off the one piece of itself that was supposed to fix the problem.

The Actual Breakthrough

Two changes together got the loop moving.

Horizon compression. I scaled the discount factor down from γ = 0.99 to γ = 0.90. That directly compresses the baseline: V(s) ≈ 0.04 / (1 − 0.90) = 0.40. Dropping the state baseline from roughly 4.14 down to 0.40 meant the action-dependent reward differences suddenly represented a meaningful fraction of the total loss instead of getting lost underneath it, which gave the optimizer an actual reason to engage with the action-processing layers at all.

Leaky ReLU. Separately, I found a dead ReLU cascade sitting inside the advantage stream. While the oversized baseline had been dominating early updates, the pre-activations feeding the action-processing layers had been pushed permanently negative, and a standard ReLU, max(0, x), clips the gradient to exactly zero for any negative input. Once a unit died that way, it could never receive another update, no matter what changed afterward. Swapping in Leaky ReLU (F.leaky_relu, negative slope 0.01) fixed that directly: for x < 0, the derivative becomes 0.01 instead of 0, small, but nonzero, which is the entire difference between a permanently closed gradient path and one that stays open.

Proof of Life

Ran a fresh 15,000-step diagnostic with both interventions in place:

Evaluated State Q-Value      : 1.6346
Analytical Gradient (dQ/da)  : Steering: -0.00108051 | Throttle/Brake: 0.00395266
Critic Standard Deviation    : 0.0026 (curvature unlocked)

Steering gradient negative, throttle gradient positive, and neither one sitting at zero anymore. That negative steering gradient is the critic actually saying something now: from this starting frame, steering further right reduces expected future return, so steer straighter or slightly left. The positive throttle gradient says the opposite kind of thing: increasing throttle directly increases cumulative discounted return. Small numbers, but real direction behind them for the first time.

Worth noting why gradients this small could move anything at all. SAC trains with Adam, and Adam normalizes updates by the running root-mean-square of the gradient, which makes it scale-invariant to consistent sign. It's not the raw magnitude of a gradient that drives the policy somewhere, it's whether that gradient reliably points the same direction, update after update. Once the critic's landscape had any curvature at all, however faint, that was enough for Adam to actually steer the policy with it.

Phase 1: Turning LIDAR Into a Grid, for Real

The theory said: project raw LIDAR into a bird's-eye occupancy grid. The actual math behind that starts with converting an array index into a physical angle.

With 240 LIDAR rays sweeping a full circle, the angular step between rays is Δθ = 2π / 240 = π/120 radians, which works out to 1.5° per ray. Each ray's angle follows θᵢ = -i × π/120, negative because the sweep runs clockwise, something I'd confirmed empirically back in Part 4.

From there it's just polar coordinates turning into cartesian ones. Each raw LIDAR value is normalized between 0 and 1, so the first step is denormalizing it back into an actual distance: dᵢ = lidarArray[i] × 50 (50 meters being MetaDrive's max sensor range). Then standard polar-to-cartesian conversion: xᵢ = dᵢ × cos(θᵢ), yᵢ = dᵢ × sin(θᵢ).

Then cartesian coordinates have to land somewhere on an actual grid. With a 50-meter physical range mapped onto 64 cells, resolution comes out to 50 / 64 ≈ 0.78125 meters per cell. Center row and center column both sit at 32, the middle of the grid. Mapping x (forward/backward) to row and y (left/right) to column:

rowᵢ = 32 − floor(xᵢ / 0.78125) colᵢ = 32 + floor(yᵢ / 0.78125)

Clean enough on paper. It broke almost immediately once I actually ran it.

The Off-By-One That Wasn't Off By One

Running the projection function gave me row and column values of -28 and -15. Both negative. Both way outside the valid [0, 63] range a 64-cell grid can actually index into.

My first instinct was that this had to be some kind of indexing mistake, an off-by-one somewhere in the formula. It wasn't. The math was correct. The problem was the assumption baked into it: I'd centered the car symmetrically at row 32, column 32, giving it 32 cells of vision in every direction. But a car doesn't need to see equally far behind it as in front of it. Anything relevant enough to matter is almost always ahead. So a symmetric grid was wasting half its resolution looking at empty space behind the vehicle, and when something genuinely far ahead showed up, the math had nowhere left to put it.

I looked at three ways to fix it. Keep the same 50-meter physical range but bump up the cell count, which fixes the resolution problem but costs hardware, going from 4,096 cells to 16,384 is a real jump for a CPU-only setup. Shrink the max range down to 25 meters instead, which keeps the grid the same size but trades away how far ahead the car can actually see, a real safety concern at any real driving speed. Or keep the 64×64 grid as-is and just expand the physical area it covers, out to 100×100 meters, which sounds reasonable until you realize resolution then drops to 1.5625 meters per cell, coarse enough that the car effectively loses the ability to thread between obstacles in traffic.

None of those were quite right on their own. The actual fix was to stop assuming the car needs to be centered at all. Recenter the origin and shift it down in the grid: 48 cells of forward vision (37.5 meters) and only 16 cells of rear vision (12.5 meters). Same 64×64 grid, same resolution, but the vision budget goes where it's actually needed.

Phase 2: From Grid to Latent Vector

With a working grid, next came turning it into something the network could actually reason over.

First snag: PyTorch's Conv2d layer doesn't accept a flat 2D matrix. It expects a channel dimension representing feature depth, so a raw (64, 64) matrix has to become a (1, 64, 64) tensor, channels first, then height, then width. That channel dimension turned out to be useful for more than just satisfying PyTorch, though. Stacking the last four frames together, so the network can infer the speed of surrounding vehicles from how they move between frames rather than just where they sit in a single snapshot, gives a (4, 64, 64) tensor. And SAC doesn't train on one transition at a time anyway, it trains on random batches of past transitions, so the actual tensor shape flowing through the network is (batch_size, channels, height, width), which in practice meant (256, 4, 64, 64).

From there, three convolutional layers do the downsampling. Each uses kernel size 3, stride 2, padding 1, which halves the spatial dimensions cleanly at every step: 64 → 32 → 16 → 8. Channel count grows the opposite direction as spatial size shrinks: 4 → 8 → 16 → 32. Thirty-two channels is enough, since the network only needs to pick out geometric gaps and the rough rectangular blob of a moving vehicle, not fine texture. By the third layer, the original 64×64 grid has been compressed down into an 8×8 feature map, (32, 8, 8).

Then came the part I had to think through more carefully: getting that feature map into shape for the attention layer. A 2D convolution outputs (batch, channels, height, width). An attention layer wants (batch, sequence_length, embedding_dimension). So the CNN's (32, 8, 8) output effectively becomes 64 small "tokens," each one representing a different patch of the original grid, like slicing the image into an 8×8 checkerboard of tiles. Feed that sequence through nn.MultiheadAttention and it evaluates the relationships between distant patches of the grid, letting the network reason about a vehicle three lanes over and gaps opening up somewhere else on the board at the same time. Flatten the attention output back down and you get a dense 1D vector of 2,048 dimensions, torch.flatten(attn_output, start_dim=1).

One more question came up here that I didn't want to skip past: does the attention layer actually need positional information? Without it, two options existed. Give the network explicit positional encoding, telling it exactly where each tile sits in the grid. Or let it work that out on its own through training, which keeps the code simpler but asks more of the model. I tested it. The shallow CNN genuinely could not work out spatial position on its own, no matter how much training data it got, there just wasn't enough depth in three conv layers for it to infer geometry implicitly. So positional encoding it was, giving the RL agent a representation that's not just rich, but actually spatially aware.

The Rolling Buffer, and the Bug That Taught Me Something

Stacking four frames sounds simple until you have to actually implement "keep the last four, drop anything older" cleanly. Python's collections.deque with maxlen=4 handles that exact behavior, fixed size, push-new-drop-old, and it does it in O(1) time, which matters when this runs every single environment step.

That raised an immediate cold-start question, though: what happens at env.reset(), before four real frames even exist yet? I went with repeating the very first frame four times, effectively telling the network "the ego vehicle and all surrounding obstacles are currently stationary relative to my temporal window." Not perfectly accurate, but a reasonable assumption for a single instant in time.

Stacking order mattered too, entirely for my own sanity more than anything mathematically required: newest frame at the highest index, oldest at index 0, consistently, every time.

The wrapper architecture that ties all of this together works in five steps. The RL algorithm calls wrapper.step(action). The wrapper calls the underlying simulator's env.step(action), retrieving the raw LIDAR array. The wrapper passes that raw array into a stateless project_lidar_to_bev function and gets a 64×64 grid back. That grid gets appended to an internal deque capped at four frames. Finally the deque converts into a (4, 64, 64) tensor and gets handed back to the RL agent.

I finally got test_framestack.py running without crashing. And then immediately hit something worse than a crash: silent wrong output. The test reported newest frame changed after step: False. It was supposed to say True.

My first two questions, before touching any code: if there were no obstacles anywhere near the car at reset time, what would the occupancy grid actually look like, mostly zeros, or full of ones? And if the car has barely moved after a single step at zero throttle, should I actually expect the surrounding environment to have changed meaningfully in that one step, or would it plausibly look identical to the frame before it, simply because nothing happened?

That second question was the real flaw in my test design, not a flaw in the code. Two things were true simultaneously, and I hadn't put them together. Mathematically: the projection function I'd built specifically filters out and drops any LIDAR return of 0.99 or higher, treating "nothing detected" as empty space. In a scene with no obstacles nearby, that gives you a full 64×64 matrix of absolute zero. And physically: with zero throttle applied, there's zero propulsion, so the ego vehicle's velocity stays at zero and its displacement across that single simulated step is precisely 0.0 meters. When the frame stack pushes that new grid into the deque, it's mathematically identical to the one before it, and the test correctly reported that the newest frame hadn't changed. Because it genuinely hadn't.

So the bug wasn't a bug. It was a test that couldn't have passed under those conditions no matter how correct the underlying code was. I fixed the test itself in three steps: added a debug print checking torch.sum(obs), since a sum of exactly zero confirms the empty-scene theory directly. Forced an actual physical state transition by changing the test action to [0.0, 1.0], maximum throttle. And accounted for spatial resolution, since a single 0.1-second step at low starting speed might only move the car a few centimeters, nowhere near enough to shift an obstacle's projection into a genuinely different grid cell. To meaningfully test the rolling buffer at all, I needed a loop applying full throttle across ten to twenty steps, not one.

And even after all that, the sum still came back zero. Which meant the actual bug wasn't in the frame stack logic at all, it was sitting one layer deeper, back in project_lidar_to_bev itself. That's the same off-by-one-that-wasn't from Phase 1, the -28/-15 out-of-bounds row and column values. Two separate debugging threads, same root cause, and I only found the real one because the "obvious" symptom in the frame stack test forced me to question an assumption I hadn't actually verified yet.

Phase 3: Teaching SAC to Speak in Steering Angles

MetaDrive expects a continuous action between -1 and 1. SAC's actor, by construction, samples from an unbounded Gaussian distribution. Somewhere between those two facts, you need a function that squashes an unbounded real number into a strict range, and the standard choice is the hyperbolic tangent, tanh, in PyTorch.

Worth being explicit about why this has to be tanh specifically and not something simpler like clamping. Clamping isn't differentiable at the boundary, the gradient just dies the moment a value hits the clip. SAC needs gradients to flow cleanly through the entire action-generation step during training, so the squashing function has to be smooth everywhere, not just bounded. tanh fits: a_squashed = tanh(a_raw), and its derivative, d/dx tanh(x) = 1 − tanh²(x), stays well-defined across the whole range, which is exactly what backpropagation needs to work through it.

The Change-of-Variables Trap

Here's the part that isn't obvious until it bites you. Once you apply a nonlinear transform like tanh to a random variable, you fundamentally alter its probability density. The log-probability SAC needs for training isn't just the log-probability of the raw, pre-squash Gaussian sample, that number is now wrong, because the actual distribution of actions is the squashed one, not the raw one.

Fixing that means applying the change-of-variables formula from probability theory, sometimes called the Jacobian correction:

1. a_raw = μ + σ × ε

2. Compute the normal log-probability of a_raw under the Gaussian.

3. a_squashed = tanh(a_raw)

4. Jacobian correction: log_prob = normal_log_prob − log(1 − tanh²(a_raw) + ε)

That correction term increases the final log-probability, makes it less negative, and it took me a minute to convince myself why that direction made sense. The same amount of total probability mass, all of it, 1.0 worth, is now squeezed into a drastically smaller area once you've squashed the distribution through tanh. Squeeze the same mass into a smaller space and density has to go up. Subtracting that negative log-density term is what correctly accounts for that physical compression of probability mass. Get this step wrong and SAC's entropy term, the thing actively encouraging exploration, ends up computing exploration incentives against a distribution that doesn't actually match the agent's real behavior.

One implementation detail worth flagging on its own: that small epsilon inside the log has to be placed directly inside the log, not added somewhere else in the expression, specifically to protect the codebase from a floating-point failure when 1 − tanh²(a_raw) gets close enough to zero that the log would otherwise blow up.

Where Exactly Does tanh Get Applied?

I wanted the precise lifecycle of the tensor clear in my head before calling this piece of the actor done. The actor network evaluates the raw LIDAR state and outputs the parameters of a Gaussian distribution, mean and standard deviation. Using the reparameterization trick, it samples a standard normal noise vector, ε ~ N(0,1). That gets combined into the unbounded raw action, a_raw = μ + σ·ε. Only at the very end of that sequence does the unbounded sample get squashed into the strict [-1, 1] range MetaDrive actually requires, a = tanh(a_raw).

Steering and Throttle Aren't the Same Question

The actor outputs two independent values, steering and throttle, each with its own log-probability. If two events are genuinely independent, the probability of both happening together is P(A) × P(B), and given log-probability values already, that becomes log(P(A)) + log(P(B)). Which means combining the two dimensions correctly requires .sum(dim=-1), not .mean(). Summing is the algebraically correct way to compute a joint log-probability across independent events. Averaging would quietly fabricate a probabilistic concept that doesn't actually exist, there's no meaningful "average log-probability" of two unrelated control decisions.

And one more question that seemed worth settling explicitly: does a fully deterministic action, at evaluation time, still need to go through tanh? Yes, unconditionally. The squashing function isn't just a training-time trick, it's required to safely compress whatever the raw neural output is into a valid physical control signal, stochastic sample or not. The log-probability calculation, on the other hand, genuinely is only meaningful during training, so it's intentionally skipped at evaluation time rather than computed and thrown away.

Phase 4: The Training Loop

Strip away all the RL terminology and the training loop does exactly three things. It runs the agent through the environment, collecting experience. It stores that experience somewhere. And it periodically samples from that storage to actually update the network's weights.

Storing Experience Without Bottlenecking the CPU

First instinct was a Python list of tuples. Wrong instinct, that bottlenecks the CPU almost immediately once you're storing tens of thousands of transitions and sampling from them constantly. The fix was a custom replay buffer built on pre-allocated, fixed-size NumPy arrays instead: self.obs_actor_buffer = np.zeros((capacity, 4, 64, 64), dtype=np.float32), with matching pre-allocated arrays for actions, rewards, and everything else. Sampling a batch just means drawing random indices, np.random.randint(0, current_size, batch_size), and indexing straight into those arrays.

Soft Target Updates

Target networks stabilize training by moderating how fast Q-value estimates are allowed to shift, and the mechanism for that is Polyak averaging: θ_target = τ·θ_live + (1 − τ)·θ_target, where τ sits around 0.005. Each update absorbs only about half a percent of the live network's new values, retaining 99.5 percent of what the target network already knew. Slow, deliberate, and exactly what you want when the thing you're trying to stabilize is itself a moving target.

Assembling What the Critic Actually Sees

The critic's full input vector breaks down into three segments: [0:19] is ego kinematics plus navigation, unchanged from the raw observation. [19:35] is the privileged sixteen-dimensional num_others state. [35:275] is the full 240-ray LIDAR sweep. That layout matters, since getting the slice indices wrong here means silently feeding the critic garbage instead of the ground-truth it's actually supposed to have.

A Buffer That Was Too Big to Actually Exist

I'd initially set replay buffer capacity to 100,000, which sounded reasonable until I actually computed the memory footprint. capacity × 4 frames × 64 height × 64 width × 4 bytes works out to roughly 6,553,660,000 bytes, about 6.10 GiB, just for the obs_actor_buffer. And the replay buffer also has to instantiate an identically-sized next_obs_actor_buffer right alongside it, doubling that to roughly 12.20 GiB. Not remotely reasonable for a CPU-only setup with limited RAM. The fix was simply reducing capacity down to somewhere in the 25,000 to 50,000 range instead.

Terminated Isn't the Same as Truncated

"Done" isn't a single concept, it's actually two different signals wearing one name. terminated = True means the episode genuinely ended, the vehicle crashed, or it safely reached its destination. truncated = True means something entirely different: the vehicle was driving perfectly safely the whole time, but an internal step-count clock simply ran out. Conflating those two would teach the agent that running out of time is somehow the same kind of failure as crashing into a wall, which is exactly backwards. The fix is separating the signals explicitly: done_buffer = 1.0 if terminated, and 0.0 even if truncated is true, since a truncated episode shouldn't zero out the Bellman target the way a genuine terminal state should.

Redesigning the Frame Wrapper Around All of This

Once the asymmetric actor-critic design from Part 1 was actually being implemented, the wrapper needed a cleaner separation of responsibilities than it started with.

The wrapper's own job is narrow: the dictionary coming out of env.step() has to contain exactly two raw, unadulterated tensors, the (4, 64, 64) stacked bird's-eye grid, and the (16,) num_others vector. Nothing processed, nothing latent, just the raw material.

The replay buffer's job is to respect that separation, too. It can never store latent vectors, only the raw grid in one array and the raw num_others in a completely separate one. Otherwise you'd be baking a specific version of the CNN's weights into stored data that's supposed to outlive many rounds of training on that same CNN.

The actual assembly happens at update time, in four steps. Sample the raw batch, a grid tensor of shape (256, 4, 64, 64) and a num_others tensor of shape (256, 16). Run a dynamic forward pass, pushing the grid batch through the Phase 2 CNN-and-attention module to get a latent batch of shape (256, 2048). Route to the actor: since its state representation relies entirely on spatial geometry, obs_actor = latent_batch, full stop. And assemble the critic's privileged input by concatenating the latent with the ground-truth vehicle data: obs_critic = torch.cat([latent_batch, num_others_batch], dim=1).

Why Any of This Actually Matters

Two reasons this structure earns its complexity, and neither is just theoretical.

Algorithmic stability and variance reduction. Continuous control RL runs into a real scarcity of high-quality samples, which is exactly why experience replay exists, to store and reuse transitions instead of learning from each one once and discarding it. If the vehicle executes a long, sweeping left turn across fifty consecutive steps, and you perform fifty consecutive gradient updates purely on that stretch, the network forgets the generalized knowledge it previously built up about ordinary straight-line driving. Collecting a batch of varied experience before updating lets the replay buffer feed the network a wider, more representative slice of the actual driving problem, not just whatever's happening in this exact moment.

Computational efficiency and hardware constraints. Decoupling data collection from the actual network update lets the system balance its workload deliberately instead of doing both at once on every single step. In practice that means tracking a global step counter, using it to dictate the warmup-phase logic, and checking it against a modulo operator to decide exactly when an update actually fires.

That's Phase 4, and with it, the full pipeline: raw LIDAR in, a trained continuous-control policy out. What's left for a possible Part 6 is the actual training run itself, whatever that ends up looking like once it's been running long enough to say something real about it.

So the first question I had to answer, before making any assumptions, was simple: what does MetaDrive actually hand you by default? I knew it was LIDAR-based, raw distance readings fanning out around the car. But raw readings aren't a decision. Something has to turn them into one.

That left me with two options. Let the network figure the geometry out on its own, straight from the raw scan. Or process everything down into a clean set of, say, sixteen state values first.

Option two sounds smarter on paper, right? Structured, tidy, easier to reason about. Except Indian roads don't give you lanes to structure around. There's no clean line to snap a state vector to. So option one it was, the messier, dynamic one, because it's the only one that doesn't assume a road that doesn't exist.

Bear with me here, there's going to be a lot of buts and what-ifs.

Going with the raw route meant the network had to figure out the geometry itself. And that's where the curse of dimensionality showed up, along with what I've started calling exponential data hunger. With sixteen clean states, the state space stays tight and manageable. Feed it raw LIDAR instead, and that space explodes into the thousands. Which means you lean harder on a deep neural network to make sense of it. But a network that size doesn't learn a workable policy in a thousand steps. Not even close.

So DQN and PPO were out before I'd really started.

Then there was representation learning. Before the model can learn anything about driving, it first has to learn what an edge is, what an object is, basically relearn the world from scratch. And that means every time the car hits new terrain, it's back to square one, re-learning structure it should already know. Which defeats the entire point of a generalizable driving agent.

On top of that: a brutal credit assignment problem. If the car hits a wall, it has no idea which of the last however-many actions actually caused it. Was it the steering three steps back? The braking before that? No signal tells you.

So, how do you fix all of that at once?

State abstraction. Instead of raw points, work off a 2D bird's-eye view, an occupancy grid. Compress the chaos into something spatial and structured before the network ever sees it.

A specialized architecture: CNN plus self-attention. A plain multi-layer perceptron just won't cut it here, there's too much spatial structure for a flat network to hold onto. But run the data through convolutional layers alongside self-attention and you get something workable. The attention layer flags what actually matters locally, a vehicle closing in fast, say. The CNN then compresses the broader spatial picture into a dense, lower-dimensional vector. From there, the agent, whether it's PPO or DQN, operates on that latent vector instead of wrestling with the raw point cloud directly.

That's when something I'd read earlier came back to me, and led straight into the next question.

What about actor-critic? (Spoiler: this is the one that actually works.)

It handles high-dimensional state spaces well, which tracks, since that's exactly what a chaotic LIDAR point cloud is. It solves the continuous control problem too, the steering wheel being the obvious example. And it tames training instability through bootstrapping, which lets you actually trace which action led to which reward. That's my wall-collision problem, solved. Add baseline subtraction on top and you're not fighting numerical instability either.

Is that the whole story? No. Unfortunately, no.

Because now you have to pick which actor-critic algorithm to actually use:

• Deep Deterministic Policy Gradient (DDPG)

• Twin Delayed DDPG (TD3)

• Soft Actor-Critic (SAC)

I'll get into each of those properly in a bit. First, another fork in the road.

Discrete or continuous? Worth flagging: SUMO had me working in discrete action space. MetaDrive's data isn't built that way. Steering angle, throttle, brake pressure, all of it is continuous by default.

And you can't just bolt a value-based policy like DQN onto continuous actions. Think about what DQN actually needs to do: pick the action with the highest expected reward, which means computing the argmax of a Q-function. Fine in a discrete space where you can just check every option. In a continuous one, there is no fixed set of options to check, the argmax itself becomes an optimization problem you'd need to solve at every single step. That's not a policy, that's a bottleneck.

Actor-critic sidesteps that entirely, which is exactly why it kept coming back around.

The discrete path isn't dead on arrival either, to be fair. You don't have to go continuous. Dueling DQN improves on the base architecture by splitting evaluation in two: one stream estimates how good it is to just be in a given state, the other calculates the specific advantage of a particular action from there.

So before deciding, I ran the actual trade-off. Discrete: if your intervals are too coarse, the controls get jerky and the car crashes at higher speeds. Make the intervals finer and the number of possible action combinations explodes, which slows Q-learning down to a crawl. Continuous: heavier training instability, and hyperparameter tuning that gets complicated fast.

Then a dumb but honestly useful question hit me: why wasn't SUMO jerky in the first place? Because SUMO was never a high-fidelity vehicle dynamics simulator to begin with, it's a traffic flow simulator. It didn't need to be smooth at the wheel level. Translate that same discrete logic into something meant to resemble a real car, though, and it falls apart. Take -1 for full left and +1 for full right and the steering only ever sits at the extremes. No in-between. What you actually need is finer resolution, real control, not a switch.

Can discrete threading actually get a car through real traffic? No.

Which brings it back to what MetaDrive expects by default anyway: a 2D continuous vector, a float between -1 and 1 for steering. It runs on a physics engine built around continuous tire friction and momentum. Force discrete steps into that and you get artificial jerkiness that fights the physics rather than working with it, and at speed, that's the kind of thing that destabilizes the whole vehicle.

Conclusion: continuous it is.

Before I get into why I landed on SAC specifically, there's one more problem worth pulling apart, because I actually mislabeled it at first. I called it an "RL algorithm compute problem." It isn't. It's a perception network compute problem.

Here's the distinction: whichever actor-critic algorithm I end up choosing, TD3 or SAC, doesn't change the fact that I still need a feature extractor my CPU can actually run in a reasonable time frame. No dedicated GPU here. So the fix isn't in the RL algorithm at all, it's upstream. Break the raw data down into something abstract, a 2D bird's-eye grid instead of a full point cloud, and lean on optimization tricks like model pruning to keep it lean.

Step one, then, isn't picking SAC or TD3. It's building a genuinely lightweight network for the perception layer, something small enough to actually run on what I've got.

Picking an Algorithm

Three actor-critic candidates, and I went through them one at a time.

DDPG. How it works: it takes deep Q-learning and drags it into continuous control, merging the strengths of deep learning with policy gradients to output a specific continuous action directly. On the plus side, it's sample efficient and gives clean gradient signals. On the downside, it's known for overestimation bias, the Q-value estimates run high, and that instability means you basically need a dual critic setup just to keep it stable.

TD3, Twin Delayed DDPG. Built directly on top of DDPG to fix that overestimation problem. The core trick is the "twin" structure: two independent critic networks evaluate the actor's action, and you take the minimum of the two instead of trusting either one alone. It also delays the actor's policy updates so they lag behind the critic, which helps stability. Pros: stable, converges well, fast at inference. The catch is that it's still deterministic at heart, and deterministic policies tend to struggle in chaotic, highly dynamic environments. Which, given what I'm building for, is basically the whole job description.

SAC, Soft Actor-Critic. This one solves a different failure mode entirely, an agent converging prematurely into a narrow policy and never really exploring past it. It optimizes for both maximum expected reward and maximum policy entropy at the same time, entropy regularization, so the agent stays a little uncertain on purpose instead of locking in early. The cost is higher computational overhead.

So, TD3 or SAC? SAC made more intuitive sense for the actual driving problem, but I didn't want to pick it just because it felt right. I wanted to know where that overhead was actually coming from before committing.

Turns out both algorithms share the same underlying structure, dual networks either way. The overhead isn't in the architecture or the input. It's in the output. TD3 outputs a single, exact continuous action. SAC outputs a full continuous probability distribution instead, and sampling from that costs more. But that extra cost is negligible in absolute terms. It only becomes a real bottleneck if you're running something like PPO on a dedicated GPU at scale, not for an actor-critic setup running on CPU. So: SAC it is.

Which meant the real bottleneck was never the RL algorithm to begin with. I'd been calling it an "RL algorithm compute problem," but the honest name for it is a perception network compute problem. Whichever algorithm I picked, TD3 or SAC, I still needed a feature extractor my CPU could run in a reasonable time frame. So step one wasn't the RL side at all. It was designing a genuinely lightweight CNN for perception, and figuring out exactly how lightweight it could get before it stopped seeing anything useful.

Designing the Lightweight CNN

A few concrete moves here, each one trading raw capacity for speed:

• Drop self-attention. Just abandon the attention layers entirely for the base network. They're expensive, and expensive is the one thing a CPU-only pipeline can't afford by default.

• Keep the convolutional depth shallow. Two to three layers, max. The goal isn't a rich hierarchical feature extractor, it's just enough to pull out the macroscopic geometry: road boundaries, obstacle shapes, nothing fussier than that.

• Downsample aggressively. Large strides or max-pooling right after the initial convolutions, shrinking the spatial dimensions fast so the CPU isn't pushing floating-point operations through a huge feature map at every step.

• Go coarse on input resolution. A 256×256 grid is overkill. Bring it down to 64×64, or even a 32×32 binary grid, and you've cut the workload dramatically without losing the information that actually matters for driving decisions.

Which gave me a rough final shape: MetaDrive's raw LIDAR gets manually projected into a low-resolution bird's-eye occupancy grid, that grid gets fed into a shallow, downsampled CNN, and the CNN compresses the spatial geometry into a dense one-dimensional latent vector. That latent vector gets handed to SAC's actor and critic heads, which evaluate the state and output continuous steering and throttle commands.

But I'd dropped attention entirely to get there, and that came back to bite me almost immediately.

The Receptive Field Problem

With only two or three conv layers, the effective receptive field, meaning the total area of the original input that actually influences a single node deep in the network, stays small. On a 64×64 grid, that works out to something like a 10×10 local patch. The network can see what's directly around a point. It can't see the whole board at once. Which means it literally cannot calculate that a vehicle three lanes over is closing in fast, because that information never reaches the same part of the network at the same time. I needed something in between full self-attention and no attention at all.

The middle option: place attention strategically instead of everywhere. Let the shallow CNN do its job compressing the 64×64 grid down to something like an 8×8 feature map first, and only then apply self-attention, on that already-compressed representation. At that resolution, the attention layer only has to reason about relationships between 64 elements instead of thousands, which is cheap enough to actually run in real time, and it's exactly what lets the model "thread through traffic" instead of only reacting to whatever's immediately adjacent.

I still wanted to check the assumption that attention is expensive before committing to skipping it everywhere. Computing self-attention on a raw 64×64 grid directly would absolutely stall a CPU control loop. Computing it on an already-compressed 8×8 feature map is a different story entirely.

Final call: a middle-ground architecture. Build a shallow CNN to rapidly downsample the grid, then attach a single self-attention head to the compressed feature map so the agent can reason across distant gaps in the scene, while staying inside a budget that actually fits a real-time CPU loop.

Let's Start Building: Phase 1, Occupancy Grid Projection

Theory's one thing. The first real step was turning MetaDrive's raw output into a usable BEV grid, and before writing any of that, I wanted to actually see what MetaDrive's LIDAR output looks like structurally, not just conceptually.

I ran a quick script to read the raw observation and just looked at it. Three sensors were active: lidar, side_detector, and lane_line_detector. What came back was a 256-value array, every value between 0 and 1. So the float values were some combination of all three sensors mixed together, and my first real task was just isolating the LIDAR portion from that combined vector.

Digging into MetaDrive's observation code led me to two relevant classes: StateObservation and LidarObservation.

StateObservation.observe() builds a state vector out of the ego state (nine-plus values: speed, steering, heading difference, and so on) concatenated with navigation info (ten values covering checkpoint distance and direction).

LidarObservation.observe() builds on top of that. Its final vector is the state (ego state plus navigation) concatenated with other-vehicle info, meaning nearby vehicle data plus the raw LIDAR points themselves.

The documentation claims 275 total values. I was getting 259. A sixteen-value gap. My first guess was that the missing sixteen was the "other vehicle info" block, so I went digging through the config to check the defaults. Turned out num_lasers = 240 and num_others = 0, meaning zero nearby-vehicle slots were being included at all. 240 plus the 19 remaining values, nine for ego state and ten for navigation, lines up exactly to 259.

So the 259-length vector breaks down as:

• [0:9] → ego vehicle state (speed, steering, and so on)

• [9:19] → navigation info (checkpoint distance and angle)

• [19:259] → 240 raw LIDAR distance readings, sweeping clockwise starting from the vehicle's heading

Now the real question: what do I do with num_others sitting at zero?

Option one, turn it on and use the structured vehicle info directly. Tempting, but it defeats the entire point of the project. If I hand the network relative speed and object identification for free, that's not perception, that's a cheat sheet. A real sensor gives you detailed contextual information the hard way. Using num_others as a shortcut means the model never actually learns to work with an unstructured environment, which is the whole reason this project exists in the first place.

Option two, pure LIDAR only. The problem there is you genuinely can't distinguish a car from a wall using raw distance readings alone. Same signal, different object.

So if I do end up using num_others, there are two ways to structure it. A purist end-to-end approach, strictly LIDAR, full stop, which pushes training time and compute cost up hard. Or an asymmetric actor-critic setup, learning with privileged information, where I use the num_others ground truth without actually violating the constraint I set for myself. The actor network receives only the raw stacked LIDAR occupancy grid and never sees num_others at all. The critic network receives the raw LIDAR plus the perfectly clean num_others array. The actor stays honest to a real sensor's limitations. The critic gets to be the all-seeing teacher grading its decisions. That approach works cleanly.

Making the Asymmetry Actually Work

Before committing, I had to check something: does SAC's standard formulation even support different inputs for the actor versus the critic? Looking at the underlying Markov decision process structure, the actor-critic architecture already maintains two separate functional representations by design. The actor is an explicit policy, π_θ(a|s), mapping a state to a probability distribution over the action space. The critic is a value function, Q_φ(s,a), mapping a state-action pair to a real number representing expected cumulative reward. Nothing in that structure requires the actor and critic to be looking at the same input. So asymmetry isn't a hack bolted onto SAC, it fits the existing shape of the algorithm. It just meant restructuring my pipeline to actually take advantage of it.

A few concrete changes fell out of that:

Redesigning the replay buffer. A standard SAC buffer stores transitions as (state, action, reward, next_state, done). To support asymmetry, that has to expand and decouple the observable state from the privileged one: (obs_actor, obs_critic, action, reward, next_obs_actor, next_obs_critic, done), where obs_actor is the raw LIDAR occupancy grid and obs_critic is that same LIDAR grid plus the clean 16-value num_others array.

Splitting the network input dimensions. The actor network's input size is just the LIDAR latent dimension. The critic networks take a wider input: LIDAR latent dimension plus num_others dimension plus action dimension.

The critic's forward pass, step by step. Think of it almost like a blind student and a sighted teacher. The target actor is the blind student, asked to rely only on next_obs_actor (raw LIDAR) and make a decision from that alone. The target critic is the teacher who can see everything, it takes next_obs_critic (LIDAR plus the clean vehicle data) along with the actor's proposed action and estimates the future value of that decision. Since the critic has the full picture, it computes an accurate target Q-value, the correct grade for that action. The loss update then compares that target against the critic's own earlier estimate and nudges the critic closer to the accurate target.

The actor's forward pass. Because the critic has access to num_others, its Q-value estimates come out sharp, stable, and accurate, which means the gradient signal it feeds back to the actor is precise about exactly why a given steering angle was good or bad. The actor never sees the privileged data directly. It just gets better at acting on what it does have, guided by a critic that understands the scene more fully than it ever will.

The core formulas, for the record:

Target Q-value:target_Q = reward + γ × (min(target_critic1(next_obs_critic, a′), target_critic2(next_obs_critic, a′)) − α × log_prob)

Both target critics evaluate the proposed next action, a′, and we take the minimum of the two to correct for the overestimation problem TD3 first identified. Entropy, the mathematical measure of spread in a probability distribution, gets folded in through α × log_prob, controlling how much the update prioritizes exploration. That combined target is what trains the two primary critic networks, compared against their current Q-value estimate to check how accurate they are.

Actor loss:loss_actor = α × log_prob − min(critic1(obs_critic, a), critic2(obs_critic, a))

The actor is maximizing the negative of that minimum critic estimate, in effect getting graded conservatively on safety, since the critic always takes the more cautious of its two estimates. The +α × log_prob term maximizes entropy on top of that, actively pushing the actor to keep exploring rather than settling too early.

A few smaller details worth nailing down while I was at it: a′ has to be sampled from the target actor network, and that network only ever sees next_obs_actor, never the privileged data. The current action a, meanwhile, comes from the live actor network through the reparameterization trick, a = μ + σ × ε. And obs_critic should never require gradients, it has to stay a frozen input tensor, since it's ground-truth context, not something the network should be learning to reshape.

One Small Thing That Almost Broke Everything: What Does a LIDAR Reading of 1.0 Actually Mean?

Small detail, but it mattered. MetaDrive returns a normalized reading of 1.0 whenever there's nothing within the ray's 50-meter range, the ray just never bounces back. Read naively, the model could end up interpreting that as the car being boxed in on every side, trapped in a cylinder of walls that don't actually exist, since "maximum distance" and "wall right here" can end up looking suspiciously similar once everything's normalized between 0 and 1.

The fix: filter out any ray returning 0.99 or higher and treat it as empty space instead, dropping it to 0.0 in the occupancy matrix so open space actually reads as open. I also considered just binarizing the whole grid to 1s and 0s at that point, occupied or not, and confirmed it doesn't break anything downstream either.

That's Phase 1: the reasoning that got me from "what does MetaDrive even give me by default" to a working occupancy grid pipeline and an asymmetric SAC setup that respects the constraints I set for the project. Phase 2 picks up from the actual training run, and everything that broke along the way.

It made me think, how? On Indian roads of all places. Personally, living and driving on Bangalore roads is a near-impossible task even as a human, the unpredictability, the lack of road rules, the aggressive traffic, and so much more.

Looking further into Swaayatt, I understood that they use the concept of passive-aggressive driving. But before I could get into that, I had to learn reinforcement learning first, thanks to a pile of Stanford and MIT lectures on deep RL. There were three main components. What can the agent do? That's the action space. What can the agent see, or what state can it be in? That's the state space. And how do we tell if the decision the agent made was good? That's the reward system.

In theory I had it all down, but there was only one way to know if it actually worked, and that's where the 2D sim came into the picture. Starting off simple, I tested my agent on a straight road, and of course even that had its challenges. So let me start from the beginning.

Given a straight road, starting off with just two lanes, one-directional traffic, I asked myself three main questions.

What do I need the agent to do? Taking reference from how I drive and what I actually do behind the wheel, steering, accelerating, braking were the basic actions. I needed my agent to be able to move left or right, go straight, speed up, or slow down.

What does the agent need to observe or look at? What do I look at when I'm driving? I look at the road, the car in front if there is one, and if there's anything behind, obstacles. I act based on that.

And what do I reward it for? Not crashing, speeding up when it's safe to, overtaking, switching lanes, and so on.

Now that I had a brief idea about the components, what do I actually do with them?

That's where Markov Decision Processes (MDPs) came in. What is an MDP? It's the mathematical framework used to define the environment in RL. It's a 5-tuple system: (S, A, P, R, gamma).

S is all possible states or configurations the agent can be in. A is the set of all valid moves or decisions the agent can make. P is the transition probability, also known as the rules, for example, if the agent is in state x and takes action y, what's the probability it ends up in state z, that's the Markov property. R is the reward function, the numerical feedback the agent gets after an action.

State + action → next state + reward.

The goal was to maximize reward, so the agent has to think about long-term outcomes, not just the next step. That's where gamma comes in, the discount factor, ranging between 0 and 1, controlling how much the agent cares about distant future rewards versus immediate ones. 0 is short-sighted, 1 is long-sighted.

Once I had that working, I moved on to four lanes.

With four lanes, what the agent sees, the state space, changes. Not just the lanes to the right or left of the ego car, but the relative speed of the cars around it too. Rather than just checking distance, I needed the agent to know if the car ahead was speeding up or slowing down.

That's not all either, I needed the agent to observe itself: what speed the ego car was going, what lane it was in. There wasn't much else to explore here, the car just had to travel down a straight road, switching lanes when needed.

Moving on to junctions, there are a few possibilities when a car approaches one: take a U-turn, turn right or left, or go straight.

The car needed to understand it has to slow down and not crash while performing any of these, and not just that, the steering changes too, right?

That was my first thought, that the steering angle varies, and unlike switching lanes, the agent would need to execute a whole U-turn or a full right turn.

Except that's not actually true, and it took me a while to see why.

I sat with it and realised I was mixing up two different things. Turning at a junction and switching lanes are both just "steering left or right" at the end of the day, same wheel, same two hands. What's actually different is when the car does it and why. A human doesn't have a separate turning mechanism for junctions, you just use the same steering wheel you already have, you just know it's time to use it because you can see the junction coming up.

So the fix wasn't a bigger action space. It was giving the agent the same information a human already has: knowing a junction exists, and knowing which way it needs to go. Two new things to sense, not new things to do.

I added two values to what the agent sees, how far away the next junction is, and what turn it's supposed to make there, straight, left, right, or a U-turn. That last part matters because on my sim there's no real destination the agent is picking, I just assign it a route at the start of every episode, kind of like Google Maps already telling you to turn left in 200m instead of the car figuring out on its own where it wants to go. The car's whole job is to execute that safely, not decide it.

Steering stayed exactly the same, left, right, or maintain. Speed too. The agent just had to learn, from those two new numbers plus everything it already knew about the cars around it, when to get into the right lane, when to slow down, and how to time the turn without getting hit.

I genuinely wasn't sure this would work without a bigger action space. Tested it, and it did. I watched the car actually wait for a gap in oncoming traffic and then cut across once it was clear, one of those moments where you realise the model is doing something you didn't explicitly tell it to do, it just learned that waiting is sometimes the better move.

Along the way I caught two mistakes I made myself, worth mentioning because they're the kind of thing that doesn't throw an error, they just quietly produce the wrong answer.

First, I had the relative speed of a car behind me calculated the exact same way as a car in front of me. Sounds fine until you actually think about it. If a car ahead of you is going slower than you, you're closing in on it, that's the dangerous case. But if a car behind you is going slower than you, that's completely safe, they're falling back. Same formula, opposite meaning depending on direction. I had to flip the subtraction for anything behind the car so that "the gap is closing" always means the same thing everywhere, front, back, either side.

Second, and this one was dumber and took longer to catch. I set up a junction route thinking I'd built a right turn, car goes north into the junction then exits east. Except east isn't right, it's right on a map, not right from inside the car. If you're facing north and you turn toward east, that's a left turn from where you're sitting. I'd been reading the map like a bird instead of like a driver. Fixed the route, and once I thought about it properly it was obvious, but for a solid chunk of debugging I was convinced the code was broken when actually my own assumption was wrong.

By the end of the SUMO phase, I had a car that could handle a straight road full of genuinely chaotic traffic, some vehicles polite, some aggressive tailgaters, some crawling along, and a junction where it had to negotiate crossing traffic to turn. I ran it a bunch of times with randomised traffic each run so it wasn't just memorising one lucky scenario, and it held up.

But SUMO isn't a real driving simulator, it's a traffic flow simulator. No real physics, no cameras, no lidar, the car just knows exactly where everything is because the simulation tells it directly. Which is fine for figuring out the decision-making side of things, but it's not testing anything close to how a real self-driving car actually perceives the world. That's where the next part of this comes in.

Turns out China isn't picking a side in the vision-vs-lidar debate at all. They're doing both, aggressively, and building an entire road network to back it up. And then there's India, which doesn't fit into either camp because the roads themselves refuse to cooperate with any of this.

So this part is the "everyone else" part.

China Isn't Choosing a Lane

Tesla bets on cameras. Waymo bets on cameras plus lidar plus a fully mapped world. China's mainstream approach sits in between those two, leaning hard into what's called multi-sensor fusion, cameras, lidar, and radar together, run through end-to-end neural networks. Not a compromise exactly. More like refusing to accept the premise that you have to pick one.

A few names came up over and over while I was reading through this.

Baidu Apollo is the closest thing China has to a Waymo. Apollo Go runs as a fully driverless L4 robotaxi service in over ten cities, Wuhan and Beijing among them, and their newest robotaxi, the RT6, carries 38 sensors. Thirty-eight. Compare that to Waymo's 23 and you start to see the philosophy, redundancy on redundancy, because if one sensor gets confused, three others are still watching.

Huawei doesn't even make cars, they license the tech to companies like BAIC and Changan. Their system, ADS, runs on what they call a "World Action Model" alongside a general obstacle detection network, trained specifically on virtual edge cases. Which is a genuinely smart move when you think about it, you can't wait around for a real cow to wander into traffic to teach your model what a cow wandering into traffic looks like, so you simulate it instead.

XPeng is the one that actually surprised me. Their high-end cars run sensor fusion, two lidars, five radars, two dozen cameras, feeding into a Vision-Language-Action model they call X-Mind. But their budget model, the Mona M03, strips the lidar out entirely and goes pure camera. That's Tesla's whole philosophy, just imported into a fifteen-thousand-dollar car. Same company, two completely different bets depending on the price point.

BYD does this even more explicitly with their "God's Eye" system, split into three tiers. Tier A gets three lidars for the luxury Yangwang U8. Tier C, on a car that costs about ten thousand dollars, is pure vision, no lidar at all. So the debate I spent all of part 1 treating as an either/or is actually something BYD just solves by tier. Rich people get redundancy, everyone else gets cameras. Practical, even if it's a little uncomfortable to say out loud.

Mapless Driving, and the Smart Road Idea

Here's something I hadn't really considered before this. HD maps are expensive and slow to update, and China's biggest players, Huawei, XPeng, Li Auto, are moving toward "mapless" driving instead, where the car just reads the road live instead of relying on a pre-built map of it. This solves Waymo's actual weak point, the whole geofencing thing, needing a city to be mapped down to the centimetre before the car can drive there at all.

But mapless still leaves the camera-blindspot problem sitting there, unsolved. That's where V2X comes in, and it's honestly the part of this research that made me stop and think the longest.

The idea is simple enough on paper: instead of only making the car smarter, make the road smarter too. Cameras mounted on street poles feeding perception data straight into the car. Traffic lights talking to vehicles over 5G. A car that can't see around a truck getting the missing picture handed to it by the infrastructure instead of guessing.

There's one example that stuck with me. A construction site, red light, but a worker waving traffic through anyway, the kind of situation a purely visual AI genuinely struggles to parse because the rules and the reality are contradicting each other. With V2X, that worker could carry a chip that overrides the signal directly for the car. Could. Not does, not yet. Which is the catch with basically all of this, it's a real solution to a real problem, sitting entirely in the future tense.

And that future tense matters more the further you get from China. This kind of coordinated infrastructure only really works if a government can mandate it across an entire country at once. The US can't do that cleanly, every state sets its own rules. Realistically, most of the world can't either. Smart traffic lights sound great until you ask who's paying to install them in a city that still has potholes nobody's filled in a decade.

BYD vs Tesla, Away From the US

If you actually zoom out to markets where both companies compete freely, Europe, Southeast Asia, South America, the story flips almost entirely. BYD dethroned Tesla as the world's top-selling EV maker in 2024, 4.27 million vehicles against Tesla's 1.78 million, and has since started outselling Tesla monthly even in the UK.

The reason comes down to something I didn't expect going in: BYD builds almost everything itself. Battery, chips, motors, all in-house, which they estimate gives a car like the Seal something like a $4,700 structural cost edge over a comparable Model 3. And "God's Eye" ships free on most of their lineup, while Tesla still charges $8,000 upfront for FSD.

The engineering philosophies genuinely don't match up. Tesla is software-first and deliberately minimal, one neural network taking raw pixels straight to steering output, trained on their own Dojo supercomputer. BYD goes the opposite way, wiring the AI directly into the physical chassis through their DiSus body control system. Their cars can read the road ahead with lidar and adjust the suspension before hitting a bump, or independently torque each wheel during a blowout to keep the car stable on three wheels. Tesla is trying to make the brain smarter. BYD is trying to make the whole body smarter, brain included.

None of that means BYD's software is actually good, though. Owners have reported phantom braking, ghost steering that jerks the car toward oncoming lanes, a car suddenly accelerating to 93 km/h in a 60 zone. Free doesn't mean finished. And BYD makes about $1,400 profit per car versus Tesla's $4,583, which tells you the free-software strategy is buying market share at the direct cost of margin. Whether that trade is worth it long-term, I genuinely don't know. Ask again in five years.

Worth saying too, none of this is reaching the US any time soon. BYD's effectively locked out by 247.5% tariffs, and the EU just added another 17% on top of an existing 10%, partly over the same data-privacy suspicion that follows every Chinese-made camera-and-sensor product right now.

Then There's India

Everything above assumes a road that mostly behaves. Lane markings people follow, a predictable set of other vehicles, rules that get broken occasionally rather than as a baseline condition. India doesn't offer any of that, and that's not a knock, it's just structurally what the roads are.

This is where Swaayatt Robotics comes in, and honestly their whole approach reframes the problem instead of trying to solve it head-on. Their system treats the entire driving environment as one continuous corner case. No waiting for a gap to safely appear, because on an Indian road it often just doesn't. Instead the car negotiates, adversarial multi-agent negotiation is the actual term they use, which is a very academic way of describing what basically amounts to nosing into space the way you'd have to as a human driver here.

If a cow or a rickshaw or an oncoming car doesn't yield, the system changes its plan stochastically, meaning probabilistically, not according to some fixed rulebook. Swaayatt calls the resulting behaviour passive-aggressive navigation, and reading that phrase for the first time actually made me laugh, because it's such an accurate description of how you have to drive in Bangalore. Not reckless. Just refusing to freeze up and wait for permission that was never going to come.

Full self-driving in Indian cities, at least any time soon, still feels like a stretch to me even after reading all this. But ADAS, the assisted layer rather than the fully autonomous one, feels far more realistic as the actual next step here. And there's a data problem underneath all of it that I keep circling back to: Waymo's data is fully unsupervised, real self-driving, while Tesla's and the Chinese systems are still supervised, a human present the whole time. So even setting the chaos of Indian roads aside, we genuinely can't compare apples to apples on data quality alone, let alone once you throw in traffic that doesn't follow anyone's rules to begin with.

Which is basically the exact question I couldn't stop thinking about after all this reading. If passive-aggressive negotiation is what makes a human survive Indian traffic, can you actually get a reinforcement learning agent to learn that same behaviour from scratch? Not hardcode it in, but have it genuinely arrive there on its own, the way a human does after enough time behind the wheel.

That's part 3. And that one isn't research anymore, that's me actually trying to build it.

Tesla sells you a car. Waymo sells you a ride. Comparing them directly is like comparing a running shoe to a taxi, they're solving different problems for different people. So let's fix the question first. Assume Waymo starts selling cars, or Tesla launches a cab service. Same category, fair fight. Now how do they actually stack up?

Turns out they don't even run on the same tech.

Tesla runs almost entirely on computer vision. Cameras feed 2D images into a model that figures out what's around the car. The logic makes sense on the surface, humans drive using their eyes, so if you train a model to see the way we do, that should be enough. Waymo disagreed. Their cars carry 23 sensors, cameras plus LiDAR, which shoots laser pulses and builds a 3D map of everything around the car. Where Tesla is estimating depth from a flat image, Waymo knows exactly how far away that pedestrian is.

On paper, Waymo wins on safety. And honestly, vision isn't the only thing I use while driving either.

But here's where it gets interesting. All those sensors generate a ridiculous amount of data, and a self driving car needs to process all of it and make a decision in milliseconds. The overhead is massive. Tesla, running on a single camera input, has a fraction of that processing load. Simpler input, faster decisions, lower cost.

Then there's the data question. Tesla has collected roughly 10 billion miles of autopilot driving. Waymo has less than a quarter of that. Looks like Tesla wins on training data, until you look closer. Every mile Tesla collected had a human sitting in the driver's seat, hands hovering over the wheel. That's supervised data. Waymo's miles were fully driverless, passengers in the back, no one near the wheel. Less data but cleaner data for what actually matters, real self driving.

So Waymo has better data quality, Tesla has more of it. Neither is a clean win.

The last thing that pushed my verdict was geo-fencing. Waymo only operates within pre-mapped areas, constantly updating routes to account for road work and changes. That adds to an already heavy data load. Tesla takes the opposite approach, it basically resets every drive, treating each road like it's seeing it fresh. Less elegant but it means a Tesla can drive anywhere without needing a pre-built map of the place.

My verdict is Tesla. Not because their technology is more impressive, it's not. But because the approach scales better. One input to refine, lower costs, works anywhere. And with Cybercab coming, fully driverless Tesla vehicles aren't that far off. Waymo is more capable today but it's carrying a weight that's hard to justify at scale.

Neither is perfect. But one of them is easier to fix.