Games › Neon Brawlers

How Neon Brawlers hides a quarter of a second of lag

The netcode for Neon Brawlers, an arena brawler for two to four players on phones, is finished and tested. Netcode is the part that keeps every player's screen in sync over the internet. It's the part I learnt the most from, so this page walks through how it works and how it held up. The game itself is on hold, and so far every test has run on PCs.

Game on hold · netcode finished and tested
A real four-player match online
In one minute
What I did
Designed and wrote all of the netcode: the prediction on the player's device, the game server, the wire format and the test tools.
Built with
Godot 4 and GDScript, UDP through ENet, and a headless Linux server on a VPS in Germany.
What came of it
Your own moves show on the next frame even at 250 ms ping. In 13 of 13 test matches both players ended identical to the server, checked by a hash of the game state. Updates average 338 bytes.
Not done yet
No timing on real phones, no load test with many rooms, and the newer Core Rush mode hasn't been through the lag runs.
0 framesof delay on your own moves. They show up on the very next frame, even with a 250 ms ping.
13 of 13test matches (two players plus two server bots each) ended with both players' games identical to the server's, checked by a hash of the whole game state.
338 bytesper update on average, in a real match played from Pakistan on a server in Germany.
252 msto reconnect by itself after the connection drops.

How it works

The server runs the real game. Each phone sends what its player pressed, and the server sends back what happened. The hard part is hiding how long that round trip takes.

Everything travels over UDP, the internet's quick way of sending packets with no promise that they arrive. I use ENet, a small library on top of UDP that adds delivery checks only where I ask for them. The server is the game itself: the same Godot project, built for Linux to run without a screen, on a small rented server in Germany.

Packet layouts, trade-offs and what broke first are further down, for engineers.

Your phone
1Sends your button presses 60 times a second. Each one is 5 bytes, and every packet repeats the last 8, so losing a packet now and then costs nothing.
4Moves your fighter straight away instead of waiting for the server. This is called client-side prediction. When the server's answer arrives, your phone resets to the server's state and replays every press the server hasn't used yet, which is called server reconciliation. Small differences are smoothed out, and big ones (over 48 px) snap into place.
5Shows the other players 100 ms in the past, so it can slide them smoothly between two real updates from the server. This is entity interpolation. The price is that you aim a little ahead of someone who's running.
The server
2Runs the one real copy of each match, 60 ticks (steps) a second, so it has the final say on every hit. Phones only send stick and button values, and the server works out everything else (server-side validation). The simulation is deterministic, so the same inputs always give the same result.
3Sends back only what changed since the last update your phone confirmed (delta compression), then compresses that with deflate, the method zip files use. Hits and pickups are sent again until your phone confirms them, so a lost update never loses a hit.
+Keeps hits fair without rewinding time. The usual answer, lag compensation, rewinds the other players to where the shooter saw them. Instead, as a player's ping goes up, the players they attack get a slightly bigger hitbox, from 1 px up to 6 px at most, set when the match starts. The laggy player still has to aim ahead of a moving target, and their targets can be hit by a shot that only just missed.

Pick a lag level

I ran full matches through a lag simulator that adds delay, ±15 ms of jitter (random wobble in the delay) and 1% packet loss, four matches at each level with two players in each. Pick one to see how they went.

In separate, earlier runs at up to 250 ms ± 30 ms with 3% loss, full matches also ended with every game in sync.

Measured ping149 to 152 ms
Average update size315 to 413 B
Ticks your phone replays per frame, on average5.2 to 5.6

An update arrives every other frame, and only then does your phone replay the inputs the server hasn't used yet. So each update replays about twice this number.

Hits or pickups shown twice0 of ~2,900
All 4 matches ended in sync: 8 of 8 players matched the serverOther players froze for 0 to 1 frames per match

A real match over the internet

Two PCs in Pakistan played a full match (5,441 ticks) on the live server in Germany. Ping stayed between 143 and 156 ms, apart from one real spike to 296 ms.

  • Other fighters never froze, and wobbled by less than a third of a pixel.
  • Updates averaged 338 bytes. At 30 updates a second that's about 10 KB a second for each player, which matches the 157 kbps the server logged for two players in an earlier test.
  • Both players' games ended in exactly the same state as the server's.

When the connection breaks

Connection drops
Reconnects by itself in 252 ms.
3 s of silence
A bot holds your seat after 1 second. You're back in control 632 ms after the signal returns.
App closed
A bot takes over after 1 second. You're back in the match 538 ms after reopening the app.

How I tested and shipped it

  • A lag simulator that adds delay, jitter and packet loss on demand.
  • Bot players that run without a screen and compare their final game state with the server's.
  • Playtests with two game windows side by side that measure input delay, and can cut the signal or close the app mid-match.
  • One script builds the Linux server and gives it a quick test. Another puts it live, checks it's healthy and rolls back by itself if it isn't.

For engineers

This part is the same system in more detail: the wire format, how updates are built and checked, the trade-offs I made and the bugs that changed the design. Every number here comes from the game's tests, server logs, docs and playtest reports.

Transport and timing

Transport
Everything goes over UDP through ENet. I drive Godot's low-level ENet connection directly, without its high-level multiplayer API. Channel 0 is reliable and ordered (lobby, match start and end). Channel 1 is unreliable and sequenced (inputs, updates and pings). The game uses one UDP port, 7777.
Server
It runs the game's own GDScript code, exported from Godot 4.7.2 as a headless Linux x86-64 build, as a locked-down systemd service on a small Ubuntu VPS. It uses about 90 MB of memory when idle, and one process is capped at 128 rooms.
Ticks
The server steps every room 60 times a second. Phones send input 60 times a second, each packet carrying the newest 8 inputs. Each phone gets its own update 30 times a second (every 2 ticks).
Other players
They are drawn 6 ticks (100 ms) behind the newest update, always between two real updates and never guessed past the newest one.
Prediction
When an update arrives, the phone resets its own copy of the world to it and replays every input the server hasn't applied yet, up to 60. At 150 ms that's about 10 ticks. Updates arrive every other frame, so averaged over every frame it comes to 5.2 to 5.6, the number in the lag panel above. A correction keeps 80% of its offset each tick and snaps if it is over 48 px.
Bandwidth
Each player downloads 6.6 to 15 KB/s of updates depending on the action (about 10 KB/s at the 338 B average), plus about 1.2 KB/s of UDP and ENet headers. It uploads input packets of about 48 B, which comes to about 5.3 KB/s with headers.
Dropouts
A bot takes a seat after 1 s without input. ENet drops a connection that has been silent for 3 to 8 s.

Sequence numbers and acks

  • Every input gets a sequence number. An input packet carries the newest one, plus an ack: the tick of the newest update the phone has decoded.
  • Every update carries a base tick (the update it is a delta against, or -1 for a full snapshot) and an applied seq (the newest of this phone's inputs the server has used). The phone keeps every input after applied seq to replay on top of the update.
  • The server keeps 90 ticks (1.5 s) of past snapshots. A phone's next update is a delta against the newest snapshot it acknowledged, or a full snapshot if that one is gone. Events such as hits, pickups and KOs from after the phone's ack ride along in every update, so they repeat until the phone confirms them.
  • The server applies inputs in order, one per tick. An input that never arrived in any of the 8 packets that carried it is skipped once newer ones are queued. An empty queue repeats the last input. With more than 4 queued, two ticks merge into one with their buttons combined so no press is lost, except that a tick which starts a press is never merged.
  • After a reconnect the phone numbers its inputs from 1 again, so the server reads a sequence more than 120 below what it has seen as a new numbering.

Packet layout

INPUT (message 40)phone to server, channel 1, 60 a second, about 48 B
  1. u8type 40
  2. varuintmatch id
  3. varintack: newest update tick decoded (-1 = none)
  4. varuintnewest input seq
  5. u8n, the input count (8)
  6. n × 5 Beach input: move x, move y, aim x, aim y as s8, then a buttons byte
SNAPSHOT (message 41)server to phone, channel 1, 30 a second
  1. u8type 41
  2. varuintmatch id
  3. varuinttick
  4. varintbase tick (-1 = full)
  5. varuintapplied seq
  6. u16your ping in ms
  7. restpayload, below
SNAPSHOT payloadbuilt by the snapshot codec, 338 B on average in the Germany match
  1. u8compression: 0 none, 1 deflate
  2. varuintsize before compression
  3. deflatedvaruint structure length, the structure (a mode byte per entity, change bitmasks, list counts, ids), then one serialised array of the changed values with the events last

varuint is LEB128 and varint is zigzag then LEB128. Fixed-size numbers are little-endian. Sticks are signed bytes on a 1/127 grid. The buttons byte holds attack (1), dash (2) and skill (4), plus 8, 16 and 32 for picking an upgrade card in Core Rush. Buttons are sent as held, and the simulation finds the presses itself. Control messages on channel 0 are a type byte followed by one serialised dictionary.

How updates are built

A snapshot is the whole world as plain arrays: the world values (tick, random number state, timers, scores), then one array per fighter, barrel, pad, hacker and spike, matched by index, and one per shot and strike, matched by id. Core Rush adds each player's wallet and upgrades, all 59 robots and breakables, and the loot on the floor.

For each entity the encoder writes one mode byte: same, changed or full. A changed entity gets a bitmask with one bit per field, and only the changed values are written. Values are compared bit for bit, so 0.0 and -0.0 count as different. Nothing is quantised. The server builds each delta once and reuses it for every phone on the same baseline.

Payload, 4 bots, whole matchClassicCore Rush
Delta against 2 ticks back, average / largest234 / 1,089 B421 / 1,360 B
Delta against 20 ticks back (a 300 ms phone), average310 B883 B
Full snapshot, average / largest1,075 / 1,364 B2,535 / 3,055 B

From the latest full test run (6,901 and 14,760 ticks). Encoding takes 0.213 ms and decoding 0.084 ms per Classic snapshot on my PC, and the tests fail if a mode goes over its byte budget.

How I checked “same state”

“Same state” means the same SHA-256 hash of the whole world, of which I keep the first 8 bytes as a 64-bit number. The hash covers the serialised snapshot: every field of every fighter, shot, barrel, pad, hacker, strike and spike, plus the tick, the random number state, the next entity id, the round, the timers, the scores and the zone. In Core Rush it also covers every wallet, robot and loot pile.

At the last tick the server sends its hash with the result and keeps resending the final update until every phone has acknowledged it. My test clients rebuild the world from the updates they decoded and compare the two hashes. In the Germany match both PCs ended on 1549435187332146272. The same hash checks the server's recording of every input, which replays offline to the same end state.

This proves the deltas, the compression and lossy UDP together delivered the server's state bit for bit. It doesn't prove the prediction was right along the way, so the playtests count corrections separately: 17 to 26 per match at 150 ms. Before any of this, the handshake compares the protocol version, the game version, the simulation version and a hash of the game's data files, and turns away any mismatch with “Please update the game”.

Trade-offs

The phones do roll back, but only their own prediction, and only against the server's state, so they never have to agree with each other.

What I choseInstead ofWhyWhat it costs
One server runs the match, and each phone predicts its own fighterPeer-to-peer rollback, where every phone runs the match and rewinds when a late input arrivesMobile carriers often block phone-to-phone connections. Rollback also needs every phone to compute exactly the same result, while with one server in charge a phone that drifts just gets corrected. The same code runs the internet server, the Wi-Fi host and the tests.I pay for a server ($5 to $7 a month). Other players are drawn 100 ms plus half a round trip in the past, and the spark of a hit you land waits for the server: a median of 9 frames at 150 ms.
A capped hitbox allowance per attacker, fixed when the match startsLag compensation that rewinds targets to where the attacker saw themRewinding needs a history of positions in the world state. That makes updates bigger, and restoring an update would not restore the history, which breaks prediction and replays. The allowance is part of the match setup, so a match still replays exactly.Laggy players have to lead moving targets, which are drawn about 15 px behind on average and up to 90 px during a dash. Their targets lose a little too: a hit counts up to the allowance outside the 13 px hurt radius (3.25 px at 150 ms). A ping spike after the start is not covered.
Lossless field deltas plus deflateQuantised positions and speeds, the usual way to save bytesPrediction starts from each update, and the end-of-match check needs the exact state.Updates are bigger. A full snapshot only goes out after a join, a reconnect or a long gap, and in Core Rush one with its second of events reaches 4,307 B. ENet splits that into fragments just when the network is having trouble.
ENet driven directly, with my own message formatGodot's high-level multiplayer APIOne process can hold a server and several clients: the Wi-Fi host plays on its own server, the internet server runs many rooms, and a test runs a server and clients together.I write and version every message myself. The protocol is on version 6, and the handshake has to stay readable for old builds so it can tell them to update.
Inputs applied in order from a queue of up to 4Always use the newest input that arrivedNo press is ever dropped, and the queue absorbs jitter (see the first story below).An input can wait a few ticks in the queue, and a long queue merges two ticks into one.
Other players drawn 100 ms behind (interpolation)Extrapolating them forward from their last known movementOther fighters always sit between two real positions. In the lag tests they froze for 0 to 1 frames per match and wobbled by about a quarter of a pixel while walking.Everyone else is in the past, so you aim ahead of a running opponent.

What broke first

Three parts of my first version failed in testing. Here is what the numbers showed and what I changed.

  1. The input queue was too short. My first server kept at most 2 inputs queued before it merged ticks to catch up. In a run at 150 ms ± 20 ms with 2% loss, it ran out of input on 113 ticks and had to repeat the player's last one, merged 95 ticks, and the phone corrected its prediction 132 times. With room for 4, those fell to 21, 0 and 41.
  2. Catching up could redirect a dash. A merged tick keeps the newer stick direction. When the first of the two ticks was a dash press, the 107 px dash went wherever the next tick pointed, and the player's phone, which had predicted the dash they actually aimed, snapped when the server's answer came back. The playtest caught one 97 px snap. Now the server never merges a tick that starts a press.
  3. A reconnect handed your fighter to a bot. In a playtest where I cut the connection 25 s into a match, the client reconnected by itself in 252 ms. Then the server put a bot in the seat 2 s later and only gave it back 24 s after that, while the player's own screen fought 153 corrections (about 20 is normal). The resumed match numbered inputs from 1 again, but inputs from the old match, numbered around 1,500, arrived after the server had reset the seat. From then on every new input looked older than what the server had seen, so it dropped them. I fixed both ends: the client sends nothing while it reconnects, and the server reads a sequence more than 120 below what it has seen as a new numbering. A test now cuts the connection 15 s into a match and checks that all 180 ticks in the next 3 s use the player's inputs with no bot. It fails on the old code.

Not tested yet

I haven't run lots of matches at once. Core Rush is the game's main mode: a four-minute match on a big map where you fight robots for credits, spend them on upgrades and battle the other players for the core in the middle. A busy Core Rush match costs the server about 1.25 ms per tick on my PC, so I estimate it can run about 13 of them per CPU core, but I haven't done a load test to prove it.

I also haven't timed the netcode on a real phone yet. On my PC one prediction reset in Core Rush takes 1.8 to 1.9 ms, and I haven't measured how much slower a phone is.

The lag tests and the Germany match were played in the original Classic mode, before Core Rush existed. Core Rush has played online in the automated tests, but it hasn't been through the 80, 150 and 250 ms runs. I also haven't tested the server against a deliberately modified client.

Fresh Powder uses the same idea of one piece of code in two places: before a sled run counts, the server checks it with the game's own code.