How Neon Brawlers hides a quarter of a second of lag
The netcode for Neon Brawlers, an arena brawler for two to four players on phones, is finished and tested. Netcode is the part that keeps every player's screen in sync over the internet. It's the part I learnt the most from, so this page walks through how it works and how it held up. The game itself is on hold, and so far every test has run on PCs.
Game on hold · netcode finished and tested
A real four-player match online- What I did
- Designed and wrote all of the netcode: the prediction on the player's device, the game server, the wire format and the test tools.
- Built with
- Godot 4 and GDScript, UDP through ENet, and a headless Linux server on a VPS in Germany.
- What came of it
- Your own moves show on the next frame even at 250 ms ping. In 13 of 13 test matches both players ended identical to the server, checked by a hash of the game state. Updates average 338 bytes.
- Not done yet
- No timing on real phones, no load test with many rooms, and the newer Core Rush mode hasn't been through the lag runs.
How it works
The server runs the real game. Each phone sends what its player pressed, and the server sends back what happened. The hard part is hiding how long that round trip takes.
Everything travels over UDP, the internet's quick way of sending packets with no promise that they arrive. I use ENet, a small library on top of UDP that adds delivery checks only where I ask for them. The server is the game itself: the same Godot project, built for Linux to run without a screen, on a small rented server in Germany.
Packet layouts, trade-offs and what broke first are further down, for engineers.
Pick a lag level
I ran full matches through a lag simulator that adds delay, ±15 ms of jitter (random wobble in the delay) and 1% packet loss, four matches at each level with two players in each. Pick one to see how they went.
In separate, earlier runs at up to 250 ms ± 30 ms with 3% loss, full matches also ended with every game in sync.
An update arrives every other frame, and only then does your phone replay the inputs the server hasn't used yet. So each update replays about twice this number.
A real match over the internet
Two PCs in Pakistan played a full match (5,441 ticks) on the live server in Germany. Ping stayed between 143 and 156 ms, apart from one real spike to 296 ms.
- Other fighters never froze, and wobbled by less than a third of a pixel.
- Updates averaged 338 bytes. At 30 updates a second that's about 10 KB a second for each player, which matches the 157 kbps the server logged for two players in an earlier test.
- Both players' games ended in exactly the same state as the server's.
When the connection breaks
- Connection drops
- Reconnects by itself in 252 ms.
- 3 s of silence
- A bot holds your seat after 1 second. You're back in control 632 ms after the signal returns.
- App closed
- A bot takes over after 1 second. You're back in the match 538 ms after reopening the app.
How I tested and shipped it
- A lag simulator that adds delay, jitter and packet loss on demand.
- Bot players that run without a screen and compare their final game state with the server's.
- Playtests with two game windows side by side that measure input delay, and can cut the signal or close the app mid-match.
- One script builds the Linux server and gives it a quick test. Another puts it live, checks it's healthy and rolls back by itself if it isn't.
For engineers
This part is the same system in more detail: the wire format, how updates are built and checked, the trade-offs I made and the bugs that changed the design. Every number here comes from the game's tests, server logs, docs and playtest reports.
Transport and timing
- Transport
- Everything goes over UDP through ENet. I drive Godot's low-level ENet connection directly, without its high-level multiplayer API. Channel 0 is reliable and ordered (lobby, match start and end). Channel 1 is unreliable and sequenced (inputs, updates and pings). The game uses one UDP port, 7777.
- Server
- It runs the game's own GDScript code, exported from Godot 4.7.2 as a headless Linux x86-64 build, as a locked-down systemd service on a small Ubuntu VPS. It uses about 90 MB of memory when idle, and one process is capped at 128 rooms.
- Ticks
- The server steps every room 60 times a second. Phones send input 60 times a second, each packet carrying the newest 8 inputs. Each phone gets its own update 30 times a second (every 2 ticks).
- Other players
- They are drawn 6 ticks (100 ms) behind the newest update, always between two real updates and never guessed past the newest one.
- Prediction
- When an update arrives, the phone resets its own copy of the world to it and replays every input the server hasn't applied yet, up to 60. At 150 ms that's about 10 ticks. Updates arrive every other frame, so averaged over every frame it comes to 5.2 to 5.6, the number in the lag panel above. A correction keeps 80% of its offset each tick and snaps if it is over 48 px.
- Bandwidth
- Each player downloads 6.6 to 15 KB/s of updates depending on the action (about 10 KB/s at the 338 B average), plus about 1.2 KB/s of UDP and ENet headers. It uploads input packets of about 48 B, which comes to about 5.3 KB/s with headers.
- Dropouts
- A bot takes a seat after 1 s without input. ENet drops a connection that has been silent for 3 to 8 s.
Sequence numbers and acks
- Every input gets a sequence number. An input packet carries the newest one, plus an
ack: the tick of the newest update the phone has decoded. - Every update carries a
base tick(the update it is a delta against, or -1 for a full snapshot) and anapplied seq(the newest of this phone's inputs the server has used). The phone keeps every input afterapplied seqto replay on top of the update. - The server keeps 90 ticks (1.5 s) of past snapshots. A phone's next update is a delta against the newest snapshot it acknowledged, or a full snapshot if that one is gone. Events such as hits, pickups and KOs from after the phone's ack ride along in every update, so they repeat until the phone confirms them.
- The server applies inputs in order, one per tick. An input that never arrived in any of the 8 packets that carried it is skipped once newer ones are queued. An empty queue repeats the last input. With more than 4 queued, two ticks merge into one with their buttons combined so no press is lost, except that a tick which starts a press is never merged.
- After a reconnect the phone numbers its inputs from 1 again, so the server reads a sequence more than 120 below what it has seen as a new numbering.
Packet layout
- u8type 40
- varuintmatch id
- varintack: newest update tick decoded (-1 = none)
- varuintnewest input seq
- u8n, the input count (8)
- n × 5 Beach input: move x, move y, aim x, aim y as s8, then a buttons byte
- u8type 41
- varuintmatch id
- varuinttick
- varintbase tick (-1 = full)
- varuintapplied seq
- u16your ping in ms
- restpayload, below
- u8compression: 0 none, 1 deflate
- varuintsize before compression
- deflatedvaruint structure length, the structure (a mode byte per entity, change bitmasks, list counts, ids), then one serialised array of the changed values with the events last
varuint is LEB128 and varint is zigzag then LEB128. Fixed-size numbers are little-endian. Sticks are signed bytes on a 1/127 grid. The buttons byte holds attack (1), dash (2) and skill (4), plus 8, 16 and 32 for picking an upgrade card in Core Rush. Buttons are sent as held, and the simulation finds the presses itself. Control messages on channel 0 are a type byte followed by one serialised dictionary.
How updates are built
A snapshot is the whole world as plain arrays: the world values (tick, random number state, timers, scores), then one array per fighter, barrel, pad, hacker and spike, matched by index, and one per shot and strike, matched by id. Core Rush adds each player's wallet and upgrades, all 59 robots and breakables, and the loot on the floor.
For each entity the encoder writes one mode byte: same, changed or full. A changed entity gets a bitmask with one bit per field, and only the changed values are written. Values are compared bit for bit, so 0.0 and -0.0 count as different. Nothing is quantised. The server builds each delta once and reuses it for every phone on the same baseline.
| Payload, 4 bots, whole match | Classic | Core Rush |
|---|---|---|
| Delta against 2 ticks back, average / largest | 234 / 1,089 B | 421 / 1,360 B |
| Delta against 20 ticks back (a 300 ms phone), average | 310 B | 883 B |
| Full snapshot, average / largest | 1,075 / 1,364 B | 2,535 / 3,055 B |
From the latest full test run (6,901 and 14,760 ticks). Encoding takes 0.213 ms and decoding 0.084 ms per Classic snapshot on my PC, and the tests fail if a mode goes over its byte budget.
How I checked “same state”
“Same state” means the same SHA-256 hash of the whole world, of which I keep the first 8 bytes as a 64-bit number. The hash covers the serialised snapshot: every field of every fighter, shot, barrel, pad, hacker, strike and spike, plus the tick, the random number state, the next entity id, the round, the timers, the scores and the zone. In Core Rush it also covers every wallet, robot and loot pile.
At the last tick the server sends its hash with the result and keeps resending the final update until every phone has acknowledged it. My test clients rebuild the world from the updates they decoded and compare the two hashes. In the Germany match both PCs ended on 1549435187332146272. The same hash checks the server's recording of every input, which replays offline to the same end state.
This proves the deltas, the compression and lossy UDP together delivered the server's state bit for bit. It doesn't prove the prediction was right along the way, so the playtests count corrections separately: 17 to 26 per match at 150 ms. Before any of this, the handshake compares the protocol version, the game version, the simulation version and a hash of the game's data files, and turns away any mismatch with “Please update the game”.
Trade-offs
The phones do roll back, but only their own prediction, and only against the server's state, so they never have to agree with each other.
| What I chose | Instead of | Why | What it costs |
|---|---|---|---|
| One server runs the match, and each phone predicts its own fighter | Peer-to-peer rollback, where every phone runs the match and rewinds when a late input arrives | Mobile carriers often block phone-to-phone connections. Rollback also needs every phone to compute exactly the same result, while with one server in charge a phone that drifts just gets corrected. The same code runs the internet server, the Wi-Fi host and the tests. | I pay for a server ($5 to $7 a month). Other players are drawn 100 ms plus half a round trip in the past, and the spark of a hit you land waits for the server: a median of 9 frames at 150 ms. |
| A capped hitbox allowance per attacker, fixed when the match starts | Lag compensation that rewinds targets to where the attacker saw them | Rewinding needs a history of positions in the world state. That makes updates bigger, and restoring an update would not restore the history, which breaks prediction and replays. The allowance is part of the match setup, so a match still replays exactly. | Laggy players have to lead moving targets, which are drawn about 15 px behind on average and up to 90 px during a dash. Their targets lose a little too: a hit counts up to the allowance outside the 13 px hurt radius (3.25 px at 150 ms). A ping spike after the start is not covered. |
| Lossless field deltas plus deflate | Quantised positions and speeds, the usual way to save bytes | Prediction starts from each update, and the end-of-match check needs the exact state. | Updates are bigger. A full snapshot only goes out after a join, a reconnect or a long gap, and in Core Rush one with its second of events reaches 4,307 B. ENet splits that into fragments just when the network is having trouble. |
| ENet driven directly, with my own message format | Godot's high-level multiplayer API | One process can hold a server and several clients: the Wi-Fi host plays on its own server, the internet server runs many rooms, and a test runs a server and clients together. | I write and version every message myself. The protocol is on version 6, and the handshake has to stay readable for old builds so it can tell them to update. |
| Inputs applied in order from a queue of up to 4 | Always use the newest input that arrived | No press is ever dropped, and the queue absorbs jitter (see the first story below). | An input can wait a few ticks in the queue, and a long queue merges two ticks into one. |
| Other players drawn 100 ms behind (interpolation) | Extrapolating them forward from their last known movement | Other fighters always sit between two real positions. In the lag tests they froze for 0 to 1 frames per match and wobbled by about a quarter of a pixel while walking. | Everyone else is in the past, so you aim ahead of a running opponent. |
What broke first
Three parts of my first version failed in testing. Here is what the numbers showed and what I changed.
- The input queue was too short. My first server kept at most 2 inputs queued before it merged ticks to catch up. In a run at 150 ms ± 20 ms with 2% loss, it ran out of input on 113 ticks and had to repeat the player's last one, merged 95 ticks, and the phone corrected its prediction 132 times. With room for 4, those fell to 21, 0 and 41.
- Catching up could redirect a dash. A merged tick keeps the newer stick direction. When the first of the two ticks was a dash press, the 107 px dash went wherever the next tick pointed, and the player's phone, which had predicted the dash they actually aimed, snapped when the server's answer came back. The playtest caught one 97 px snap. Now the server never merges a tick that starts a press.
- A reconnect handed your fighter to a bot. In a playtest where I cut the connection 25 s into a match, the client reconnected by itself in 252 ms. Then the server put a bot in the seat 2 s later and only gave it back 24 s after that, while the player's own screen fought 153 corrections (about 20 is normal). The resumed match numbered inputs from 1 again, but inputs from the old match, numbered around 1,500, arrived after the server had reset the seat. From then on every new input looked older than what the server had seen, so it dropped them. I fixed both ends: the client sends nothing while it reconnects, and the server reads a sequence more than 120 below what it has seen as a new numbering. A test now cuts the connection 15 s into a match and checks that all 180 ticks in the next 3 s use the player's inputs with no bot. It fails on the old code.
Not tested yet
I haven't run lots of matches at once. Core Rush is the game's main mode: a four-minute match on a big map where you fight robots for credits, spend them on upgrades and battle the other players for the core in the middle. A busy Core Rush match costs the server about 1.25 ms per tick on my PC, so I estimate it can run about 13 of them per CPU core, but I haven't done a load test to prove it.
I also haven't timed the netcode on a real phone yet. On my PC one prediction reset in Core Rush takes 1.8 to 1.9 ms, and I haven't measured how much slower a phone is.
The lag tests and the Germany match were played in the original Classic mode, before Core Rush existed. Core Rush has played online in the automated tests, but it hasn't been through the 80, 150 and 250 ms runs. I also haven't tested the server against a deliberately modified client.
Fresh Powder uses the same idea of one piece of code in two places: before a sled run counts, the server checks it with the game's own code.