Bashing Bevy To Bait Internet Strangers Into Improving My Code

"Online, the best way to obtain information is not to ask a question, but to state incorrect information and wait for someone to correct you." - Common Internet Wisdom

It is said that one can judge the nerdiness level of a programming language based off the ratio of games-to-game-engines it has.

Considering that - without looking anything off and purely off the top of my head - I am incapable of naming a single published Rust game, but can name at least 4 Rust game engines, (Bevy, Macroquad, ggez, Fyrox) I'd say we're dealing with a particularly severe case Terminal Technowizardry Syndrome.

Like many brave dungeoneers before me, I have ventured off into the damp caves of Rust game development in a quest to right these wrongs. And just like all those aforementioned adventurers, I am failing epically.

However, instead of being reasonable and recognizing my personal shortcomings, I will now proceed to become the protagonist of the adage "The shoddy craftsman blames their tools."

Saint the slugcat, very frustrated over some puzzle pieces

Background

Bevy is currently the most popular Rust game engine out there. In fact, it is so popular that non-Rust programmers might have actually heard of it, which is a feat of its own. As it is designed by Rust programmers, it is sculpted in the image of the substance which composes it - that means, it is extremely opinionated and devoid of any user interface.

Based on the ECS (Entity, Component, Systems) design style, it adopts a philosophy that banishes feeble concepts such as "node trees" or "inheritance" back into the OOP abyss where they belong.

"I don't think ECS is a new concept... like a lot of current trends it's making the old ways of doing things sound new again. It does do some exciting stuff... but it's ultimately just structs and composition, like how we used to do things - but adapted for the modern Unity Node Tree palette." - Evie, my most highly esteemed and very opinionated mentor

Because looking at the people currently raking in boatloads of money and deciding to do things completely differently is cool, Bevy stretches ECS to the extreme, making even UI elements be ECS entities and forcing all code to run inside Systems.

I have completed 2 small projects with Bevy, both open source:

And, I am currently working on my larger scale forever-project:

I will now proceed to bash what I believe are currently Bevy's greatest shortcomings, with the highly devious plan of getting Internet strangers to go "You absolute buffoon! How can you forget to use [HIGHLY ESSENTIAL BEVY FEATURE I NEVER HEARD OF] to save yourself hours of time? You dunce! You fluff-bloated addlepated simpleton!" Because finding those things myself can be... actually, that just brings me to my first main point.

Bevy Contributors Are Way Too Productive

I was only just finishing up updating my game to match Bevy 0.12's new conventions (December 2023) when they released Bevy 0.13 (February 2024). I must now once again comb through my code to knock out all now-obsolete conventions, such as the TextureAtlasSprites liberally scattered throughout my code.

Bevy needs to be hyped up to get people working on it, I fully understand that. But it is not a complete project by any means. Working with it feels like constantly reaching for a tool in your toolbox which you expect to be there, but instead, your hand simply passes through the case and enters a wormhole reaching into a Draft pull request on their GitHub. Tutorials become full of incorrect and outdated information in a matter of months - and as a result, people are not very motivated to actually make them. A significant portion of Bevy-related knowledge is found on YouTube, which is, in my opinion, NOT a good resource when it comes to fetching a quick answer like "how do I schedule this System to only run in this specific GameState".

Bevy programmers probably already know this, but beyond the official docs.rs Bevy documentation, these two resources are highly valuable and tend to be updated... occasionally.

In addition to this, a large portion of Bevy development is fetching third-party crates to complement your Bevy development experience. For instance, I make extensive use of the Bevy Tweening crate in my own projects. I am fully dependent on their maintainers to keep up the pace with new Bevy releases - if a cosmic ray vaporizes every atom of their bodies in an unfortunate astral accident, I will be forced to update their libraries myself to keep my game running with the latest Bevy version. Bevy Tweening maintainers are very active right now, but that might not be the case for the dozen of external Bevy crates a game developer might see themselves using for a larger scale project.

Bevy Types Expand In Mass Until They Blot Out The Sun

Hmm, yes, dear waiter, I believe I'll be getting a ParamSet<(Query<(&Transform, &mut Species, &mut SoulBreath, &mut AxiomEffects, &mut Animator<Transform>, &mut Position, Has<RealityAnchor>)>, Query<&Position>, Query<&Species>, Query<&SoulBreath>, Query<(&Position, &Transform), With<RealityAnchor>>)> for breakfast, por favor. Keep the change.

Bevy Queries are black magic. Given thousands of entities with multiple components, they will somehow fetch every single one that possess only the ones you want in record time, once every frame. This somehow does not turn my 8 GB of RAM computer into a localized micro-Sun, and is quite efficient. I probably should devote some time to understanding at least 10% of the process of how this is done.

However, this is still Rust, and as the commandments go, Thou shall not access a mutable and an immutable reference to the same object simultaneously. This causes issues.

One of the Systems in The Games Foxes Play is tracking the spells cast by creatures and executing their effects. It looks roughly like this:

/*
 We obtain the entity on which the spell is getting cast, 
 what the function of the spell is, 
 and what info we can get from the creature that casted it.
*/
let (entity, function, mut info) = world_map.targeted_axioms.pop().unwrap(); 
/* 
 We grab a bunch of mutable fields and
 get ready to modify the targeted creature's data. 
*/
if let Ok((transform_source, mut species, mut breath, 
	mut effects, mut anim, mut pos, is_player)) = creatures.p0().get_mut(entity.to_owned()) { 
	match function {
	    Function::Teleport { x, y } => {
	    	//SNIP
	    	// The creature has been teleported, 
	    	// edit the "x" and "y" fields on its Position component.
	    	(pos.x, pos.y) = (x, y); 
	    	//SNIP
	    }
    	Function::ApplyEffect { effect } => {
    		//SNIP
    		 // The creature has received a status effect, 
    		 // edit the "status" field on its AxiomEffects component.
		    effects.status.push(effect);
		    //SNIP
		},
	// LOTS, and I mean LOTS, of other spell effects

As you can see, I require accessing all those fields to manage all these possible spell effects. That is the Query<(&Transform, &mut Species, &mut SoulBreath, &mut AxiomEffects, &mut Animator<Transform>, &mut Position, Has<RealityAnchor>)> part of the giga-type I posted at the summit of this chapter. That is creature.p0().

But, let's say that we want to only grab an entity's Position or Species component. No modifications, no mutability, we simply want to know where a creature is currently located in terms of "x" and "y" coordinates, or what species it belongs to.

Function::Collide { with } => { // "with" is the entity being collided with.
    let coll_species = creatures.p2().get(with).unwrap().clone();
    let coll_pos = creatures.p1().get(with).map(|e| (e.x, e.y)).unwrap();

We cannot reuse creatures.p0() as it is currently in use by get_mut. Its ownership is unavailable, and creating a new immutable reference to Position while it is currently mutably borrowed by creatures.p0() (in order to mutate coordinates to fulfill Teleport) is impossible. We must use Bevy's ParamSet type, designed to handle these conflicts.

Even in a situation where I could re-use creatures, I wouldn't necessarily need all those fields but would still need to call every single one, resulting in tons of unused variables:

let stem_block = if let Ok((entity, _queue, _species, _effects,
	_breath, pos, _is_player)) = creatures.get(*segment) {

And that is the source of this extreme bloat. This entire spell handling system thus has the following function fields:

fn dispense_functions(
    mut creatures: ParamSet<(
        Query<(&Transform, &mut Species, &mut SoulBreath, &mut AxiomEffects, 
        	&mut Animator<Transform>, &mut Position, Has<RealityAnchor>)>,
        Query<&Position>,
        Query<&Species>,
        Query<&SoulBreath>,
        Query<(&Position, &Transform), With<RealityAnchor>>,
    )>,
    mut plant: Query<&mut Plant>,
    faction: Query<&Faction>,
    check_wound: Query<Entity, With<Wounded>>,
    mut next_state: ResMut<NextState<TurnState>>,
    mut world_map: ResMut<WorldMap>,
    mut souls: Query<(&mut Animator<Transform>, &Transform, 
    	&mut TextureAtlasSprite, &mut Soul), Without<Position>>,
    ui_center: Res<CenterOfWheel>,
    time: Res<SoulRotationTimer>,
    mut events: EventWriter<LogMessage>,
    mut zoom: ResMut<ZoomInEffect>,
    mut commands: Commands,
    mut current_crea_display: ResMut<CurrentEntityInUI>,
    texture_atlas_handle: Res<SpriteSheetHandle>,

){

Rust's Clippy (an assistant that tries to improve your code) is NOT a fan. Not only is this massively bulky, this forces me to keep my code contained inside this single system. It is very hard to outsource work to other functions, because passing around data is a nightmare. I need to unpack components into relevant fields, call the function, get an output, and then mutate the fields with that output, back inside the giga-system.

For example, if I want to pass around Position to do some calculations and mutate it, I can't just have a function that accepts &mut Position. The actual Bevy type is Mut<'_, Position>. It needs to be turned into fields "x" and "y". Only then can I pass those numbers to a "calculate_pathfinding" function, which returns numbers, and allows me to edit Bevy Components back inside the System.

As a result, that dispense_functions System is currently at 594 lines of code, and will keep growing until it swallows the universe.

I find it quite peculiar how Bevy demos focus very highly on presentation and little on gameplay. Art is obviously a huge part of game development, but reading changelogs show there is a bias towards talking about the latest shader and high performance wgpu computing improvements.

Spawning a transparent 3D glass sphere in Bevy that distorts your vision when you gaze through is now well-supported. But games in the vein of "Dwarf Fortress" or "Zachtronics" with a large quantity of interactions feel much, much harder to do in Bevy due to how much data needs to be passed around.

Bevy Is Not "Rusteic"

"Getting it to compile will be difficult. But when it does compile, it will usually do everything you expected it to." - Rust Cultists

I am proud to say I have never once used a classical runtime debugger with breakpoints in Rust. I never felt like I needed one. The compiler (and its trusty watchman Rust-Analyzer) judge my every move, like they are a team of senior engineers with decades of experience hovering over my shoulders and slowly drooling on the floor, waiting for the moment I make a mistake to bark out at me.

"THAT VARIABLE DOESN'T LIVE LONG ENOUGH!!"

As a former JavaScript plebeian who has only been semi-recently illuminated by the suspiciously pastel pink, white and blue radiance of Rust developers, NOT having to sit in my web console debugger for hours pushing some lovingly crafted [object Object] or undefined is a blessing.

But Bevy, despite being built in Rust, does not offer such guarantees, and I find myself constantly reaching for those dbg! macros I thought I could mostly forget.

  1. There is the lack of warnings. In the previous chapter, I presented some particularly bloated Queries - well, I occasionally find one with some unneeded Without filters. The compiler, of course, is blind to such things.

  2. There is also the grave matter of System Scheduling. You see, without a precise order being given to them, Bevy systems run in complete chaos, and there is no guarantee that they will always continue their execution in the same order. It took me a little while to figure this out, and I now carefully specify game states to ensure Systems are running when they should:

impl Plugin for TurnPlugin {
    fn build(&self, app: &mut App) {
        app.add_systems(Update, calculate_actions
        	.run_if(in_state(TurnState::CalculatingResponse)));
        app.add_systems(Update, execute_turn
        	.run_if(in_state(TurnState::ExecutingTurn)));
        app.add_systems(Update, dispense_functions
        	.run_if(in_state(TurnState::DispensingFunctions)));
        app.add_systems(Update, unpack_animations
        	.run_if(in_state(TurnState::UnpackingAnimation)));
        app.add_systems(Update, fade_effects);
        app.insert_resource(TurnCount{turns: 0});
    }
}

However, unscheduled systems can be very stealthy in how they induce bugs. A single oversight can mean your game works correctly 99% of the time, then suddenly has unexpected behaviour in a very specific situation because a System outsped another when it was least predictable.

  1. Another feature is the extremely bloated standard "Bevy library". Yes, Bevy is divided into multiple sub-crates (like "bevy-ui"), but most developers will simply import the whole thing. This makes compilation times quite severe (my 2016 glorified calculator laptop is incapable of even finishing the compilation process). It is possible to activate "dynamic linking" for a mild speedup:

bevy = { version = "0.13", features = ["dynamic_linking"]}

but, you'll need to REMEMBER to remove this when releasing your game, or you'll need to bundle some obscure module libbevy_dylib alongside your game to ensure it works at all. Fun.

Yes, it's just 8-12 seconds, but when trying to design an UI and reloading every so often to see how it looks, it adds up. My Rust mentor is developing her own Rust game in SDL, and compiling that is more in the realm of a single second.

There is an ocean of features, structs and functions in Bevy you won't ever use if you are merely making a 2D game. It is meant to be a general purpose game engine and a competitor to the likes of Unity, and it shows.

Conclusion

Bevy is an amazing piece of technology that isn't ready yet.

I often end up feeling like I don't actually "own" every part of my game when using Bevy, which is what makes Rust so fun for me. When using Generic JavaScript Framework #28375, I don't feel like a programmer, I feel like a clueless child who has found black boxes of ancient alien technology and a tube of superglue to stitch them together. This is naturally a requirement in some giga-enterprise codebase - you can't understand the whole thing top to bottom - but when it comes to a personal gamedev project, I'd prefer having at least an idea of what is going on behind the scenes.

When reading Bevy changelogs, I constantly think "if only that feature was around when I was developing THAT system in my game!"

Bevy will be a great thing eventually that will challenge the status quo of the gamedev sphere.

But, in my opinion, that time has not yet come.

If you enjoyed this writeup, feel free to contact me! I am currently looking for:

  • Summer internships
  • Full time positions after April 2025
  • People to join my Northsec team - no large proficiency in security required, but the drive to learn about technology is mandatory

Discord: oneirical

Email: julien-robert@videotron.ca

Proper Work-Life Balance In The Packet Factory - First Place in Operating Systems at CS Games

"We have discovered a file crawler in a mysterious PC containing an unfamiliar OS. The resistance believes that it could contain crucial information against the green threat. Therefore, your mission is to interact with it and extract all its secrets. But be careful: who knows what defence mechanisms this device may have?"

Well? How hard could this possibly be?

Spoiler: very much so.

Source Code

Background

On the 15-16-17 weekend of March 2024, I participated in the CS Games competition for the first time, one of the largest undergraduate computer science contests in the noble province of Québec. I expected questionable odours, high levels of introversion and extremely difficult challenges. It appears only the latter conformed to my expectations, for the contenders truly did know how to party with the vast array of costumes, singing and loud electronic music at their disposition. Much to the chagrin of my fragile and sensitive ears.

I "selected" the challenges Operating Systems, Moldable Development and High Performance Computing. Since I am the only person who could not attend the Concordia introductory meeting, my team leaders chose for me - but I must celebrate their skill at making decisions in my place, for I thoroughly enjoyed two out of the three. Sorry, Moldable Development organizers, but I'd rather NOT drown in a deluge of UI micro-buttons and programming syntax hauled straight from the 1970s.

Mildly disappointed at the end of the competition, I sincerely thought I was going to win nothing. My bewildered face is now immortalized on the podium photo of the Operating Systems category, in the highly unexpected ranking of first place.

This post will reveal to you a fragment of the endured suffering.

The Assignment

You may read here the instructions given to me and my teammate, Jaspreet, at the beginning of this 3 hour battle.

If you, dear reader, are NOT a demigod of technology, your expression upon consulting this document may come to closely resemble mine at the time I opened this file for the first time:

A slugcat screaming AAAAAAAAAAa

First course of action: dumb it down a bit so that it may be parsed by my feeble brain where 50% of storage is clogged by fluffiness and pictures of cute cats. Clearly, we are dealing here with a MACHINE that RECEIVES THINGS, does some stuff with them, and outputs a transformed version of those THINGS.

Also known as a "server". Yeah, I know what those are, right? I used to play Minecraft all the time on those! I can choose any programming language for this task...

The answer is obviously Rust. Sure, it has unimportant features like "multithreading", "memory safety" or "static typing", but what I truly care about is how its proponents occasionally possess profile pictures on Github of cute Pokémon wearing adorable bowties and ribbons. And, if someone got a job in technology with this little professionalism, then I would be wise to follow their every word of advice.

The challenge allows for use of any tool, including LLMs and full internet access. Sweet! It's just like working at a real job.

Even ChatGPT won't save you. Good luck. - Mr Gorley, challenge organizer

Things, In And Out

Warning: I am a Biology undergraduate student self-teaching myself technology to escape the gulf of academia. I make mistakes. If anything in this post is wrong, I genuinely want to be corrected so that I may learn. Contact me at my email: julien-robert@videotron.ca, or message me on Discord - my username is "oneirical".

So, according to the intructions, the poor file crawler is currently throwing its delivery crates ("packets") in the abyss known as "port 7331" where no one cares about it or gives it any attention. Let us fix that.

fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:7331")?;

    for stream in listener.incoming() {
        match stream {
            Ok(stream) => {
                thread::spawn(move || {
                    handle_connection(stream).unwrap();
                });
            }
            Err(e) => {
                eprintln!("Failed to accept connection; err = {:?}", e);
            }
        }
    }

    Ok(())
}

Now, I have no idea what a TCP or UDP stream is. I don't really remember what they mean, probably something like Tragically Cute Puppies or Unrealistically Desirable Pets. What I did, however, find out, is that when it comes to these kinds of server challenges, you'll usually want to make your connection follow the TCP type. Put a metaphorical pin in that, this will be relevant later.

Each "stream" is like a conveyor belt transporting packets into my little letterbox. All such streams should be correct (marked as "Ok"), in which case a "thread" will spawn to start taking care of the deliveries, like a loyal servant volunteer assigned to unpacking the delivery crates. This thread-worker needs to be able to actually touch the crates to do anything, so the "move" keyword transmits ownership of the data to them! Let's see what is happening inside the other end of the stream-conveyor-belt.

Packets Of All Shapes And Sizes

enum Packet {
    UPLD { crawler_id: u16, file_path: String },
    MODE { mode: Mode },
    SEQN { seq_num: u16 },
    DATA { upload_id: u16, seq_num: u16, data: Vec<u8> },
}

#[derive(Clone, Copy)]
enum Mode {
    Block,
    Compressed,
}

Not all packets serve the same purpose! We want that juicy intel contained in those DATA packets, yes, but such precious payload is to be handled with care. To prepare ourselves, we have:

  • UPLD packets announcing the arrival of a fresh delivery. They contain the identification number of the crawler (conveyor belt) from where the delivery is coming, and the file path in the source OS ("country of origin") from which the data is getting extracted.
  • MODE packets only announce whether or not the incoming data is compressed and needs to have some water splashed onto it to revert to its original form, or if it's good as it currently is. Note the two Mode enums dictating this.
  • SEQN packets contain the "sequence number" of the packets. I am not as confident about these, but I believe they are counting what is the current number of DATA packets received, and if, for example, the next one will be the fifth or tenth one.

Let's Unpack This

fn handle_connection(mut stream: TcpStream) -> std::io::Result<()> {
    let mut buffer = [0; 508]; // Maximum packet size
    let mut mode: Mode;

    loop {
        let n = match stream.read(&mut buffer) {
            Ok(n) if n == 0 => return Ok(()), // Connection closed
            Ok(n) => n,
            Err(e) => {
                eprintln!("Failed to read from socket; err = {:?}", e);
                return Err(e);
            }
        };

        // Parse the packet
        let packet = parse_packet(&buffer[..n]);
        match packet {
            Ok(packet) => {
                // Handle the packet
                mode = match packet {
                    Packet::MODE { mode } => {
                        mode
                    },
                    _ => Mode::Block,
                };
                handle_packet(&mut stream, packet, mode)?;
            }
            Err(e) => {
                eprintln!("Failed to parse packet; err = {:?}", e);
                // Send error response
                send_error_response(&mut stream, "Failed to parse packet")?;
            }
        }
    }
}

Have a glance inside our "volunteer"'s crate-unpacking room. Because there is no such thing as sensible work-life balance in MY factory, their task continues eternally with the "loop" keyword (same thing as "while true") until they are finally done and no further packets are coming out of the stream. The "n" variable is monitoring how many bytes are currently on the stream - how many crates remain on the conveyor belt, if you will. n = 0 returns Ok(()) and terminates our poor worker's employment.

"508" is the maximum size of a Packet, measured in bytes.

First, packets are parsed. This is because the shipping company sending these crates to us is laughably incompetent, and is basically just throwing a bunch of hexadecimal numbers on the conveyor belts wrapped in flimsy plastic packaging. The only way to identify where the crates begin and end, and what they actually are (UPLD, MODE, etc.) is through their header, a 10-byte sequence containing:

  • 2 bytes: Magic number 0xC505
  • 2 bytes: Total packet size, including header
  • 2 bytes: Crawler identifier
  • 4 bytes: Command name in ASCII

Basically, it's some cheap paper stickers slapped on top of the incoming transmission. We can do better. The "parse_packet" function makes all those messy bytes get tucked into a pristine box - the Packet enum shown earlier (UPLD, MODE, etc.) - all safe and sound for processing :3

I will demonstrate its inner workings in the following chapter. For now, the neatly processed Packet arrives in "match packet", where a quick check verifies if our esteemed volunteer did their job correctly. Yes, sometimes, the incoming data contains spiky or dangerous things poking out - most importantly "bit flips" causing erronous tagging. This is a part of the assignment! Should that be detected by "parse_packet", the "match packet" will throw the suspicious delivery into the incinerator of "send_error_response", where it shall be destroyed:

fn send_error_response(socket: &mut TcpStream, message: &str) -> std::io::Result<()> {
    let response = format!("IAMERR\x00{:02X}{}", message.len() + 4, message);
    socket.write_all(response.as_bytes())?;
    Ok(())
}

Hopefully, the next packet will fare better than its predecessor.

Let us return our attention to jobs well done - instances of "Ok(packet)". First, we check if we are currently dealing with a MODE packet - if yes, change the current active mode to match it - this will enable or disable the miraculous Decompressor 9000 located inside "handle_packet". We shall return to that later.

Nice And Tidy Boxes

fn parse_packet(data: &[u8]) -> Result<Packet, &'static str> {
    // Check if the data is long enough to contain a header
    if data.len() < 10 {
        return Err("Packet too short");
    }

    // Parse the header
    let magic_number = u16::from_be_bytes([data[0], data[1]]);
    let total_packet_size = u16::from_be_bytes([data[2], data[3]]);
    let crawler_id = u16::from_be_bytes([data[4], data[5]]);
    let command_name = &data[6..10];

    // Validate the magic number
    
    if magic_number != 0xC505 {
        return Err("Invalid magic number");
    }

    // Validate that the received data is of the expected length
    if data.len() != total_packet_size as usize {
        return Err("Packet size mismatch");
    }

    // Determine the packet type based on the command name
    match command_name {
        b"UPLD" => {
            // Parse the UPLD packet
            let file_path = String::from_utf8_lossy(&data[10..]).to_string();
            Ok(Packet::UPLD { crawler_id, file_path })
        },
        b"MODE" => {
            // Parse the MODE packet
            let new_mode = match String::from_utf8_lossy(&data[10..]).as_ref() {
                "block" => Mode::Block,
                "compressed" => Mode::Compressed,
                _ => Mode::Block,
            };
            Ok(Packet::MODE { mode: new_mode })
        },
        b"SEQN" => {
            // Parse the SEQN packet
            if data.len() < 12 {
                return Err("Packet too short for SEQN");
            }
            let seq_num = u16::from_be_bytes([data[10], data[11]]);
            Ok(Packet::SEQN { seq_num })
        },
        b"DATA" => {
            // Parse the DATA packet
            if data.len() < 14 {
                return Err("Packet too short for DATA");
            }
            let upload_id = u16::from_be_bytes([data[10], data[11]]);
            let seq_num = u16::from_be_bytes([data[12], data[13]]);
            let data = data[14..].to_vec();
            Ok(Packet::DATA { upload_id, seq_num, data })
        },
        _ => Err("Unknown command"),
    }
}

Ah, the glorious packaging facility. Where the unruly and chaotic come to be crushed and rearranged into flawless order and conformity.

First, we ensure that the header - that flimy paper tag - actually exists. We check that at least 10 bytes are present, then begin slotting them into their respective categories. "from_be_bytes" is swapping out those pesky hexadecimal tags (C5 05 00 1B 00 01 55 50, ewww!) into a glorious human-readable number.

We then check that 1. the magic number is present, and that 2. the packet size written in the header actually corresponds to its real weight. If you've played Papers Please, it's just like weighing people at the customs on a scale to make sure they aren't hiding any contraband.

Should everything appear in order, we then package the contents! The "b" before each match statement is parsing the Packet type as a "byte string" - because the aforementioned incompetent packaging company, of course, just HAD to give us numbers, and not human-readable letters!

Cracking open what is inside the Packet allows us to fill up each field of the Packet enum (for example, DATA contains upload_id, seq_num and data). This finalized version is what is shipped to "handle_packet", covered in the next chapter.

Payment Received

fn handle_packet(socket: &mut TcpStream, packet: Packet, mode: Mode) -> std::io::Result<()> {
    match packet {
        Packet::UPLD { crawler_id, file_path } => {
            // Acknowledge the upload
            let response = format!("UPLOADING\x00{:02X}", crawler_id);
            socket.write_all(response.as_bytes())?;
            Ok(())
        }
        Packet::MODE { mode } => {
            // Acknowledge the MODE packet
            let response = format!("METADATAMODE");
            socket.write_all(response.as_bytes())?;
            Ok(())
        }
        Packet::SEQN { seq_num } => {
            // Acknowledge the SEQN packet
            let response = format!("METADATASEQN");
            socket.write_all(response.as_bytes())?;
            Ok(())
        }
        Packet::DATA { upload_id, seq_num, mut data } => {
            // Handle DATA packet
            match mode {
                Mode::Block => {
                    // Handle block mode data
                },
                Mode::Compressed => {
                    // Handle compressed mode data
                    data = decompress_rle(&data);
                },
            }
            handle_data_packet(socket, upload_id, seq_num, data)
        }
    }
}

Receiving an UPLD, MODE or SEQN packet isn't very complicated. We pretty much only need to scream out that we got it, and it stops there. In the case of MODE packets, we already used its contents to swap the current active mode to Block or Compressed. DATA Packets are much more interesting...

First, they may or may not face the aforementioned Decompressor 9000. Observe, and be awed:

fn decompress_rle(data: &[u8]) -> Vec<u8> {
    let mut result = Vec::new();
    let mut index = 0;

    while index < data.len() {
        let value = data[index];
        let count = data[index + 1];
        result.resize(result.len() + count as usize, value);
        index += 2;
    }

    result
}

"RLE is a form of lossless data compression in which runs (a run is sequence of consecutive values that are the same) of data are stored as a single count and data value.

I'm sorry, dear Mr Gorley - the competition organizer - but my hands were already quite full. I just looked up RLE decompression online, and translated the code into Rust. Forgive me, Linus Torvalds.

You copied code without understanding it, as a result your code is trash. AGAIN.

Following this process, DATA packets are sent to the final step of their journey:

fn handle_data_packet(socket: &mut TcpStream, upload_id: u16, seq_num: u16, data: Vec<u8>) -> std::io::Result<()> {
    
    let clean_data = remove_redundant_bits(&data);
    
    // Example: Write data to a file named after the upload_id
    let file_path = format!("upload_{}.bin", upload_id);
    let mut file = OpenOptions::new()
        .write(true)
        .append(true)
        .create(true)
        .open(file_path)?;

    let mut writer = BufWriter::new(file);
    writer.write_all(&clean_data)?;

    // If the data is empty, this is the end of the file transfer
    let file_path = format!("upload_{}.bin", upload_id);
    if data.is_empty() {
        // Send a response indicating the upload is complete
        let response = format!("UPLOAD END\x00{}", file_path);
        socket.write_all(response.as_bytes())?;
    }

    Ok(())
}

That "clean_data" step is a real brain liquefier. I'll come back to it later, I still do not fully understand it.

In order to capture that sweet intel, we need a place to store it. That is a file - possibly the most "Operating Systems" part of this challenge... As far as I am aware, most of the work done so far is more in the realm of Networking. And not the "talking to ambitious entrepreneurs at a fancy cocktail" type, I had my fair share of that one too at the CS Games ending banquet.

This file has full permissions, and has the entire contents of the data field of the DATA packet dumped into it. The end of this task is announced with the glorious victory chant of "UPLOAD END". Such celebrations must not last long - the next packet awaiting processing is already on the way...

The Part Where It Sucked

Alright, so:

fn remove_redundant_bits(data: &[u8]) -> Vec<u8> {
    let mut result = Vec::new();
    let mut buffer = 0u16; // Temporary buffer to hold 2 bytes of data
    let mut bit_count = 0; // Keeps track of the number of bits processed

    for &byte in data {
        buffer = (buffer << 8) | (byte as u16); // Shift the buffer and add the new byte
        bit_count += 8; // Increment the bit count

        // Process 2 bytes of data at a time
        while bit_count >= 9 {
            // Extract 9 bits from the buffer
            let value = (buffer >> (bit_count - 9)) & 0x1FF; // Mask to keep only the last 9 bits
            result.push(value as u8); // Add the value to the result
            bit_count -= 9; // Update the bit count
        }
    }

    // Handle any remaining bits in the buffer
    if bit_count > 0 {
        let value = buffer & ((1 << bit_count) - 1); // Mask to keep only the last bit_count bits
        result.push(value as u8); // Add the value to the result
    }

    result
}

This is my attempt at making SOMETHING that would somewhat resemble the challenge requirement about Hamming codes. That part could have been written in Swahili and I would have probably understood it better.

I mean, just look at this:

We can see that G0000, P0001, and P0010 don't hold. We therefore know that bit 30011 was flipped, because 0001 | 0010 = 0011. ORing/adding the positions of the parity bits that have an error gives the position of the errorneous bit.

I beg your pardon?

In the challenge description, there is this tiny line of text: "Redundant bits should be removed from the received data before saving to disk". I wondered if that had anything to do with "Hamming error correction". This resulted in hacked-together code in the final 30 minutes of the challenge, knit together from StackOverflow and LLM outputs. I understand what it does - it repeatedly combines the last byte in each buffer with a new byte, preparing for the next 9-bit extraction. This effectively makes little bundles of 9 bits and will drop any overflowing or redundant bits not part of the bundling process.

I'm pretty sure this has nothing to do with the Hamming-thingimagibob. I would obviously spend time and effort learning it in an actual job, but it wasn't something I could fit within the 3 hour time delay.

EDIT: Worlds Ender's Solution

After posting this blog on Reddit, a highly dedicated user going by the name of WorldsEnder - possibly about 2 or 3 planes of existence above my own level of technowizardry - decided to implement parity checking as a fun challenge. They mentioned how a minimum viable product took 30 minutes, and the polished version, 3 whole hours! If someone on this level of expertise took that long, actually implementing this in the competition would have been a distant dream. I am glad I wasted little time on it.

You may find their code here.

If you, Internet wanderer, is currently feeling as confused by this theory as I previously was, please check out 3Blue1Brown's video on the topic! There is also a follow-up on the implementation itself.

Final Scoring

I was told my final score was 48/58, which really surprised me. I was imagining every other university team flawlessly implementing that Hamming error correction feature and annihilating us.

There was a Python script provided with the challenge, meant to "test" our server. However, it contained more bugs than my previous internship in entomological research, so I disregarded it entirely. Jaspreet, my teammate, had the brilliant idea to test the connection by simply plugging the IP address in the web browser, which gave us a big confidence boost when we found out the packet input was indeed successful. However, this was not real testing. I pretty much shipped the entire code for review in pure YOLO action.

It appears other teams DID manage to fix and use the script. They listened to the outgoing connection with Wireshark, and found out the protocol in use was UDP, and not TCP like we chose. I got crushed when I first heard this, as this would basically mean that our entire program would not work at all. At this point, I became convinced of my utmost failure.

However, the judge, in his magnanimity, only deducted a few points for this. After all, as far as I'm aware, changing the protocol wouldn't be so hard - just a matter of swapping out a function here and there.

The Université de Montréal team, who came in second place, visited me after the announcement of victory, saying that they found it quite weird that they did not get first place. They mentioned how they managed to implement every feature, including Hamming correction, and that they tested their code extensively using a modified version of the Python script. In fact, I remember them spending a LOT of time debugging it and showing newfound mistakes to the competition organizer.

Personally, I preferred to use that time to polish the error handling of my server. Testing isn't as mandatory in Rust - if the compiler accepts it, it has a good chance of working on the first try!

Anyhow, I invite the UdeM team, who may be reading this, to open source their code like I did. I am intrigued to see how you two managed to implement that mysterious Hamming error correction feature.

Special Thanks

  • Jaspreet, my teammate for this challenge. You may not know Rust, but I am glad you agreed to let me use it for our code. Your ideas and discussions really propelled the project forwards, and your idea near the end to use a web browser to test the connection removed so much stress off my shoulders!

"It's actually making me want to learn rust right now."

  • Evie, my most highly esteemed mentor and grand Rust arch-sorceress. Working on RGBFIX - a Game Boy ROM fixing tool - under your guidance before attempting this challenge helped tremendously, from proper error handling, to read/write operations, and, of course, parsing hexadecimal headers. Beyond your knowledge, you are also an amazing friend, and talking to you is one of the things I look the most forward to daily.

"I’m really happy for you. I can’t believe you were convinced you failed and then got first place :3"

If you enjoyed this writeup, feel free to contact me! I am currently looking for:

  • Summer internships
  • Full time positions after April 2025
  • People to join my Northsec team - no large proficiency in security required, but the drive to learn about technology is mandatory

Discord: oneirical

Email: julien-robert@videotron.ca