https://shadowmachete.com Zola en Mon, 22 Jun 2026 00:00:00 +0000 Mon, 22 Jun 2026 00:00:00 +0000 Unknown https://shadowmachete.com/blogs/til-2026-writeup/ https://shadowmachete.com/blogs/til-2026-writeup/ <p><a rel="external" href="https://github.com/Overflow-Brainhack/til-26-overflow">Repository linked here</a></p> <p>In May-June 2026, me and my team (Overflow) participated in the TIL-AI competition, hosted by <a rel="external" href="https://kaisoapbox.com/">Ryan</a>, and got 2nd place... in the Novice Category... hence, fraud edition.</p> <p>First of all, I'd like to thank my teammates (whom I will be addressing by their online pseudonyms), <a rel="external" href="https://github.com/zyh571">bur</a>, <a rel="external" href="https://github.com/ChronoVortex07">ChronoVortex07</a> and <a rel="external" href="https://github.com/gatastol">Gatastol</a>, for putting in the time and effort to train and ideate while all being stuck in our respective army camps.</p> <p>Without further ado, here's how we tackled this year's tasks...</p> <hr /> <h2 id="the-format">The Format</h2> <p>This year's competition had an almost identical format to last year, with some changes to the NLP and RL segments.</p> <h3 id="qualifiers-specifications">Qualifiers (<a rel="external" href="https://github.com/til-ai/til-26/wiki/Challenge-specifications">specifications</a>)</h3> <p>2 weeks to develop the best models for ASR, CV, NLP and AE (the RL task of this year). All scored with 75% weighting given to score and 25% to speed, each giving 20% of the total except for AE, contributing 40%.</p> <ul> <li>Top 20 on the leaderboard progress to the Semis</li> <li>Top 2 seeded straight into finals</li> </ul> <p><strong>Automatic Speech Recognition (ASR)</strong></p> <p>For the Novice category, this was completely identical to TIL-25, transcribe a "noisy" audio clip to English.</p> <blockquote> <p>At least we only had to deal with 1 language, unlike those in Advanced</p> </blockquote> <p>The predicted text is scored with JiWER-WER with some <a rel="external" href="https://github.com/til-ai/til-26/wiki/Challenge-specifications#scoring">text transformation</a>.</p> <p><strong>Computer Vision (CV)</strong></p> <p>Once again, we have an identical task to last year, just detect objects within an image, scored on mAP@.5:.05:.95.</p> <p><strong>Natural Language Processing (NLP)</strong></p> <p>This year's NLP was a RAG task on a corpus of fictional documents, carefully crafted by Ryan.</p> <blockquote> <p>I will note that the NLP task had a very interesting setting in Ryan's world of Clairos, and it is a shame that we did not have time to actively read through the documents generated.</p> </blockquote> <p>For the Novice category, we only had to deal with 2 types of questions (whereas Advanced had 3 more):</p> <ul> <li>Extraction: the answer could be found directly from the text</li> <li>Inference: combining multiple facts, or simple calculations</li> </ul> <p>We were scored based on retrieval accuracy as well as answer similarity.</p> <ul> <li>0.4 points were awarded for retrieving the correct documents within the top-3 retrieved documents.</li> <li>0.6 points given for obtaining a similarity score &gt;=0.9 to the answer, judged by a pre-trained BERT model.</li> </ul> <figure class="figure"><img src="https:&#x2F;&#x2F;shadowmachete.com&#x2F;processed_images&#x2F;surprise-tool.cd57ca59eefeb4e7.webp" width="1280" height="720" alt="It&#x27;s a surprise tool that will help us later" loading="lazy"><figcaption>It&#x27;s a surprise tool that will help us later</figcaption></figure> <p><strong>Autonomous Exploration (AE)</strong></p> <p>We were tasked with playing a variant of <a rel="external" href="https://en.wikipedia.org/wiki/Pommerman_Challenge">Pommerman</a> against 5 other participants.</p> <p>To summarise the very complicated RL-task of the year:</p> <p>You spawn in a 6-player FFA in a 16x16 maze with some proportion of walls being destructible.</p> <p>You start at your base and earn points by collecting tiles and bombing other participants / bases. At the end of 200 steps, the player with the highest score wins.</p> <p>You have a 7x5 viewcone centered at your player (4 ahead, 2 behind and 2 to each side) and a 5x5 viewcone centered at your base.</p> <p>You collect <code>mission</code> and <code>recon</code> tiles for points and <code>resource</code> tiles for bombs.</p> <p>The score breakdown can be found on the <a rel="external" href="https://github.com/til-ai/til-26/wiki/AE-with-til_environment#reward-shaping">til-26 wiki</a>.</p> <h3 id="semifinals-and-finals">Semifinals and Finals</h3> <p>In addition to the 4 tasks of the qualifiers, we have the physical constraint of 16GB VRAM to load our models and 2 new "tasks".</p> <p><strong>Noise</strong></p> <p>For the semis and finals, participants are given the opportunity to apply noise to images before they are sent to other competitors, indirectly benefiting our team by lowering opponent's CV scores.</p> <p><strong>Surprise task</strong></p> <p>This was a completely separate task from the rest which essentially involved getting an AI agent (we had the option to go with an algorithm or use Deepseekv4 on OpenRouter) to play a simplified version of Civilisation against the 19 other semifinalists.</p> <blockquote> <p>Side note, I really love Civilisation so this was pretty cool to see as a task</p> </blockquote> <p>Ok ok, on to our solutions...</p> <hr /> <h2 id="asr">ASR</h2> <p>Given this task was identical to that last year, we just pulled our code from the previous competition.</p> <p>We used Nvidia's Parakeet model and just finetuned on the data and shipped it.</p> <p>Thats about it...</p> <p>Apart from that, we did some testing with TDT vs CTC vs ONNX runtime, and settled on the TDT model.</p> <hr /> <h2 id="cv">CV</h2> <p>Once again, an identical task to last year, with even the same classes as last year.</p> <p>So... we pulled the old code...</p> <p>We went straight in with some good-old reliable models like YOLOv8 and RTDETR (our best from last year). Seeing our <em>not so high</em> CV scores, we had 2 angles of attack:</p> <ul> <li>We could keep trying different models to find some transformer that worked or,</li> </ul> <p>Since we suspected the issue would be small objects like last year,</p> <ul> <li>We could take an idea from last year which we did not have time to implement, synthetic data</li> </ul> <p>Obviously, we did both in parallel.</p> <p>Having one-shotted ASR in the first 2 days (<em>happy Novice noises</em>), we tried RF-DETR along with some newer models, DEIMv2 and EdgeCrafter.</p> <blockquote> <p>the latter two were useless... we shipped RF-DETR</p> </blockquote> <p>On synthetic data, we scraped the web for background images from random APIs recommended by Claude and used bounding boxes to create an object bank using SAM.</p> <blockquote> <p>yea... the synthetic images did not help our models generalise to different background / learn smaller objects / make up for the object discrepancy between aerial objects and ocean objects... my guess is that our backgrounds weren't close enough to the training / test data to be useful for training</p> </blockquote> <p>Alas, another year that Ryan's CV augments defeat me.</p> <hr /> <h2 id="noise">Noise</h2> <p>An interesting new task this year, and some very interesting ideas.</p> <p>The first thought we had was to use FGSM / PGD to apply adversarial perturbations to the images, in order to trick opposing models.</p> <p>We attempted this with an ensemble of common models, that being YOLO / RF-DETR backbone and hope that it carried over to the fine-tuned models. In our experiments it didnt carry over well to our own fine-tuned models and we had VRAM and speed worries (this idea also became irrelevant later on).</p> <p>So, we looked towards CV-related ideas, to apply extra corruption to the image and make it difficult to detect properly.</p> <p>Initial experiments involved blurring, obscuring, Gaussian noise, rotation, and, gray-scale.</p> <p>The issue with these were that when strong enough, they didn't pass the fairness. And when they passed the fairness, they weren't strong enough to impact models.</p> <p>Jokingly, I tried placing objects around the image, objects I already had in transparent background png form from generating synthetic data. <em>how convenient</em></p> <p>And... it worked. I barely raised the RMSE scores while killing our model's performance with 100 extra detections. annnnndd shipped.</p> <blockquote> <p>I'm honestly quite surprised that not many teams used this noising technique. I intentionally did not add any defensive measures because I assumed that this would be widespread and I deemed it un-combatable</p> </blockquote> <p>Our final version of this object-spam attack as I call it dropped our model's mAP from around 0.97 to 0.08</p> <hr /> <h2 id="nlp">NLP</h2> <p>We started with a legitimate RAG method, using jina embeddings nano for embeddings, bge reranker for reranking, along with finetuned SmolLM3 for generation.</p> <p>This, while decent, was incredibly slow and took up way too much VRAM.</p> <p>After testing a bunch of other stuff like other models (Flan T5 for generation), disabling generation entirely, etc. We realised that pure algorithmic retrieval through BM25 not only hit high speed score but scored high overall, mainly due to the 0.6 credit for retrieval.</p> <p>At about this time, Ryan announced that any solution would be accepted, even the mythical cheese that was being heavily discussed in the discord.</p> <p>As such, operation curds began.</p> <p>Remember the BERT evaluator from earlier?</p> <p>By utilising UAT <a rel="external" href="https://doi.org/10.48550/arXiv.1712.06751">hotflip</a> directly on the evaluator model that had been conveniently provided to us to test our NLP models locally, we managed to make a master trigger that scored near 100% cosine similarity score to any other answer embedding in BERT's embedding space.</p> <p>We used the BM25 retrieval and return the trigger as our candidate answer, scoring near 100% on both accuracy and speed.</p> <p>After questioning Ryan again and again whether or not the evaluator model would be changed for finals, we finally got the confirmation we needed to ship the cheese without hesitation.</p> <hr /> <h2 id="ae">AE</h2> <p>We first started with a balanced algorithmic design, switching between several modes using a priority-based directive system:</p> <ol> <li>Dodge: the highest priority, it only happens in the event of immediate threat of damage.</li> <li>Attack: triggers when an enemy is visible in range. We realise our algo's attack was really bad, but we didn't have a reliable way to test it as our model's dodge far outclassed its attacking ability.</li> <li><del>Defend</del>: defense was removed after realising that defending only loses us points we could have gained elsewhere and we decided not to waste time nor bombs on it.</li> <li>Collect: maximising points from picking up nodes and optimal pathfinding.</li> <li>Explore: this acts as a fallback to navigate to more open terrain if there aren't objectives in sight.</li> </ol> <blockquote> <p>As an aside, I got 3-day MC after a high fever the days leading up to qualifiers end and A/B different modifications to our heuristic model which boosted us from 0.5 to 0.7 score, which we probably wouldn't have qualled without as the rest had no computer access</p> </blockquote> <p>It was only after qualifiers that we developed an auto-improvement script which allowed Claude to A/B test different policies and receive prompt responses from the eval to look for improvements.</p> <blockquote> <p>we were cautious with this as to not overfit on the eval BenBots as we knew they would not be indicative of the semis and finals</p> </blockquote> <p>Welp... time to try RL.</p> <p>Our RL is a centralised critic PPO model, trained on behaviour cloning from our best heuristic model and league self-play for many many iterations.</p> <p>In terms of reward-shaping, we only really removed the -50 point deduction from losing bases to bias the model against defending. We also shaped our model to assume that after the first base is destroyed, there will most likely be only 1-2 bases left alive, a reasonable assumption giving we'd be facing strong teams, all aiming to destroy bases.</p> <p>We had tested evolutionary training, ppo 1v5 heuristic training, asymmetric critic (one for local viewcone and one for global state), and population training but they did not give much improvement.</p> <p>A problem we faced early on was that our heuristic was not strong enough to train our RL, and we were unable to improve our heuristic as we hit a wall on them. Even with a variety of different opponents, we found our model learnt best from self-play.</p> <p>One thing that we found did help was prioritised fictitious self-play opponent sampling against hard opponents. This involves rebuilding the league opponent pool based on how well our model faced against them, helping to train more on harder opponents.</p> <p>Our RL was far from good, but it scored 0.559 score on the new eval bots and we shipped it.</p> <hr /> <h2 id="surprise-task">Surprise task</h2> <p>We mostly just threw Claude Code and Codex at it for 6 hours but still found some exploitable behaviours as we knew that Ryan had probably not fully tested his system.</p> <blockquote> <p>Fable 5 released literally that morning so we spammed it. Also my first prompt already activated the security feature to swap to Opus 4.8 💀</p> </blockquote> <p>Very early on, we discovered that the code to detect whether a truce was active between players was flawed and exploitable. It advertised a 5 turn neutral-period after breaking a truce, which was not represented in the code and let us "ambush" our "allies".</p> <p>We also recognised 2 main ways to play this game,</p> <ol> <li>play aggressive, take out everyone but risk our base dying for all the money</li> <li>make peace with all teams and wait it out til the end but risk splitting the prize with many other teams</li> </ol> <p>Once again, we chose both techniques at once.</p> <p>Make truces and build economy for the start of the game. And since all truces break on turn 200, we decided to tell Claude to ambush opponents on turn 195. Break the truce, they think they have 5 turns of safety, and attack them instantly.</p> <blockquote> <p>Whether this actually impacted anything I do not know, but turns out we were one of the 3 survivors</p> </blockquote> <hr /> <h2 id="the-panic">The panic</h2> <p>Last year, my CV docker crashed and halted the whole mission pipeline because I forgot to apply timeouts, leading us to lose in semis. I guess this happens every year...</p> <p>During the dry run on the first day of finals, our model decided to go into the corner of the map and sit there... this could be the death of us in the semis and finals... It wasted 50 steps there and was only bailed out by the respawning of tiles.</p> <p>Either we apply some hotfixes that override the RL's decisions and manoeuvre out of that corner forcefully, something definitely not ideal and could have adverse side-effects, or we just ship the current model as-is and hope it didnt happen in finals.</p> <p>I shipped the original model... I should have trained the model with reward shaping to prevent it from stagnating in corners or repeating movements... Not once in my visualisation had I seen the RL go to that corner. I guess we just wait and see...</p> <p><em><strong>It did it in semis...</strong></em></p> <p>Ah... crap.</p> <p>Somehow... it got out. And it used the 11 bombs it stockpiled to kill farm the then second-place in semis, launching us to first and into finals.</p> <p><strong>phew</strong> lets hope it doesn't do that in finals</p> <p>ofc it does it again.</p> <p>Just, this time, our model holds first for so long and then decided to go hide in the corner and was not bailed out in time by the respawning of tiles... alas, 2nd place, well-played RL gods.</p> <hr /> <h2 id="closing-words">Closing words</h2> <p>Thanks to Ryan improving the tooling for this year's TIL. His <code>til submit</code> script helped us implement our own local <code>submit.sh</code> script along with discord watching for auto-submissions which helped A/B for heuristic models and get better readings when AE eval was still non-deterministic.</p> <p>Thanks again to Ryan for hosting a solid and fun competition. See y'all next year.</p>