What is a "Shiny" Pokémon?

Simply put, a shiny pokémon is a rare variant with identical statistics and abilities, but a modified color palette. Their rarity makes them a fun collectible, but other than that they serve no purpose. Below is a comparison of the pokémon Piplup - both shiny and normal.

Why automate the process?

Finding a shiny Pokémon is extremely rare. I was hunting for a shiny Piplup in Pokémon Brilliant Diamond, a game for the Nintendo Switch. In this game, every time you encounter a Pokémon, there is a 1/4096 (0.024%) chance of finding a shiny variant! But, to make matters worse, Piplup is a "starter Pokémon". This means it is impossible to find them in the wild in the game - there is only one opportunity, at the very start. Getting to the point where the Piplup is visible on screen takes an excruciatingly long 2 minutes! This means that according to the shiny odds, it would take around 8192 minutes, or 136.53 hours. A repetititive task with a clearly defined end target? It was just begging to be automated.

Showing the game to my computer

Since the game would be running on an unmodified Nintendo Switch, there was no direct hook into the game's code. This meant the only way to decide if the piplup on screen was a shiny or not was by analyzing the output image itself. For convenience, I already play my Switch on my desk by using a video capture card like the Elgato 4K60 Pro. This card in my PC provides an HDMI input port, and I can use software such as OBS Studio to show the Switch's output on my computer monitor. This means that any program running on my computer with the ability to capture an image from a monitor would be able to see the game being played.

Recognizing a Piplup

This part of the project I knew was possible, but at first, I wasn't sure of the best way to go about doing it. The first challenge was capturing and formatting the training dataset. Thankfully, I came across something called roboflow. Roboflow allowed me to generate my training dataset exactly as I needed - I could upload a video to their website, which contained the animated piplups in various angles, poses, etc. Then, frame by frame in the browser, I could tag the piplup in frame as either regular or shiny. Roboflow's simple user interface made this process a breeze, even though I had to tag over a hundred training images.

Now that the dataset was created and categorized, the next step was to select an object detection algorithm. While many exist, I settled on a convolutional neural network (CNN) called YOLO, or "You Only Look Once". Without getting into the specifics, YOLO excels in placing bounding boxes around objects in an image that it is trained to detect. More importantly, it is optimized for speed, and has been shown to operate at up to 45 FPS on relatively moderate hardware. The dataset was downloaded from roboflow, and training the YOLO network (YOLO V3 specifically) began. Every thousand iterations or so, an output file that contained the weights for the trained neural network was generated.

YOLO is implemented using OpenCV's deep neural network class, and the weights generated from training were loaded in through a file. Using Pillow to perform the screengrab which was fed to OpenCV, my computer was successfully able to determine whether or not a shiny Piplup was on screen.

Controlling the Nintendo Switch automatically

This portion of the project represented the biggest uncertainty. I knew it would be possible to have my computer detect a shiny or regular Piplup on screen - but it was moot if the computer couldn't control the Switch accordingly! This is where the Arduino platform came in. I knew from experience that a certain product, the Arduino Pro Micro, that could come in handy. The Arduino Micro runs off a ATmega32U4 chip - critically for this project, that allows the device to be initialized as a Human Interface Device. Online research showed that it was possible to have the Switch recognize a Pro Micro as a wired controller, so the project could continue!

While I now knew it was possible, getting everything to work was still quite complicated. Thankfully, a lot of the controller-side legwork had been done for me already. A user on GitHub had published the Switch Fightstick project, which used an ATmega32U4-powered Arduino, and was an arcade-style fight stick compatible with the Nintendo Switch.

I edited the source code from that project - instead of performing pre-programmed macros when certain electrical contacts were made, I had an Arduino (let's call this A1) perform button presses based on what characters were fed to it via UART serial. Then, I hooked up a second Arduino (A2) to my PC, and connected the 2 together over their TX/RX pins. On my computer, I would enter a character into A2 over USB. This would relay the message to A1 over the UART communication lines. Finally, A1 would send a button press to the Switch over USB. It worked! As a note - it is certainly possible to replace A2 with a more standard USB-UART adapter; I used another Arduino Pro Micro because I had them on hand, and it was simple.

Once the Arduinos were set up, it moved all of the programming into Python, where I could prototype code more quickly. In the end, the code was simple. First, the initialization routine that runs when A1 is plugged in:

1. Send the appropriate buttons to register the device as a new wired controller
2. Exit out of the controller setup screen and wait on the Switch's home screen. Begin the loop of waiting for characters over Serial.

Once in the loop, A1 would do nothing unless a character was sent to it. The code below shows the Python script that would run, getting through all of the in-game loading times and cutscenes until the Piplup was on screen. If no shiny was detected, the code automatically closes the game, and restarts the process.

As you can see, the majority of the code is simply timed sleeps. The code also prints out constant messages (containing snippets of the cutscene text), so at a glance the user can ensure the system is still synced up. Line 140 is where the detection starts, resulting in either the system restarting the game if a regular piplup is found. If a shiny is found, or no piplup is on screen, user input is needed for safety.

5,717 Resets

That's how long it took. 5,717 resets, or 11,434 minutes, or 190.5 hours. But when I got home from work, I had a shiny Piplup waiting for me on screen. I was overjoyed - though it had taken multiple weeks (I let it run for ~12 hours a day), the project had been a success. Everything had worked, from the Arduino controller to the YOLO detection.

Future Improvements

Now that I know the concept is valid, there are a whole number of things I'd like to do to make it more usable. These include:

• Generalizing the Data: Instead of training the algorithm on just shiny piplups, I'd like to train it on a number of shiny pokémon. Although each looks unique, the stars / shine effect is constant. Training on different types of pokémon will make this a generalized shiny hunter for more applications.
• Optical Character Recognition (OCR): As shown in the code, the system operates mostly on timed sleeps at the moment. While this did work, there were a few instances where it failed. Sometimes the Switch would take an extra second or two to load the game, or would get hung up for a small amount of time. Using OCR to track where in the dialog the game is would make the whole system more resilient to those kinds of errors.
• Improved User Experience: Right now, the code is simply run through the terminal, and must be executed while the Switch is in the correct part of the menu. Though not critical, it would be nice to build a small UI for the project, and perhaps use image recognition to allow the user to start from any screen on the Switch.

Check out the whole project on GitHub! (Coming Soon™!)