The previous post glossed over teaching the agent to pick card rewards. Beginner StS players usually look online for "tier lists" - resources that give a letter grade to every card. Then, they pick the highest-tier card at every card reward and die horrible deaths due to having clunky decks.
Our cloned expert (spirecomm) chose cards exactly this way: it defined a strict ordering over all cards, with a cap so it doesn't end up running 6 copies of Footwork. "Skip" was also in there, which meant that at every card reward, the agent would pick the highest-rated card that's above "skip" for which it hadn't reached the quota. Interestingly, another StS AI project (bottled AI) had a similar approach.
When I saw my agent repeatedly crash against the act 1 boss and die with a poor deck, I naturally assumed the problem was quality of card picks. Even after fixing the card ID mismatch, the agent's decks were bad. I figured the problem was that heuristics were too crude, and decided to employ a different approach. Ironically, it ended up being mathematically equivalent to creating a tier list.
The card scores were derived using logistic regression. A "deck" is simply a list of numbers denoting how many copies it has of each of the 113 possible cards. The "model" multiplies each such number by a learned weight, adds them together with a constant and does sigmoid on the result. The number that comes out, between 0 and 1, is supposed to predict the win probability.
My thinking was that once we have that, we can test each card to see how it increases the deck quality - the chance the deck will win. It was a noble thought, but because of the mathematics of the model, it was exactly equivalent to just picking the card with the highest learned weight:
higher-weight cards contribute more to the output and thus result in increased win probability. We ended up creating a tier list... WITH SCIENCE.
Even in the "increase win chance" phrasing, this approach is still limited. Essentially, it attempts to learn causality from correlation. The longer a run goes on, the more rares you're expected to encounter, and thus the model is likely to over-appreciate rares, since they will be correlated highly with success.
Another problem is slightly more subtle: the model "wins" our training objective if it can tell apart winners from losers. Nothing in how we train it forces it to give the right magnitude to cards. If I were to ask it which of two similar decks is more likely to win, and it did well on that task, that would lend more credence to the emphasis it places on cards.
All of that notwithstanding, I set out to get data. The first data source I used was my own A20 runs. At the time I had 334 runs at 41% win-rate. To augment the data, the deck at every floor was used, meaning a collection of strikes and defends and a damage common got annotated "win" if 50 floors later I would beat the heart. The model did very well at predicting winning decks and not great at ranking cards, so I decided to get more data.
I augmented my data with 53 Silent A20H runs from Jorbs sporting a 60-something winrate. Adding those to mine confused the model. We value cards differently enough, apparently, that it made the data look bimodal and made the "guess if I won" game unfair to the model (and indeed, the game is inherently unfair, since winning is not just a function of the deck but also play quality, relics, potions and draw RNG).
So, I settled on just my runs and thus concluded my first attempt: a glorified tier list that had so few params it couldn't generalize beyond a single player's style. The model needed to take more into account.
The missing ingredient was synergy. Accuracy is terrible in a deck with no shiv cards, but strong in a deck that has three copies of blade dance. The value of cards is derived from the deck composition, as well as the current floor, act boss, elites we might encounter, relics and more. StS is a complex game.
To capture this complexity, we added features for relics (1 or 0 for each of 161) and 22 game parameters: floor, HP (as percentage of max), act, boss ID and more.
The other thing we changed was what the model did. The previous model said how strong a deck (or a card) was. This one would make card picks by getting access to critical information: the opportunity cost. The model had to play "guess what the human picked" when presented with a choice between three1 card rewards and "skip", with access to the context specified above.
The model itself was considerably more elaborate. A "state encoder" turns 308 features, via a series of MLPs, into a 128-wide vector representing the current state. A "card encoder" enriches our old 25 features, again via a couple of MLPs, into a 32-wide vector. Finally, a scoring layer takes 128+32 numbers and derives a single scalar denoting which card should be picked. Overall, 202,000 learnable parameters, a far cry from our previous 114. More importantly, the extra MLP layers ensured it had the expressive power needed to go beyond a clever tier list.
Increasing the number of features and parameters necessitates an increase in data, as we saw when discussing card attributes. Fortunately, the other data source I found online was a 77-million run database containing 34,210 Silent runs and 358,870 card-pick decisions at various ascension (and player skill) levels, in which the win-rate for A20 is 1.5%.
Unleashed on this vast data set, the model converged to a reasonable (53.5%) accuracy, showing it could predict pretty well what a human would pick in a given situation. Encouragingly, it showed it learned something of synergy, evaluating Catalyst much higher when the deck had cards that applied poison than without, for example. The model did what we designed it to do.
Too bad we didn't design it to win at Slay the Spire. True to form, the bigger, more accurate model performed worse:
| Model | Offline accuracy | Beat expert / lost / tied | In-game rank |
|---|---|---|---|
| + card ID embeddings | 55.8% (best) | 12 / 23 / 15 | 4th (worst) |
| mechanical features only | 53.5% | 16 / 16 / 18 | 1st (best) |
| embeddings + propensity weighting | 52.3% | 15 / 22 / 13 | 3rd |
| propensity weighting | 49.5% | 14 / 21 / 15 | 2nd |
This is the same agent with different card-pick engines, playing the same set of 50 seeds. Models that were better at predicting what humans would pick / what decks would win performed worse when guiding card picks. The simplistic "learned tier list" fared best of my attempts, and it still lost to just using the hand-crafted list from bottled AI directly. To quote the Claude instance in charge of designing experiments and interpreting results: "The embeddings learn to predict human picks more accurately, but human picks contain systematic biases. The mechanical features in v1 partially correct for these by being unable to distinguish popular-but-mediocre cards from unpopular-but-strong ones."
There's an interesting tension here: the more robust a model is, the more data it needs. The more data, the lower the quality of play. My hope was to bootstrap with mediocre-quality data and improve with self play. But before further attempts to reinforce ourselves into 100% win-rate against the spire, we would start with a more modest goal: consistently beating a cultist.
1 - sharp-eyed readers may notice Busted Crown, Binary and Question Card all impact the number of choices in card rewards. We padded with -1 for the former cases and silently dropped the fourth pick for the latter case.
