TC101: Intro to Theorycrafting

On more than a few occasions I’ve been asked some variation of the question, “How do I get started in theorycrafting?” Which is a tough question to answer, since there’s a variety of ways to get started depending on what you’re interested in and what talents or tools you have at your disposal. Someone proficient with spreadsheets might try to write one to model a rotation, for example. But one’s first foray into theorycrafting could be as simple as doing some “napkin math” to compare two talents.

For example, my own entry into the world of theorycrafting happened when I took somebody’s prot paladin spreadsheet and translated it into MATLAB code. I wanted to analyze variation with several different input variables (i.e. the oft-misused term “scaling”), which is something that spreadsheets are traditionally poor at doing. Translating the formulas in the spreadsheet into MATLAB code provided two advantages: full text code is generally easier to debug than spreadsheet formulas are, and MATLAB is designed to work with flexible arrays of data in ways that spreadsheets simply aren’t.

In the process of performing that translation I learned a lot about the way different spells interact, how some of the different game systems worked, and so on. In a lot of cases I corrected formulas that I discovered were in error, often because I explicitly tested the formula in-game to see if it was right. It was a slow but steady process of learning, testing, and refinement. And once it was done, the learning continued as I started to expand the sorts of questions that I wanted to answer with my code.

But when somebody asks, “How do I get started,” they’re not usually thinking about a specific problem. They’re thinking about making the transition from being a person that reads guides and follows the advice given to someone who discovers and creates that advice.

Sometimes, the person asking only has a vague understanding of what it means to “theorycraft.” Most players already know that theorycrafting produces numbers that can be used to evaluate performance and ask questions, of course. But what most players don’t know is how those numbers are produced from beginning to end.

That’s what I hope to clear up with this series of blog posts. And the first step is to make it clear exactly what the term “theorycrafting” means.

What IS Theorycrafting?

At its root, theorycrafting is a process called mathematical modeling. We’re trying to take some sort of system – in this case game mechanics – and describe it mathematically so that we can generate predictive results. As with most mathematical modeling, it’s also somewhat directed. In other words, we’re not just doing this for the hell of it; we’re trying to answer specific questions, so our model is built around having the versatility to be able to answer those questions.

Generally, that doesn’t happen by spontaneously creating a very complicated model that covers everything. It happens by creating a very simple model and then slowly refining it to include all of the complications necessary to make it accurate. In other words, you don’t start with a BMW. You start with a wheel, and maybe an axle. You put those together and start adding things, one by one, until you do have a BMW.

I had a very interesting conversation with Steve Chick a few weeks ago, during which he provided a great flow chart that more or less describes this process:

Flow chart that describes the process of learning. Source unknown.

This is the basic process of problem solving (and a core part of the scientific method), and it applies equally well to theorycrafting and model creation. Steve and I have differing opinions about the best advice for how the “learn more things somehow” part should be accomplished, but we agree completely on the process.

We have a question we’re trying to answer, like “How much DPS does Judgment provide,” and we’re attempting to break it down into smaller pieces that we can answer. The goal is to then put those pieces back together and come up with the answer to our original question.

Which means that in a broader sense, theorycrafting is also an exercise in problem solving. Even though they may not quite realize it, the player asking how to get started with theorycrafting is really asking how to obtain the tools necessary to start solving problems on their own. What I hope to provide with this series of blog posts is a little guidance on exactly how to develop and use those tools.

Theorycrafting 101

As an example, let’s say that is our question: “How much DPS does Judgment provide?” Let’s break that down as if we were complete newcomers to theorycrafting.

First, do we know what “DPS” stands for, and how to calculate it? You, as a seasoned WoW player, laugh at that question. But in reality, it’s not something I’d expect a random WoW player to know. Even if they knew it meant “damage per second,” knowing how to properly calculate it wouldn’t be guaranteed. You’d be surprised how many college-age students struggle with simple ratio metrics like velocity (“meters per second”), current (“charge per second” or “mass per second” depending on whether we’re talking about electricity or fluid flow), or efficiency measures (“miles per gallon”).

Let’s say we know the general concept – that we know we want to add up the damage we do in some period of time and then divide the total amount of damage by the length of time. How long a period do we use? Ten seconds? A minute? Ten minutes? An hour? The answer to that depends on not just accuracy, but the details of our rotation. If our rotation is a fixed, repeatable cycle (like CS-J-X-CS-X-J-CS-X-X) then we could plan on using one full cycle to give us the same precision as an infinite amount of time. But if it isn’t, we might have to decide what the cutoff is. Maybe we want to simulate 300 seconds of continuous combat, or maybe we only care about a 20-second window of a fight.

Once we decide on the time, we need to figure out how to calculate the total amount of damage done by Judgment in that time. Intuition tells us that will be the average damage of each cast times the average number of casts in our time window. Again, some of that depends on rotation (number of casts). But we also need to know how we calculate the damage done per cast. So we’ve broken the problem down into two smaller problems:

  1. How much damage does Judgment do per cast
  2. What’s our rotation?
    • Determines number of casts and time interval (or equivalently, cast rate)

And we’ve come up with an equation:

$$ DPS = ( {\rm Damage Per Cast} \times {\rm Number of Casts} ) / {\rm Time }$$

And note that we haven’t gotten any farther than deciding how to calculate a relatively simple metric like DPS!

So now we try and answer each of those questions, and break them down further if we can’t. Let’s take #1 since it’s simpler – how much damage does Judgment do per cast? If we’re a complete newcomer to theorycrafting, we may not know any more than “we press a button and it does some damage.” So we need to figure out how to quantify that.

This is the part where Steve and I disagree, by the way. He suggests that you should test it and figure it out yourself. In other words, go into the lab (i.e. in game) and set up an “experiment” to measure that damage and figure out how the game is calculating it. And there are definitely advantages to this approach. Learning is often significantly aided by firsthand experience, which is why laboratory exercises are so common in the sciences. This is, in fact, the approach we’ll use for our example.

However, my first instinct is to look things up and see if someone’s done the hard work for me before. I know that I may learn something from the process of designing and carrying out an experiment, especially if I screw something up and have to re-do it (nothing aids learning like painful and/or time-consuming mistakes!). But I also know that it’ll probably be a lot faster to spend a few minutes googling. That may also be a generous way of saying “I’m lazy.”

So let’s say we want to set up this experiment. What are we going to test? Or, put another way, what factors change the damage of Judgment? First, we might already know (or guess) that it changes when our attack power changes. We might also wonder if it varies with spellpower. Maybe we’re not sure if it depends on weapon damage, or if it has a base damage value. We do know from experience that it does more damage when we get a critical strike, and that there are a few effects that boost its damage (Glyph of Double Jeopardy, Avenging Wrath, Holy Avenger). So we need to test all of those things, and in some cases how they interact (for example, is Avenging Wrath’s 20% boost multiplicative or additive with Holy Avenger?) before we can put them together.

In other words, we’ve just created a bunch of smaller questions to answer:

  1. How much damage does Judgment do per cast
    1. Does it vary with attack power (and if so, how)?
    2. Does it vary with spell power (and if so, how)?
    3. Does it have a base damage value?
    4. Does it depend on weapon damage (and if so, how)?
    5. How much more damage does it do on a critical strike?
    6. How often do we get a critical strike?
    7. How does Avenging Wrath affect the damage?
    8. How does the Glyph of Double Jeopardy affect the damage?
    9. How does Holy Avenger affect the damage?
    10. How do G, H, and I interact?

I’ve cheated a bit here and added (F) because I know there’s a hidden crit suppression against higher-level targets, but a new theorycrafter might not be aware of that fact. Similarly, they might skip test J because they’ve assumed (knowingly or not) that everything is multiplicative (it might be… or it might not be – Blizzard can be inconsistent on that from one effect to another). Both of those are errors that might not show up until a lot later (and with a lot more testing), which is one of the reasons I advocate doing a little reading first.

I’ve separated these out because each of these is going to require its own experiment (or at least, its own calculations). So we’ve now got a long list of things to test, each of which is a small component of how Judgment’s damage is calculated. Pretty much all of these are as low-level as we can get, so there’s no point in breaking them down further. They’re each things we can either answer directly (i.e. “A critical strike does 2x the damage”) or measure through experiment and analysis.

In the next blog post, I’ll talk in more detail about how we go about designing each of these experiments and putting the pieces together. For now though, I want to go back to the more abstract concept of putting the results together. Let’s say we perform some of these experiments and determine that (note that these are completely made up):

  1. Judgment does 1000 base damage.
  2. Weapon damage has no effect.
  3. Every point of attack power adds 2 damage.
  4. Every point of spell power adds 1 damage.
  5. Crits do 2x damage, and
  6. Crits occur with a probability equal our character sheet crit chance.

So we have several small pieces we can put together. We know that ignoring crits, a Judgment will do on average about $1000 + 2\times AP + 1\times SP$ damage. To apply the crits, we note that when we don’t crit (a probability of $1-C$, where $C$ is our crit chance) we do 100% damage, and when we do crit (probability $C$) we do 200% damage. That gives us a factor of

$1.00 \times (1-C)+2.00 \times C = 1 – C + 2C = 1 + C$

So our average damage per Judgment is then

$${\rm Judgment damage} = (1000 + 2\times AP + 1\times SP) \times (1+C).$$

And there we have it: our first model for Judgment damage. It’s not a complete model, obviously – we’d need to continue to refine it to account for all of the other effects that affect Judgment’s damage. And then we’d repeat this entire process for the rotation tree, and combine those results to create a model for DPS.

But that’s the essence of theorycrafting. Start with a simple model, and eventually add more detail and complexity until the model is as accurate as you need it to be. We’ll talk a little more about determining accuracy and tolerances in the next two installments.

Simulationcraft

Simulationcraft is, as you might expect, just a really big, complex numerical model. And it’s built up in exactly the same way that we built our model for Judgment damage. There are literally thousands of small moving parts within SimC taking care of each of the details that one might care about.

For example, there’s an entire system of functions to accurately calculate your hit, miss, dodge, parry, block, and crit chances against a target based on your combat ratings, the target’s base avoidance, block, and crit suppression values, and the level difference between the two of you. Another function takes all of that information and constructs attack tables and performs the rolls that determine whether you hit or miss, whether your attack is a critical strike or not, and whether the attack is blocked (provided it can be blocked at all!). All of that is done with pinpoint accuracy because we have a good understanding of how combat rolls work thanks to years of theorycrafting.

Likewise, in the paladin class module, there are special functions that handle things like Hand of Light damage, seal procs, Grand Crusader, and so on. Lots of little moving pieces that each handle one small detail, each one improving the accuracy of the model bit by bit.

Which brings us to another statement I see fairly frequently: “I’d like to contribute to Simulationcraft, but I don’t know C++.” It’s true that Simulationcraft is written in C++, and while the intent is that you don’t really need to know it to maintain a class module, in my experience our class modules simply aren’t user-friendly enough for that to be realistic.

However, not all contributions to Simulationcraft require coding knowledge. The great part about SimC is that it outputs a report that doesn’t require any programming experience to read and interpret. There are plenty of things that someone can do just by tweaking an action priority list and looking at how the output changes.

One way to think about it is that Simulationcraft has several layers. At the top, there’s the “theorycrafting layer,” where you only need the basic knowledge of how to manipulate action priority lists and read the reports the simulation generates. I call it the theorycrafting layer because this is where you try out new ideas for optimizing a character or compare simulation results to in-game testing to check for errors.

In the middle, there’s the mechanics layer. This is where the class module developers (i.e. coders) come in, because it’s the layer where the mechanics that we discover in-game get coded into the simulation. But even here, there’s room for non-coders, because we don’t always have class developers that are experts on each class. We have quite a few talented people writing code, but none of them may be experts on your class or spec. But if someone who is an expert on that spec can explain how the mechanics work to a developer, we can support that spec anyway.

At the bottom is the core layer, which is all of the under-the-hood subsystems that run the simulation. Things like how events are scheduled and executed and how (and what) data is stored. This layer really does require C++ knowledge, but we have several really dedicated devs that already take care of most of this stuff. While I’m sure they would love help, realistically the greater need is in the top two layers, since that’s the bulk of the work when we’re staring down a new expansion.

The point of all of this is that we don’t need a host of C++ gurus to help make SimC better for everyone. We need more people that can properly test and describe the mechanics to a coder, so that the coder can implement those features.

In other words, we need theorycrafters more than we need code monkeys.

Coming Soon

One goal of this series of blog posts is to give prospective theorycrafters a better idea of what they’re getting into. Another is to help them put together the basic toolbox they’ll need to actually start solving problems. Both of those aims are well served by showing actual examples of theorycrafting, like we did with Judgment in today’s post. Not coincidentally, this is exactly the same approach that most introductory textbooks take.

As you may have guessed, theorycrafting employs many of the basic techniques that any scientist would learn before going into a laboratory. So the next two blog posts will be focused on developing and understanding common experimental methods.

In the second part of this series, we’ll talk more about how to properly design in-game experiments to test and verify mechanics. Then, in the third part, we’ll focus on methods for comparing those in-game results to Simulationcraft results to check for consistency.

This entry was posted in Theck's Pounding Headaches, Theorycrafting, Uncategorized and tagged , , , , , , , , , , . Bookmark the permalink.

8 Responses to TC101: Intro to Theorycrafting

  1. Pingback: Theorycraft 101: The Statistics of WoW Spells | It's Dangerous to Go Alone

  2. Ben Shuck says:

    Interesting read, I’ve always done a bit of messing around with talents and asking healers how my health looked to them. Not so much with SoO, but quite a bit in ToT.
    I’m hoping to get into WoD to try out some things I’ve been thinking of and get a better understanding, but my chances seem pretty slim.

    My first post, but I’ve been reading for a while.

  3. Benevolent Benefactor says:

    “In other words, we need theorycrafters more than we need code monkeys.”

    Your next blog post could be just this one line repeated 100 times.

  4. Pingback: TC101: Experimental Design | Sacred Duty

  5. Pingback: TC101: Testing Simulationcraft | Sacred Duty

  6. Pingback: TC101: How Stats Are Calculated | Sacred Duty

  7. Cody C says:

    After playing my brewmaster on beta and then reading the forum posts I found that their was a very subjective undertone when considering quality of new talents. This led me to a simple question that morphed into my first TheoryCraft/SimCraft project.

    I have a decent understanding of the scientific process and data analsysis from my Biochemistry degree, and wanted to apply this to the create an objective way to think about these talents.

    I am really just getting my feet wet. I started by looking at things numerically, but quickly found that the number of variables was really high. I have since designed a brewmaster sim, in python, able to answer my specific question. As with science, answering one questions usually reveals another, so here I am.

    The post was well written and designed, I enjoyed reading it. Information within was basic, but I like where this series is going. I would find a lot of value in discussion regarding a challenging theorycrafting/simcrafting problem that you faced, and how you solved it.

  8. Pingback: Warum Games in Bibliotheken gehören? 2 | Nachrichten für öffentliche Bibliotheken in NRW

Leave a Reply