In catching up on the news from this week, I found myself reading comments on mmo-champion. I know from previous experience this is a bad idea. I know that reading these comment threads is only going to make me angry, and that brain cells will commit hare-kari in protest over having to suffer through it. And yet, like any good train wreck, I simply can’t tear my eyes away.
In particular, I was reading the recent post about DREAM Paragon’s world first 10-man Mogu’Shan Vaults clear. Which is a great achievement, no argument there. But of course, in the comments, people immediately started turning it into an argument about whether 10-man is harder than 25-man. Some choice quotes:
Ummm…. there is WAYYYYY more 10 man guilds then 25man guilds and the next guild in line has 2 bosses down in 10man…… lots of guilds are almost finished with 25mans….. nuff said.
Again, 10mans much harder than 25s. Can confirm myself.
10mans do matter, especially when those are the actual challenge.
Well, I`m more for 25HC myself, but looking at the progression of other hardcore 10-man raiding guilds having only 2/5, whilst 25-man guilds are 5/6 or 4-3 alread… it makes me think 10HC might be really badass in MoP.
Now, in the interest of being completely honest, I come into this discussion with a bit of a bias. I raid 25′s primarily, and it’s no secret that I think 25-man is generally the harder content. That said, I also ran a separate 10-man alt group in Cataclysm, so I do have experience seeing both formats when the content was relevant.
I’d also like to think that my bias has some basis in rationality. I don’t have a massive dislike for 10-mans, nor do I want to see them die, nor do I think that 10-man raiders are lesser people. I think that a 10-man fight can be tuned to be as difficult or more difficult as a 25-man boss can, and there are a variety of examples from Cataclysm that support that argument. In some sense it’s simple – you can always tweak the numbers up to make a boss in either format harder.
However, I do think that in general, Blizzard doesn’t tend to make 10-man bosses harder, but easier. There are a variety of reasons for that, but most revolve around two simple logistics principles:
- You can make a 25-man fight more complicated because there are moving parts. There are a large number of mechanics that require X people to do the right thing, and that type of mechanic gets exponentially harder the higher X is.
- Conversely, you can’t tune a 10-man as tightly because there are too few moving parts. It’s harder to tune a DPS check to be challenging in 10-man because the variance of raid DPS is much larger. Class stacking affects both formats, of course, but a 10-man has some inherent degree of this thanks to discreteness. If you tune such that the “stacked” groups are challenged, nobody else has a chance, and everyone feels compelled to class stack. See, for example, heroic Spine of Deathwing on 10-man.
However, I don’t want to talk about theory in this post. Hamlet has already done a great job of outlining the theoretical argument already, and I think his analysis is spot-on. Anything that I could write would end up being a re-hash of his points anyhow.
Instead, what I want to do is look at the data. In particular, we have some information about the world first race that we can critically analyze. Unlike the commenters I quoted above, let’s think carefully about what that data means.
First, we need to agree on what the data is. From the mmo-chamion post:
- DREAM Paragon cleared 10-man heroic at the time of that post.
- The next-closest 10-man team was only 2/6 at that time.
- Method and Blood Legion were 5/6 in 25-man at that time.
- At the time I’m writing this blog post (10/15), there is only one other guild at 6/6 and five guilds at 5/6 in 10-man heroic.
- At the time I’m writing this, all of the top ten guilds have cleared 25-man heroic.
Now, you might look at that data and draw the same conclusion that some of our mmo-champion commenters did. “Well, clearly 25-man must be easier, look at how many guilds have finished 25-man compared to 10-man!” But that’s not what I see in that data, and in fact I’d argue that without further information this conclusion is fundamentally flawed.
The reasoning is pretty simple, but it’s something most people never even think about. It rests upon the assumption that everything else about the two situations is equivalent. In mathematical terms, it’s assuming that there are no other variables that affect the data except for raid size. In the terms an economist might use, we’re assuming there’s no sampling bias. One of my favorite books, Freakonomics, discusses why this is generally not a good assumption and how you can use the types of normalization principles normally used by economists analyzing large data sets to improve the conclusions you draw.
As a simple example, consider this comment by another mmo-champion poster:
Yeah, or maybe it’s because the 25 man guilds that are so progressed are raiding many many hours a day compared to 10 mans not.
The poster raises a very good point. One of the things Ghostcrawler mentioned during our discussion with him at BlizzCon (if being ranted at by a drunken, belligerent Meloree counts as a discussion anyway) is that one of the biggest factors in raid progression is how many nights/hours per week a guild raids. We’ve implicitly assumed that all of these raid groups are raiding the same amount of time. But this is probably not the case – how many guilds in the world raid as many hours as Paragon, Method, Blood Legion, and the other top guilds?
It’s not as many as you might think. Out of the top ten 25-man guilds, eight claim to raid 7 days a week (and the other two are Blood Legion, whose profile claims 6 raids a week, and Method, whose profile claims 1 – obviously inaccurate). Out of the top ten 10-man guilds, only three claim to raid 7 days a week. The rest are all 4-5 nights.
And even then, it’s not clear how many hours of raiding that works out to be. Those other 7-night guilds may not be raiding as many hours per day as Paragon. If the 25-man guilds that have cleared Mogu’Shan are all raiding 40+ hours a week, but Paragon is the only 10-man guild doing so, then the conclusion based on the progression discrepancy doesn’t hold water.
And this is just one of many unsubstantiated assumptions that go into the conclusion that 10-man is harder. For example:
- It assumes that the difficulty ramping is identical on 10- and 25-man. Maybe the third boss is a major cock-block on 10-man, but the rest of the bosses are much easier.
- It assumes that either all or none of the 10-man guilds are class stacking for the fights. Paragon’s final boss kill had 4 druids in the raid composition. I don’t know if they used the same comp for the third boss, but it’s possible that they did and that it gave them a significant advantage over the other 10-man groups.
- Related to #2, we’re assuming that all of the 10-man groups have similar class composition. Maybe none of them are class-stacking, but Paragon had a particular class or spec the others didn’t.
- Perhaps most importantly, it assumes an equal skill distribution between 10- and 25-man guilds. If the skill level of the players in the 25-man guilds is higher than that of the 10-man guilds, then the whole argument falls apart.
Now, that last point is bound to be controversial, but it’s also likely to be the most important factor to consider. So let me throw some disclaimers at you. First, I’m certainly not trying to say 10-man raiders are “bads.” All of the players in these guilds play well, far above my skill level for sure. And I don’t think that raiding 10-man makes you a bad raider, either. There’s nothing inherent about 10-man that gives you a scarlet B. In fact, I don’t even think that the statement is true on average when applied to the greater WoW population. The average 10-man raider is probably just as skilled as the average 25-man raider, within reasonable statistical deviations.
But we’re not talking about the average WoW population here. We’re talking about the best of the best of the best. And these players and guilds don’t exist in a vacuum – they live in an environment where for 4+ years (more if you count Burning Crusade), 10-man raiding has been considered inferior. Note that I’m not saying it is inferior, at least not now. But it certainly was inferior in Wrath, and that stigma stuck with it during Cataclysm.
Whether you believe 10′s are easier or harder doesn’t matter, because that stigma exists, and it does matter. Because it means that the strongest players have, for the last 4 years, generally drifted towards 25-man guilds. Why? Because that’s the world-first race that most people have been focused on. 25-man was the major leagues (and 10-man the minor leagues) for so long that a natural skill inequity built up at the top. So much so that, even if the 10-man heroics nowadays are harder content, the average player from one of the top ten 25-man guilds is a little better than the average player from one of the top ten 10-man guilds.
Now, what evidence do I have to back this statement up? Nothing, in fact, because it’s all conjecture. But it’s no more conjecture than saying that 10-man is the harder format. The only difference is that I’m being honest: I’m throwing out a hypothesis, and I admit that it’s a hypothesis rather than a hard fact. I’m more than happy to be proven wrong, or to reconsider my stance given sufficient evidence (though, proving such a statement is tricky work at best).
However, let’s take this hypothesis and see how well it matches the data we do have. Or to be more specific, let’s see whether the conclusions we draw from it match what we see in the data. If we start by assuming that the 25-man guilds are statistically “better” than the 10-man guilds (with the exception of Paragon, which we’re treating as a 25-man guild for this analysis because it came from the core of a hugely successful 25-man team), what would we expect to see in the data?
If 10-man and 25-man were roughly equivalent in difficulty, then:
- We’d expect Paragon to progress at roughly the same speed as Method, Blood Legion, and the other top-ten 25-man guilds.
- We’d also expect there to be a fairly wide gap between Paragon and the closest runner-up in the 10-man category at any point during progression, right up until Paragon hits 6/6 (at which point, the gap will narrow because Paragon has hit a plateau and the other guilds have time to play catch-up).
If 10-man were considerably easier than 25-man, then:
- We’d expect Paragon to progress considerably faster than the 25-man guilds.
- We’d still expect the gap in 10-man progression, but that gap would probably be a little smaller owing to the non-linear nature of encounter difficulty. It would also close rather quickly once Paragon reaches the 6/6 progression ceiling.
If 10-man were considerably harder than 25-man, then:
- We’d expect Paragon to trail Method and the other 25-man guilds.
- We’d still expect a gap in 10-man progression, and the gap would probably be larger due to the harder skill check being applied.
Now, looking at the data we have, the first two situations seem to match. Paragon progressed a little faster than the 25-man guilds, and there was a large gap between Paragon and the other 10-man guilds during progression. The gap has also closed rather quickly, though probably not quickly enough for 10-man to be too much easier than 25-man. Our data seems to be consistent with “10 roughly equal to 25″ and “10 slightly easier than 25,” with the additional assumption that the 25-man guilds are more skilled.
But the data is flat-out inconsistent with the third situation. Paragon didn’t trail Method, and the progression gap isn’t remaining very large (though it is non-trivial, given how few 10-man guilds have finished the instance so far). Despite the progression gap, the data simply doesn’t seem consistent with 10-mans being significantly harder (again, given that 25-man guilds are more skilled).
Now, you might quibble with me here (and rightfully so). “Theck,” you might say, “you’ve decided that 10-mans weren’t harder, but you did so based on a conjecture – namely, that 25-man guilds are more skilled. What if the inconsistency isn’t due to the difficulty, but due to that conjecture being wrong?”
That’s a fair point, and one we should address. So let’s make the counter-hypothesis. Let’s assume that all of the guilds in question are equally skilled. What do we expect to happen then?
- First of all, we don’t expect to see any significant progression gap. It should be a pretty smooth distribution during progression, and once one guild hits 6/6 most of the rest should follow.
- If 10-mans are easier, then all of the 10-man guilds (including Paragon) will progress faster than the 25-man guilds.
- If 10-mans are harder, then all of the 10-man guilds (including Paragon) will progress more slowly than the 25-man guilds
- If they’re about the same, then all of the guilds will progress at roughly the same speed.
However, it’s clear that none of these are consistent with the data. The gap between Paragon and the rest of the 10-man guilds is strongly indicative of some sort of advantage, and it’s likely to be in the skill department (though, again, it could be hours spent raiding, or some other factor – we just don’t have enough information). As soon as you throw that conditional out, the “easier/harder” comparisons fall apart, because you’re not comparing the guilds evenly. Based on Paragon, 10-mans seem easier than 25; based on the rest of the 10-man guilds, 25-mans seem easier. The only way that can happen is if there’s some inequity between Paragon and the other 10-mans.
There is one potential solution that we haven’t considered yet. What if Paragon is made up of superheroes? I say that tongue-in-cheek, but more seriously: if Paragon was not only more skilled than the rest of the 10-man guilds, but also considerably more skilled than most of the 25-man guilds, wouldn’t that fit the data set? And in fact, the answer to that question is, “yes.” If Paragon were much better than everyone else, then we’d see exactly what happened in the data set.
However, I also think that’s an unsupportable conclusion if we extend our view to include a little more data. We know from the Cataclysm world-first races that Paragon was roughly even with Method, Blood Legion, and a number of other 25-man guilds. Those other guilds got some of the world-firsts, even if Paragon got more than any other single guild. That strongly suggests that Paragon is not made up of superheroes with magical lightning reflexes and clairvoyant minds. They’re extremely talented players, of course, but they’re not significantly better than the players in Method, Blood Legion, Vodka, Exodus, etc.
Not enough, at any rate, to be leading in progression by a large margin, and certainly not enough to be leading in progression in a much harder format. The conclusion that 10-man is a much harder format almost requires that Paragon be much more talented than anyone else around, because otherwise it doesn’t make any sense for Paragon to vault ahead the way it did. I think this is the single most damning piece of evidence against that conclusion, because we have years and years of data saying that Paragon isn’t several times better than everyone else.
In closing, I want to note that we haven’t “proven” anything in this post. I don’t think it’s really possible to take such a limited set of data and “prove” anything, to be honest. But we have shown that the data we do have doesn’t seem to support the argument that 10-mans are harder and the top ten guilds in each format are equally skilled, at least simultaneously. One or the other has to give, and the model that seems to best agree with the data we have suggests that both assumptions may be in error.
In other words, the years and years of 25-man being the dominant raiding format may be coming to an end, but it has not been without its cost. Over those years, skilled players have tended towards 25-man, such that there’s probably a noticeable gap in average skill between 10-man guilds and 25-man guilds. And furthermore, the progression results from Mogu’shan Vaults suggest that in addition to this skill discrepancy, the heroic 10-man version of the instance is roughly equivalent or a little easier than the heroic 25-man version.
Again, this analysis is completely constrained to the top 10-15 guilds in the world for each format. Don’t read this article and come away thinking that 10-man raiders are less skilled. Because that is not what I said, nor is it what I think. Such a statement is probably completely false once you consider the wider WoW population. But when you constrain your view to a very small subset of the population, such discrepancies can exist. And if one looks at the data with a critical eye before trying to draw conclusions, I think it becomes very apparent.