Crowdsourcing TMI

As I’ve mentioned a few times on Twitter already, I’ve been working on refining the formula used to calculate the Theck-Meloree Index. The current version certainly works, or at least, gives me the numerical effects that I initially wanted. But in the 6+ months since we defined the metric, we’ve learned a lot more about the quirks involved with having a raw exponential metric. Several of which are more rooted in psychology than mathematics!

(If you’re keeping score, that’s Wrathblood – 1, Theck – 0)

In any event, after a bit of playing with the possibilities I’ve finally decided how I want to modify the formula. It’s all but finished, in fact. The only thing left to do is fine-tune some constants, which I think I’ve done sufficiently well already. But the only good way to test that is to generate a lot of data and see if it’s working the way I want it to.

Which normally would be fine, but there are a few issues with doing that myself.

  • It takes a long time to generate the amount of data I’m looking for. Think several hundred simulations, each with 25k iterations, and calculating 10 scale factors.
  • I want to test it on a variety of gear sets. Again, it takes a lot of time to put together gear sets, and I don’t really want to troll the armory looking for random players to import.
  • I want to test it on all five tanking classes (or at least, the ones that SimC supports). Again, short of trolling the armory, it would be a formidable task to find an appropriate number of players to get a proper sample. And would take a long time.
  • I want to test it against multiple different TMI bosses… so multiply all of those time investments by a factor of four or five.

I’m busy enough as it is with all sorts of other projects (*cough* and Diablo III), not to mention my job, that it’s not feasible for me to generate all of this data myself. Unless you want to wait for the new TMI definition until December. Of 2017.

I could just release the new metric into the wild, of course. I’m pretty sure it’s functioning properly, after all. But I’d much rather be able to do some rigorous testing of it in case there are weird problems that I didn’t anticipate.

This is where you come in. Instead of running several hundred simulations myself, I’m asking each of you to run a simulation or two for me. Basically, you could consider this the public beta test of TMI 2.0.

How To Contribute Data

I’ve coded the new TMI definition into Simulationcraft, and it’s available as an option in version 547-2. By default, it will calculate stat weights using the old formula. However, you can enable the new formula with the argument new_tmi=1.

You can do this by adding that line to the Simulate tab as shown in the screenshot below:

On the Simulate tab, add "new_tmi=1" after your character definition to enable the new formula.

Add “new_tmi=1″ after your character definition to enable the new formula.

The results page will then report TMI as calculated using the new formula.

If every reader of the blog runs their own character through the sim, I will have a veritable sea of data to swim through (as in, many thousands of simulations). I’m not that optimistic about a 100% reader-to-data-submission conversion rate, so if you can run your character several times with different options (i.e. against different TMI bosses), that’s even better.

Here are the basic guidelines that I’m looking for in submissions:

  • 25000 iterations
  • Standard Patchwerk fight (these should all be SimC defaults)
    • Length: 450
    • Vary Length: 20%
    • Style: Patchwerk
    • Level: Raid Boss
    • Target Race: humanoid
    • Num Enemies: 1
    • Challenge Mode: Disabled
  • Standard Player settings (again, defaults)
    • World Lag: Low
    • Player Skill: Elite
  • Scale Factors (make sure you choose to scale over “tmi”)
    • Strength or Agility (depending on your class)
    • Stamina
    • Expertise
    • Hit
    • Crit
    • Haste
    • Mastery
    • Armor
    • Dodge
    • Parry

All of these options can be found on either the Options->Globals tab or the Options->Scaling tab. First, a quick look at the Globals tab:

SimC's General Options tab.

SimC’s Options -> Global tab.

You can see that I have all of the settings at default here. The only two I want you to play with are the TMI Standard Boss and the TMI Window.

For the TMI Boss, pick one (or more) ilvl-appropriate bosses. For example, if you’re in heroic T16 gear, then you shouldn’t bother simming against the T15 bosses at all, and probably not against T16N10. Stick to T16H bosses or the 17Q boss. Please do not use “custom” – that will pit you against Fluffy Pillow, who is not so fluffy anymore now that he learned how to perform melee and spell nukes.

For TMI Window, the standard is six seconds. Feel free to leave it at that if you’re getting reasonable results. If you get really weird-looking stat weights or your TMI is below, say, 1000, consider dropping this a little, maybe to four seconds. Please submit the wonky stat weight data anyway, because that’s also useful to me, but then submit the (hopefully) normal-looking data you get using the lower TMI window.

On to the Scaling tab:

SimC's Options -> Scaling tab.

SimC’s Options -> Scaling tab.

As you can see, I’ve checked all of the stats I’m interested in. If you’re a druid or monk tank, please check the Agility box too (in that case you can skip Strength if you want to). Above all though, make sure you’ve chosen to scale over “tmi.” I can’t stress this enough, because if you scale over DPS you’ll get scale factors that are useless to me, and it will just mean I have to spend time filtering the data to eliminate those useless data points.

Once you’ve completed the simulation, you can enter the data in the form below. Please also attach the html results (which you can get by using the “Save” button at the bottom right of the results pane in SimC) using the “Upload” button at the bottom of the form. I’m requesting the html so that I can sanity-check the data and figure out what’s happening with outliers, so you can’t submit data without first attaching that file.

There’s no limit to how many times you can submit data, so you can run several different characters through the simulation if you want to. In fact, that’s encouraged, because data from undergeared alts is just as valuable to me (if not more) as data from overgeared mains. And of course, you can run a character against several different TMI bosses and submit each result separately.

Just don’t keep re-submitting a single simulation result multiple times, because each submission after the first would be useless for obvious reasons.

If the embedded form below isn’t working for you for some reason, you can also access it directly via this link: http://goo.gl/SY36xu. Note that you’ll have to reload the page (or open the link in another new tab) in order to use the submission form again.

 

Thanks in advance for your help! Depending on how quickly the data comes rolling in, I may be able to have this all wrapped up as early as next week.

As soon as that’s done, I’ll be making a much longer post detailing what changes I’ve made, why I’ve made them, and how the new formula works, including simulated data that I used to develop the metric and actual data from this exercise.

This entry was posted in Theck's Pounding Headaches, Theorycrafting and tagged , , , , , , , , , . Bookmark the permalink.

29 Responses to Crowdsourcing TMI

  1. Akenia says:

    Good evening/morning :)
    Would it be helpful to sim against the same boss e.g. T16N25 multiple times but with different Gear or/and Talents, Glyphs?

    greetings, Akenia

  2. Bindura says:

    My alts are T15 boss appropriate, so I ran sims against T15 25N boss. The submission form only has the T16 boss options, should I just scrap the T15 boss sim or is this still useful data?

  3. Uploaded mine — although I am not sure if the values i got seem normal or not in the first place.

  4. nextalcupfan says:

    I just finished simming my main on all the T16 and the T17Q bosses when i noticed something odd with my vengeance. On the T16H25 I had 508209 max vengeance and on the T17Q i had 662881 max vengeance, what’s odd about this is that I have 974013 health fully buffed according to the stats section of the results which means I shouldn’t go over 487084.5 vengeance since vengeance is capped at half health. Is the max vengeance value theoretical or is the sim allowing your character to go over its capped vengeance?

    • Xfighter says:

      I’m pretty sure those changes were only proposed during PTR testing, but never made it live. They instead reduced vengeance by doing the # of attackers decay, and by reducing the coefficients.

      In actual tanking scenarios on 10H I frequently hit 700-800k+ vengeance on fights like Malkorok/Thok/Garrosh, despite my max health being between 1M and 1.15M

      • Çapncrunch says:

        Indeed, those vengeance caps were dropped in favor of just nerfing vengeance generation instead. The only cap on vengeance is 100% max health that was added like pre-ToT to prevent silly things like the level 85 pally that solo’d stone guards.

  5. Çapncrunch says:

    I was really looking forward to simming myself and submitting my results. But unfortunately my server is still 404 from the armory since our merger 2 weeks ago :(

    (also, you’re probably not the person to tell, but importing characters with alt-codes has become extremely frustrating ever since SimC changed the alt key to be a tab shortcut sometime around version 5.4, now I need to type my character’s name somewhere else and copy/paste it into the import page)

    • Çapncrunch says:

      On an unrelated note Theck, I had some questions I wanted to ask in general. First is what is your thoughts on our symbiosis? 1 holy power for 6s or 10% damage reduction on a 1min cd. My first reaction was that it seemed rather underwhelming and not worth spending the holy power on. But that’s just my opinion and was wondering what you (or others) thought about it.

      Second was a question I just saw someone ask on the paladin forums, was regarding the Ret-pvp 4p bonus, which generates 1 holy power every time you take direct damage with an 8s icd. While actively tanking that would be 3hp every 24s (give or take a swing timer) which seems on par with our current 4p (when maintaining EF, probably better than our 4p if using SS). Or am I missing something obvious here like it not working for prot or in raids? My main thought/concern on this is in the future as the pvp setbonuses tend to stay the same, so when we have to leave our current tanking bonus behind the pvp bonus may still be available (and if they do stick to their intent and make crit more useful for us) I could see this possibly impacting gearing for us.

      • Theck says:

        I keep meaning to run a simulation to test Barkskin, and keep not having time to do so (and/or forgetting). I think I’ve coded it into SimC properly though if you want to give it a shot. It’s probably not significantly better than using that 1 Holy Power towards SotR though.

        I honestly wasn’t even aware of the Ret PvP bonus. 3 HP every 24 seconds is a fair bit. I wouldn’t be surprised if it didn’t work while in prot spec, but I haven’t tested it specifically.

        • Çapncrunch says:

          How would one go about testing the symbiosis abilities in it? Are we able to just add them to the priority as if we always had symbiosis or does something special need to be checked to enable it for us?

          And yeah, I wasn’t aware of the pvp bonus either until someone asked about it because they were using pvp gear to supplement their new prot set anyways and it caught my eye. While it would make sense from a pve perspective that it wouldn’t work for tanks, I’m not entirely sure since there’s no pvp-tank gear (aside from shields), so tanks (as far as I know, not really a big pvper) are stuck using their class’ pvp gear anyways. Would seem sort of odd to not let pvprots use the bonus. But I dunno.

          • Theck says:

            I’m pretty sure you can just add them to the priority as if you always had symbiosis. I figured it was unlikely that we’d be gaining and losing Symbiosis during combat. I haven’t tested it very much at all, so let me know if there are any noticeable bugs in your simulation results.

            Yeah, I’m not sure they would disable the gain for prot in PvE content – it would depend on whether they recognized the potential for abuse, I’d imagine.

            Isn’t the PvP gear significantly lower ilvl though? Or did that change (I haven’t been following PvP at all this expansion)? If it’s enough of an ilvl (i.e. Stam) loss it may offset the gain.

          • Çapncrunch says:

            The pvp gear is lower ilevel, the current season is 522, and can’t be upgraded with valor. So yeah, it is lower than LFR gear atm. But given the results you got when you compared using 4 LFR tier pieces to using 4 hwf non-tier, I’d imagine that it would put the pvp pieces about on par with regular heroic off-pieces (maybe a little behind due to there probably being crit on a few of the pvp pieces and lack up valor boosts).

            So unless someone already has their 4p or access to heroic items in those slots, I wouldn’t be surprised it the pvp set was a gain…..which I don’t like.

            Though as I said my concern was mainly in regards to WoD and beyond. On whether or not we’ll “need” to pvp at the start of the expansion to get the pvpbonus until we’re able to collect a full set of tier (I’m sure all of the mythic raiders will be able to put together their tier sets fast enough that it won’t matter to them, but for a lot of players it may still be faster to cap conquest each week). And this also depends on how they handle the ilevel difference between pvp and pve in WoD, people already complain about the disparity in wpvp and with base resil and battle fatigue going away that’ll only get worse.

          • Çapncrunch says:

            Oops, apparently I was mistaken, the 522 pvp gear is actually last season’s, which is now buyable with honor. The current season’s conquest gear is actually 550 ilevel.

    • Çapncrunch says:

      Well, had to set it up manually in AMR, but simmed my pally and submitted it. Did it on 25n and 25h. My TMI on 25n was under 800, but I didn’t bother messing with the TMI window and just went with a harder boss instead (also, I’m confused, you say that if the TMI is so low to reduce the TMI window? wouldn’t that just reduce the possible size of damage spikes and make the results appear artificially smoother?)

      Also, I wasn’t sure if for the scale factors you wanted the normalized values (that’s what I put), or the raw values. The fact that your placeholder text is negative for them suggested you wanted raw, but the raw values seem so imprecise (because they’re only listed to 2 decimal places). Like for my 25H sim strength, armor, dodge and parry all came up as -0.04 raw, but once normalized they were all different and ranged from 0.41 to 0.50 which seems far more useful. So I went with normalized.

      • Theck says:

        It depends on the situation. We take the moving average over the window and multiply by 6 to make a six-second damage intake estimate. For continuous uniform damage it would make no difference, but in practice we have discrete damage and healing events. A shorter window means that there will be more cases where you didn’t get an SoI heal or multiple EF heals within the window, leading to a bigger average damage per second in that window. So when you’re swamped with self-healing, decreasing the window tends to increase TMI.

        The case that would match your intuition is if we were really summing the damage over that six seconds rather than finding the moving average and multiplying by six. In that case, you’d drop the window to reduce TMI. I’m still undecided as to whether it’s worth changing back to that method of tabulation – it’s one of the things I’ll be looking at in this data set.

        The stat weight values are another thing I’m scrutinizing carefully. I don’t like the fact that the absolute stat weights are so imprecise. The normalized ones are more useful, but those aren’t the ones displayed on the graph. There are a few ways to fix that, one of which is to just crank up the scaling factor (ex: multiply it by ten so that instead of a TMI of 800, you’d have a TMI of 8000 and all the stat weights would be higher by a factor of 10).

        • Bindura says:

          I believe the most important aspect of Simcraft for the majority of users is accurately calculating stat weights so if you are not happy with it I say change it.

          Regarding TMI window, I think it will come down to education vs simlpicity/abiltiy to understand the setting.

          When I first daw it I thought “Oh, this must be a burst window, so if I reduce it my TMI be better”. When I looked at the results I deduced that you running an average across the 6/4s, and the longer the time the more smooth the average would look, thus reducing TMI with increasing time.

          I think the readers of your blog will have a bias towards thiinking TMI window is a burst window as that is how your MATLAB sims worked. I don’t know if there is a way to get feedback from people who won’t have any preconceptions other than just releasing it as is.

          An alternative to reworking how the metric calculates would be to show an info/comment box that has an estimated equivalent burst window for each setting. This info shouldn’t display only on a mousover only (I only saw the info on sensitivity to burst in the mousover while writing this).

          Do you have enough data yet? I could use AMR to optimse my tanks using different gear sets and export the character info.

          • Çapncrunch says:

            As far as the TMI window setting, because it is more of an advanced setting, I think it’d ok if understanding its function is somewhat advanced itself, as long as there’s a comment giving a general explanation of what increasing/decreasing generally does, then it’s not necessarily important how intuitive it is or isn’t. Those interested in how it actually works can always go find that info somewhere else. For example, if the player skill option listed the numerical value instead of the colorfully named sample values it wouldn’t really matter much whether 0% meant “never mess up” or “never be right”, as long as there’s an indication as to which is which.

            As far as the raw stat weights, first for the general public I think the normalized weights are what most should be looking at (for one they look more like what statweights generally look like), although I guess the barchart is a bit more eye-catching than the table. Cranking up the TMI scale would work for the numbers, dropping some of those leading zeros for more meaningful decimals, but I’d imagine the barchart itself wouldn’t change (assuming that the bar lengths are calculated from the actual values and not the truncated ones). There’s also the issue that it’ll probably be counter-intuitive for the tank metric to be scaling up significantly in the face of the stat-squish on the way (not entirely sure how exactly the stat-squish will affect TMI, but I imagine you’d need to crank it up to it’s current/higher ranges to get that level of scale factor accuracy).

            You could also potentially crank up the stat-delta used to calculate the weights, though as you’ve pointed out in the past that carries its own issues with statcaps.

            Is it possible to just have SimC multiply the raw weights before displaying them? IE just multiply them by 10, without changing the scaling of the overall TMI? They’d no longer be a direct modification to the overall TMI, but I think the relative values are more important than their direct relation to the overall performance. Or heck, just force SimC to display 3 or 4 decimal points instead of 2.

          • Theck says:

            @Bindura: I have 184 responses so far, though I’m sure I’ll have to prune some of those (ex: people who forgot to use new_tmi=1). More data always helps, but I think what I have should be sufficient. I just need time to actually sit down and analyze it.

            @Cap: TMI should be mostly unaffected by stat squish, because it’s based around damage taken relative to player health. If anything, I expect it to go down slightly in WoD if their comments on the healing model are true. So cranking the scale factor up isn’t an awful option, but I’d need to play with the data some to see whether it causes any other problems.

            The graph is based on the rounded (displayed) values, unfortunately. SimC obviously keeps higher precision than that, which is how we get accurate normalized stat weights. I can add some code to make the graph act differently for TMI stat weights than others (e.g. multiply by ten or use normalized values or possibly even use more decimal places), but I’d like to avoid that if possible just for consistency.

          • Çapncrunch says:

            “@Cap: TMI should be mostly unaffected by stat squish, because it’s based around damage taken relative to player health. If anything, I expect it to go down slightly in WoD if their comments on the healing model are true.”

            That’s what I meant. Because TMI won’t really be affected by the Squish, scaling it up by 10 to get better raw weights would cause TMI to go up, while everyone else’ goes down. (ironically, both of those things would “appear” bad, though they’re both just scale changes).

  6. Wrathblood says:

    To be fair, prior to that particular disagreement, the record was something like 186 to 0, in Theck’s favor. I’ll send in some sims this evening.

  7. Christopher Armstrong says:

    I entered several wonky data sets and then realized I forgot to put new_tmi in. Whoops.

  8. Anonymous says:

    When filling in the linked form with a value that does not apply to the character (e.g., parry or strength for druid tanks), should I leave the form with the default value or fill it in with 0?

    • Theck says:

      I’m basically done collecting data at this point, but if you still want to submit it, you can just put a zero in that field.

  9. Rainbo says:

    Did you get enough data for Monks? I can give you some. I’m really curious about using Simc for monk tanks.

  10. Alpheus says:

    It’s important to note that if you’re importing druids from the armory, the APL will usually default to position=back. I’m not sure if this has any side effects when set to role=tank but I’ve manually changed it to front nonetheless.

Leave a Reply