Now I don't want to give the impression that I think mathammer is rubbish, but I do think there's a tendency to over rely on it when picking an army list. Look on any forum and you'll often see comparisons of troops and weapons using mathammer, resulting in a recommendation for one or the other, because on average it causes more kills.
That's great, and in absolute terms I can really see the benefit to such an analysis, even breaking it down further to establish a kills per point ratio. I don't agree however that mathammer is the only way to assess effectiveness, nor do I think it's actually representative of a unit's actual efficiency. Let me elaborate on this point therefore, using a 10 man space marine tactical squad with bolters shooting at another marine squad as an example.
Within 12" - 20 shots, 13.3 hits, 6.67 wounds, 2.2 kills.
So what that tells us is that realistically we can expect on average 2 dead opponents in a turn's shooting. Already therefore we're seeing the failings of mathammer, because we've just discounted 0.2 kills to get a realistic assessment of the likely damage.
My own approach is to look more at the level of firepower required from a certain type of weapon to inflict a certain level of damage, so using the example above I would say that in order to cause those 2 unsaved wounds, I would need to cause 6 wounds before armour saves, and therefore 12 hits, which would similarly require 18 shots.
Here's where you can clearly see the difference therefore, in that mathammer would tell you that you will cause on average 2 casualties with a full squad. My own approach tells me that a squad of nine marines will on average (4 times out of 5) cause exactly the same amount of damage, which saves me points to spend elsewhere. Alternatively, this approach can also be used to assess the firepower required to cause complete wiping out of a unit, regardless of range.
Using the previous example in order to achieve complete elimination of a ten man squad you need to cause 10 unsaved wounds. This requires 30 wounds to be caused that allow armour saves, which in return require 60 hits to cause the wounds necessary. Those 60 hits require 90 shots to be fired. I can, using this approach, figure out the level of firepower I'm going to take to cause this level of damage. My approach tells me that 9 full squads outside rapid fire range would be required. Mathammer gets you there too, but the big difference is that my approach cuts out the '0.3 of a kill' that pure mathammer can leave you with.
My second concern with mathammer therefore is that it only looks at pure damage dealt, and cannot take into consideration other things such as gut instinct and threat that units can provide over and above their pure damage dealing ability.
Resilient units therefore add to the consideration, take for example dark eldar wyches. They have the potential for significant damage dealing capabilities, particularly against lightly armoured foes, however their average toughness and minimal save mean that their combat effectiveness is drastically reduced, as they are very likely to take significant casualties before getting into combat. Conversely, terminators actually have a relatively modest damage potential, due to their low number of attacks, but they are able to apply those attacks turn after turn due to their resilience to damage and as such their usefulness is multiplied.
Thirdly then, a unit's effectiveness can come from its synergistic effects. A prime example would be Dule Sliscus from the Dark Eldar codex. This character in terms of pure damage dealing potential is relatively minor when compared to many, however his ability to influence the combat drugs used by the army as a whole, plus the boost to the potential of a unit to which he is attached mean that he is significantly more valuable than the mathammer figures alone would suggest.
Finally then, the matter of the threat posed by a model also cannot be considered by mathammer alone. Here I'll consider Abaddon, though this element can be applied to many other choices within an army.
I can't imagine many people ignoring the master of the Black Legion if he's on the table, in fact many, and I count myself among this number, would actually dedicate a disproportionate amount of effort to removing him from the field. Other situations that fall within this category could include infiltrating units forcing heavy weapons to move out of range and thus not fire, with the infiltrating unit not necessarily being capable of significant damage but its usefulness being amplified by the prevention of damage being caused to its own army.
In conclusion therefore, I support mathammer as a purely theoretical vehicle for calculating the likely damage output of a particular unit or weapon loadout, but I would strongly counsel caution against using it solely to assess a units effectiveness. In fact, I often discuss selections with my clubmates, the philosophical question of 'what's more scary' being very useful for giving you an opponents view (providing you trust them not to bluff you of course!).