IMO, it is easier to select from a group of females for all traits, than it is to try to discern the traits of the male. I don't think folks do much of a selection when it comes to males, other than they have one and it looks good, so it gets used.
If using a group of females, on the other hand, it is much easier to discern which plants would be desirable and which are not. If you have two killer female sisters, and force stamen from one and use it to pollinate it's sister, you will have a real good idea of what the progeny will hold. Much more of an idea than if you used on of it's brothers, of which you really know little about.
How many folks grow a male all the way out to check on hermie traits? How many actually smoke the male to see if it is even worth using?
IMO, pheno expression is going to/can be much more funneled down using stressed pollen than using male pollen.