there are a few molds that will infect cannabis, PM is the white spotty one on the outside of leaves, then there's botrytis which is brown and usually starts on the inside of your densest tops. Botrytis is usually brown on fresh bud, but can develop a gray fuzzy mold, and will almost always become a gray fuzzy mold on dried flowers that were moldy when they were fresh.
As I said it usually starts on the inside of the bud or on a particular bract of the bud, rarely does it start from the top or outside of the bud.
If you crack your tops that are affected open, what do they look like? if it's brown at all inside it's moldy, you'll definitely be able to smell the stale/moldy smell and it DEFINITELY gives you a headache if you take a big whiff.