Smaller storms do have the potential to strengthen and weaken more rapidly than their larger counterparts. That is due, in part, to conservation of angular momentum as the winds go around the center of the storm. The leap can be made to say that most intense hurricanes were the smaller ones, because it is easier for them to reach that level at a quicker rate -- they are influenced more both positively and negatively by their environment -- than for the larger storms.

It's not to say that large storms can't reach category 4/5 status, however; Frances would suggest otherwise, as would Mitch. You have to remember, too, that most storms grow in size (and weaken, but this is generally only weakly related to the size argument) as they head north into the midlatitudes; those that continue west are unaffected by the midlatitude steering currents and have less of a proclivity to grow in size. Camille would be the exception.