The reasons I believe the CV season starts later and ends quicker are due to 2 main reasons:
1. Water temps are at their peak then start to decrease after now. Remember, the eastern side of an ocean basin has cooler water temps farther south than the western side due to the upper level ridge of high pressure over the cooler waters that creates a wind flow conducive to upwelling. That's why the waters off of California are colder in August than NJ usually.
2. Upper level patterns start to change, with the jet stream making its return, and either systems get turned NW quickly, or get sheared apart. Later on, the ITCZ does move farther south.

There may be others, but I think those are the 2 biggies.