Until I saw one post on Matt Cutt's blog, I would think absolutely the same way you do, jcorkern. Because you are right, too many sites can get into trouble because of speed.
But Matt (deliberately or not - I don't know), unveiled the curtain. He was talking about the case when reputable service in Google's *eyes* that - here I quote - "We saw so many urls suddenly showing up ... that it triggered a flag in our system" and that was a very reputable site so that they had to manually solve the problem and they "cleared that flag" (for more details see Matt's blog post here
http://www.mattcutts.com/blog/scoble-visiting-the-plex/)
So, if there are flags (or at least - there have been flags), it means if you are greedy - it can affect your site. Of course, if backlinks are great - with time this flag should be removen, but no one knows how much time it can take for Google staff to manually review this flagged site. And no one can guarantee that if site is still being flagged (meaning site is strange or even suspicious to Google) - that it will not affect the ranks.
We are in the fields of suggestions here, and I can't guarantee you, jcorkern, that is DOES HAVE THE GREAT INFLUENCE. But out of the facts that I mentioned I see logics. Plus my link building experience that brought my site to 1st page in Google in link niche is a side proof of some real truth in what I am saying and practising. Anything can change tomorrow in algo, but that will be tomorrow
P.S. Thanks for bringing up a very good topic, jcorkern.