Everything you said is true to a degree but I'll start my answer with
a BIG it depends. There is no easy answer.
1st off markets like "foreclosures and refinancing and debt consolidation" are not just competitive it's uber competitive because you are not only dealing with shark Internet marketers but banks, loan brokers, news papers and other big media talking about the subject.
There ARE markets that have demand and not every marketer on the planet is targeting. There won't be as many searching and buying but you have a better chance of getting traffic and therefore making sales. So key is to weigh # of searches vs competing pages based on "all in title".
This thread will explain more;
Need help with SEO numbers - KeyWord Popularity vs Competing Pages?
The guys over at the 30 Day Challenge that are teaching newbies how to make just a little money for starters in their 1st 30 days are saying to find a niche with less than 30,000 competing in "quotes".
Here's a video about it.
Need help with SEO numbers - KeyWord Popularity vs Competing Pages?
"But if you pick something really obscure (I just typed in "vintage books 1900s" and came up with 53,700 hits) then you hardly have anyone to sell to/click on your ads/make money off of."
True you don't want to go totally obscure. The key is to find hidden niches that have interest but isnt too competitive. OR it could be a sub-niche of a big niche. Debt consolidation for Spanish single mothers. (Not saying that's a good market or even gets any searches, just an example.)
One idea that could possibly help you find some niches that would be good but possibly not too competitive. Go to the
SEObook KW tool.
Try some general root keywords that signify problems people are trying to solve. Single words like: eliminate, cure, pain, relief, problem, lower, relieve prevent, alleviate, soothe, ache, comfort, comfortable, lower, reduce, remove
For each one scroll down in the results until you find something that interests you that's not at the top of the list. Too many people go after those top ones. Find one midway down and then check to see how many competing pages in Google with all in title.
Sorry carpal killing, thats all I can type for now.
Hope that sorta answered and gave you some ideas.