The issue you have to contend with in the context of Middle Earth and the LOTTR universe is that moral and spiritual forces are as real in their effects as physical and environmental ones.

Thus one of the reasons why Arnor remains largely uninhabited is that there is no King (or Queen) there. In Tolkiens' world, a realm without a King cannot prosper, so that Arnor lies empty and Gondor, under the Stewards, slowly declines, the Shire, under Thains who hold authority from the King, remains static. But Rohan, Dale and Erebor, all ruled by Kings, are dynamic and vigorous.

Angmar may have been destroyed, but the evil of the Witch-King will hang over its' lands and render them infertile, as well as affecting the hearts and minds of those who settle there. In the same way, Ithilien is uninhabitable due to the evil that emanates from Minas Morgul.

Also, you left out Imladris/Rivendell, which is not just a country house, but a fortified settlement of some size. It is not only home to Elrond and his family, but a number of Sindar and Noldor from Lindon who were sent there with Elrond. It is also home to what remains of the noble families of Arthedain, along with other refugees from that realm. The Rangers are, we can guess, the Bannermen of the Chieftains. As to the remainder, they are probably farmers and craftsmen, though some might serve Elrond as men-at-arms. They may number in the thousands. There were certaily enough for King Elessar to begin re-peopling Arnor after the War of the Ring.