A common complaint I've heard made about Middle-earth, and fantasy settings in general, is stagnation. They claim that there simply isn't enough progress: not enough scientific investigation, no new technological innovations, not even major social upheavals, like you see in "real" history. I never had a problem with Middle-earth's technological progression, personally. It's extremely difficult to progress when, every few millenia (or even centuries), your entire world is torn to pieces in nothing less than cataclysmic events, where countless lives are lost and the very geography of a continent is altered.
Let's look at the First Age, for instance, from the Awakening of the Elves to the War of Wrath. Now, Middle-earth timelines can be problematic due to the nature of the world's creation, not to mention Tolkien changing his mind on the length of a Valian year. The best resource I can find for this is the redoubtable Encyclopedia of Arda, which gives us 4,902 (450 Valian years and 590 Solar years), which supports Tolkien's assertion of the First Age lasting the longest. Including the Years of the Trees complicates matters, as the EA calculates a further 14,325 years. Nonetheless, we can fairly easily say that the development of Elven technology from their awakening during the Years of the Trees (assuming that can be analogous to the dawn of civilization in our world) to the War of Wrath was pretty decent.
At the Elves' finest hour, they could construct gargantuan hidden citadels, forge synthetic jewels superior to natural gems, and flying battleships made of metal and glass. Early drafts of "The Fall of Gondolin" feature Morgoth's forces fielding armoured personnel carriers shaped like dragons. That's under 20,000 years after the equivalent of the Stone Age. However, the end of the First Age halted all that through the sinking of Beleriand. Entire stretches of the north and west continent sank beneath the waves. Everything north and west of Eriador went under. That's a disaster unseen in modern times, like the entirety of Europe sinking. So with such a catastrophic event, it's easy to see civilization was held back a little bit.
I'm not entirely satisfied with either map, but you get the idea.
So we can imagine the beginning of the Second Age to be pretty solidly post-apocalyptic. You have the surviving Edain going west to start afresh in Numenor, the few elves who stayed after the mass exodus to Tol Eressëa founding new settlements, and the Dwarves leaving their destroyed kingdoms to establish Khazad-Dum. The knowledge to craft flying ships and dragon machines was lost. It took Numenor 600 years to travel back to Middle-earth, and another 300 to establish new colonies. Fast forward another thousand years, and Numenor has become the mightiest nation in Middle-earth, with pretty impressive technology: in addition to steel longbows, nigh-impregnable masonry and incomparable armour, an enigmatic mention of "darts" like "thunder" which "pass over leagues unerring" conjures visions of advanced artillery, or even self-propelled rockets. That's when Sauron strikes back. He pretty much devastates half of Middle-earth in his war against the Elves, destroying several kingdoms and running roughshod over Eriador, in constant battling that lasts centuries. A good 1,500 years later, and at its arguable height of power, Numenor sinks under the waves, Aman and Tol Eressëa are "removed" from the world, and the world is made round. Which, again, caused massive geological upheaval.
Like the Second Age, the Third Age starts out post-apocalyptic, only to a far more significant degree. The Elves are a dying breed, restricted to isolated communities deep in the forests; the Dwarves are ever retreating into their mountain homes; Men have been tearing each other apart in warfare. Initially the remnants of Numenor do alright, founding the new kingdoms of Gondor and Arnor: the latter kingdom breaks up into three smaller realms a scant 800 years after its foundation, those realms themselves being destroyed when the Witch-King returns 500 years later. Gondor is being invaded by Easterlings and Corsairs, devastating the population. More disasters - great plagues, invasions, disasters, wars - follow. The period known as "the watchful peace" - one of the only periods of peace Middle-earth has enjoyed - starts 2000 years into the Third Age. It lasts 400 years.
Thus, the First Age: 4902 years. The Second Age: 3441 years. The Third Age: 3021 years. So, using the estimates above, around 11,364 years have elapsed since the Age of the Sun by the time of The Lord of the Rings. Let's assume that the rising of the sun is the rough analogue to the beginning of civilization. The earliest walled city yet known is Jericho, which has walls dating back to 6,800 BC. Byblos was founded around 5,000 BC, Sidon 4,000 BC. The spear has been the killing implement of choice in some form for 400,000 years, while recognizable bows have been around since 8,000 BC - though there's possible evidence of arrowheads from tens of thousands of years earlier. Taking into account the sheer amount of disasters which befell Arda and sent civilization back to the drawing board, combined with the obvious longevity of low tech, I don't think it's at all unreasonable for Middle-earth to still be using bows and arrows, especially considering the role of magic and long-lived Elves and Dwarves.
If I've made a particularly gratuitous error in my dating, I apologise: anyone more well-versed in Tolkien chronologies feel free to point out how badly I've snafud.