No one really bothers them, so it isn't very likely. The Men don't know any other races exist because they never venture beyond their small villages and castle town. The Elves leave them alone because the two groups are busy fighting each other. There would be a bit of trouble with the people living in Telslath Forest (I haven't worked on that culture yet, so I'm not positive what it's like... although I do know it's rather strange), but I don't think they would interact enough to cause a very large problem.
If anyone did decide to attack them, it would be hard to get near them, because unless they were on the northeastern side of the mountain (I know nothing of the land in that direction), they'd have to cross the river, and there's only one bridge across. Without going farther north than is practical, they couldn't just use boats or anything because the river is too swift. The river is cursed, so if you fall in you fall asleep, which causes your body temperature to fall and you end up freezing to death. It's possible to survive falling in, but unlikely.
Haha, that was probably more of an answer than necessary. I had fun explaining it though

I'm kind of losing interest in this story, though... and writing isn't much more than a hobby so I'm wondernig if I should continue editing. I might work again later, but I'm almost ready to just leave it for now. I want to start a couple more stories eventually, but I'm afraid I won't ever finish those either.