The question to ask yourself is: Which suits the core story best?
Here's a trick my mentor taught me: Genre is superficial. It is set dressing, one of many mediums with which to tell your story. Your core story can be told in any medium, in any genre, and with any number of combinations of other elements. So the ability to separate your core story from genre is extremely important in understanding what your story is truly about.
In your case, I would hone in on your core story. Your core story isn't about a hidden group of people--it's deeper than that. It's about one outcast soul trying to find acceptance, or whatever your main plot is. (I just made that last bit up by way of example.

) So figure out what that is, and what is truly important about that story. Then figure out which genre serves that core story best.
As someone who has switched genres of her stories before (sometimes during the final stages of editing, ha!), I find it helpful to imagine how the story would play out in another genre. Write up a brief plot synopsis. What are the differences? What are the pros and cons? Are there characters whose backstory makes more sense in one genre or another? Are there elements of the real world that would tie in well to your story? Are there things that would just be stretching the suspicion of disbelief too far? Which feels more natural? Which excites you more?
The best genre will "feel" right and will mold to the story. If it feels like you're forcing it, reconsider.
I think contemporary fantasy is a bit trendy right now, what with all the vampire fiction and urban dragons and whatnot, which could be good or bad. Another idea is to imagine how the cover design would look for your novel. If it's contemporary fantasy, what kind of picture would you put on it? High fantasy? Which sells the story better?