I don't consider myself an expert by any means, even though I have learned a bit of martial arts. Since no one else has replied, though, I will give my two cents.

I wonder if you could get a feel for it by watching YouTube videos of different styles? One thing I have picked up is that systems ending in -udo are newer, simplified (?) versions, while systems ending in -itsu are older versions. So, Judo vs. Jujitsu. Try watching training videos on different styles, or the differences between them. Take notes on the characteristics of different styles, and why they are that way, then pick and choose the sort of things you want. There are also multiple martial arts systems with bow staff and katana, etc. HEMA has something to do with historical martial arts as well. One fascinating tidbit I heard, but haven't found, is that a Latin American (?) country at one point was under a government that forbid them to practice fighting. So, they disguised it as a dance! Even now all their practice in that system (I can't remember what it's called) involves stepping from one side to the other in a sort of triangle. I remember seeing some stuff on here about Gracie Jujitsu, so you might try searching that. It seems that is a smaller school within Jujitsu?
One last thing, assuming you are using this for your novel, rather than trying to start a whole new system, is that you may not need to have a complete functioning system. Think Tolkien's languages. Some are complete enough to speak, others have a few words. You may be able to create the illusion of a full system, without having to actually create a fully functional martial arts tradition.