Your problem, sounds close to what I'm going thru at the moment ..I myself have changed out the starter wire/solenoid from the battery to the starter ( no fix)..I tried a higher amp fuse ( no fix) .. I bought a higher ( 380CCA. battery) ..thru GAST .. seems to have helped, but it is not the cure.
I have torn my starter apart several times and replaced the brushes / sanded the Armature ( 400 then 600 ) ..till its babyass smooth ..seems to help for awhile ,but alast ..the problem pops up its nasty head ,and I'm back to pushin .. the only thing I can figure is the Armature is getting weak .. the price for a new Amature is just a little less than a new starter .
To this I've resolved that I need to get a new starter, because the price for most ( used ) starters, isn't worth the delayed headache I'm going to get .. when the used one ....... takes a crap.