I put in two riv-nuts to mount the rear fender.
The swing arm has two small holes in it, and the rear fender comes with self-tapping screws, which will work, but is not ideal.
The riv-nuts allow the use of regular machine threaded bolts, and the threads of the riv-nut are alot deeper than just the thickness of the aluminum of the swingarm.
I used M6 riv-nuts, as they fit well in the two positions they needed to be in, and M6 is a good small size, but still big enough to secure the rear fender.
I used a .350 drill bit on the swingarm, threaded the M6 riv-nut on to the tool, the riv-nut pushed snuggly into place, then just squeeze the handle to seat the riv-nut, and unthread the tool.
It's done basically the same as a rivot, only you unthread the tool, as there is no rivot stalk to break off.
These M6's may look a little big in the area that they sit, but they should cause no issues, especially since the riv-nut is compressed against both sides of the aluminum of the swingarm.
Worst case scenario, they could possibly get stress cracks around the holes, but that's an easy fix, as an '03-'04 swingarm on ebay is $30-$40, new bearings cost the same, a few minutes to swap in new bearings, (paint or powdercoat if needed) and make a bracket to mount the rear fender instead, with two riv-nut mounts in front of the center of the tire, on the swingarm.
I don't think any of that will be nesacery, but, it's always good to have a back-up plan, lol