Before my cruiser went into the corner of the garage with a bent shift fork, I used to take my daughter for rides to school, soccer, and just out for fun. We started when she was six (she's 8 now). She could rest her feet on the pegs with enough bend in the knee to be worth a damn, and she was able to hold on well enough and be attentive enough that I had no issue taking her for a ride. What my neighbors thought never factored into my thinking at all. They live next to me and we socialize on occasion. That's the extent of our relationship. My daughter's mother is dead-set against her riding with me, but that's just one of many reasons we aren't together anymore.
I did, however, get pulled over by a State Trooper in a 7-11 lot looking to write me a ticket for something. I took my daughter with me to spray the cruiser off at the car wash, which is only 3 or so miles from the house. We weren't speeding, had all the required PPE, and my papers were in order. He passed us going the other direction as we were pulling out of the car wash, whipped a U turn and came after us with the lights on. He started writing me a ticket for speeding and tore that up when I mentioned we were actually just pulling out of the car wash when he turned around, and then he wrote up a warning instead (for speeding) and told me I shouldn't have my daughter on the bike. I was 35 years old at the time, my daughter 7. I thanked him for his opinion and disagreed respectfully. There is no minimum age on the books as far as I know in my state. That always struck me as bizarre. He was pretty young, probably new on the job. The best I could come up with was just that he thought she was too small, though he had nothing but an opinion to work with.
I don't take my daughter on the Hayabusa. She's not really tall enough to see around me, and I don't want her hanging off in an attempt to do so. We still tool around some on the old '81 CM400 I keep around while I debate fixing/selling the cruiser.