I'm all in favor of being patient except when people are taking advantage of your good nature. THEN it becomes time to do something about it. Remember, some people do not know what it's like to be a real adult and until someone else, ergo, YOU, set the boundaries for them, they will push you like a little teenage kid!
I never thought you were afraid necessarily and I understand about not wanting to make too many waves in the neighborhood. Still, I wouldn't allow it any long than a moment or two.
Try the beer thing as soon as possible. You can't really avoid the guy for long. You'll have to monopolize the conversation and get your word in is all.
Good luck, man!
--Wag--