I would definitely not send a small child to the men's bathroom by himself in a busy public space with many strangers around -- not if I (or another relative / friend) couldn't see him, and there were closed spaces like toilet or changing stalls. That is very different than an open space like a park, where I (and other people) can see what's happening.
Howard, I don't think it's paranoid to say that a 3-6-year-old probably doesn't belong in a crowded Hong Kong men's public toilet by himself. And I'm a pretty relaxed mom.
Are you trying to be sarcastic? No, of course there is no "magic" age. But we all know that young children are less aware when something inappropriate is happening -- a stranger touches them, or leads them away by the hand, or an older boy tries to bully or rob him. Tots are lighter, easier to carry away, weaker, and might not shout for help like an older child. Even if there is nothing as serious as a molester, 3-year-olds have a tendency to decide they're done peeing and wander away.
I come from a laid-back family. But my mom had to call the police once when she lost me in a huge crowd when I was 4, let go of her hand and wandered away into traffic. Yeah, I'd keep a 3-to-6-year-old within eyesight in any crowded public place in the middle of the city, even if it means a small boy in the women's bathroom.
I've never seen anyone complain about this before, in HK or overseas.