If you touch them without permission. As in, they tell you "Dont touch me" and you do so anyway or they're clearly on edge
Your body is your body. No one has the right to touch you without your permission. and we're talking within reason. If its like a tap on the shoulder for harmless purposes like catching your attention, obviously no one's face is going to get smashed up.
It doesn't make you bad company for wanting to have your own personal and respected space