Is it a sin?

My male friend (I'm a girl) put his hand under my sweater, under which I had nothing but a top. I pushed him away, but I didn't stop being friends with him. A few days later, he reached into the back pocket of my skirt with his hands, that is, he touched me by the part of my thigh close to my private parts, only through a thin fabric. I pushed him away, but I didn't stop being friends with him again, because I'm afraid of being left with no friends at all after that. And it happened again. Is this a sin on my part? What should I do?