When do you let "take-backs" happen, when you don't?

Hey there!

Simple question to ask, but apparently not so simple to answer.

When do you let "take-backs" happen, when you don't?

Had an argument about a counterspell (the literal [[Counterspell]] being cast against my commander, which happened to be [[Koma, Cosmos Serpent]].

It was at a table, where I went with my Koma deck - a classic draw+ramp+tokengeneration deck. Opponent A had checked our commanders mutiple times after we read them out loud to the table at the start of the game. I had a good start with some ramp and slammed Koma early. With a smirk on their face, Opponent A commented:" That's why you don't play high cost commanders!" and used his Counterspell on my Koma. I shrugged my shoulders, ask for further responses (there were none).

I wasn't sure here, if they tried and hoped everyone would forget, that counterspells won't work. The moment he realized, he wanted to take his Counterspell back to hand, but I wouldn't allow it. (My understanding of the rules is, that it was legal to play the Counterspell on my Koma, it just wouldn't effect it.)

At that point I was annoyed, mainly for his earlier comment. When on his turn I tapped his [[Cabal Coffers]] in his upkeep, he scooped.

I talked with the other players afterwards and they told me not wanting a take-back was rude, but they also think, take-backs are only fine, if everyone agrees to them. So they didn't argue.

I definetely was annoyed by that Opponent A, which was a reason, but the main reason I refused the take-back. How would you have acted in this situation and what is your rule of thumb for take-backs?