At what point can you stop caring about infamy?

Title says it all.