When to apologize...

How many people have to be "offended" by a remark to get an "apology"?
If you are around 1,000 people and you say something that offends very few of the people (like 50-100) do you apologize even though the huge majority agree with you?

Personally I think apologies are ruining society.
Most of the time, the offended party only says they are offended because they can say it and not because they are offended. Then the original commentator to apologizes and instead of making progress we go back to square one.

It's like the use of the term "Racist!" because after awhile, the apology becomes moot. What i mean is, it's not really an apology but just something to say so that people get off your back.