You have the right to be assertive. Other people have the right to be assertive too. Being unassertive will cause you more stress in the long-term. Being unassertive will reduce your feelings of self-worth. Being respected and respecting yourself is more important than being liked. Give a reason for your assertive response if you feel it is appropriate, but never an excuse or an apology.
With practice assertiveness will become second nature to you.