You know I'm not being mean, but it may just be that you have, in the past. given the impression of being someone who can be blamed because they can't or won't fight back.
I was the same, when young, if only because my mother taught me that everything I did was wrong and deserved a beating. So things were pushed off onto me.
So eventually I went away, and came back somebody who would still not bark back. I bit instead! The accusation and blame melted away.
I sincerely hope you never become the nasty piece of work I did, but I do think you need to develop a bit of a mean streak!
That said, it's quite generally useful to assume the worst of folk, because in nine cases out of ten, you'll be right! That's my exerience, anyway. As the saying goes "punish your kids every day - you may not know what they've done, but they will!".