The only real difference is that White people are less likely to be deliberately killed by police officers who have long been taught that the easiest solution to any crime is to shoot the nearest Black person. At the same time, White people are trained to be afraid of Black people, because they are criminals. Why are they criminals? Because the police shoot so may of them! Circular reasoning, much?
As to the rest, the White person is just as likely to lose their underpaid, overworked job at a moments' notice, and thus lose his home, not be able to feed his kids, etc. If there's one thing you can say about predatory capitalism it's that it's not racist- - it will happily destroy you whatever your colour! So no, he doesn't and never will forgive you for calling him 'privileged'. You'll just piss him off
What we need to do is to get into peoples' heads something which I've been banging on about for forty years. We're all on the same side, all up to our necks in the same shit-heap, and we all have the same enemies. Those enemies being systems and structures which concentrate too much power in the hands of too few people. If we don't fight them together, they'll destroy us separately, if in different ways.
So a divisive term like "White privilege" is useless and dangerous - even when it's true, from a given perspective. Nor should we be talking about 'White allies', because we should all be in the same army. It's not about You and Us and Them, it's about All of Us and Them.