[Detractors] are just wrong, and that's okay. They just don't see it yet. That's what I would tell myself to keep those moments of doubt, only moments.
In most pursuits, one's self is one of the biggest hurdles to get over. You can't pursue something and be committed to it if you're apologizing for it at every party.
I think it's always a good move to listen to that inner voice, if it doesn't lead to a crime.
You have to surrender to the fact that you are of too many in a highly competitive field where it is difficult to stand out. Over time, through your work, you will demonstrate who you are and what you bring to the field. Just stay with it and keep working.