The value of being around other people is dependent upon how receptive you are to accept their ideas and viewpoints. You know this to be true because of the cliques you see in the world who aren’t improving each other but reinforcing each other’s beliefs. Plenty of people stay on their side of the aisle in life, never to have their views challenged, and therefore, they never feel uncomfortable.
You’re as defensive as anybody when you find yourself near people who don’t think you’re correct. Inevitably, you’re not right about everything, so you shouldn’t be uncomfortable with that feeling. However, we need to address what you think the remedies are. What if you became as intelligent and knowledgeable as possible, so you can feel comfortable in all situations because you know you’re never wrong? That’s impossible, and therefore you shouldn’t aim for it.
In that case, what is your remedy? To be humble. In humility, you are never wrong, only learning. By being a willing and able pupil, other people will feel in control of the situation as they lead you along a better path of understanding. Not to mention, most of the time it will be true that you need their leadership. You’re not and never will be an expert on everything—embrace everyone around you as your teacher.