Many backpackers travel alone, not because they have no friends or inherent distaste of other people, but they find solo travel to be much more carefree than if the travel had conducted with others. Traveling by oneself means there is no need to consult others when making decisions on where to go, what to do, where to eat, and where to stay. The travel plan can be executed so much quicker, and changes in the itinerary can be made much more flexibly. The freedom of traveling independently comes from the fact that there is no restriction of others having different opinions, who needs to be persuaded otherwise.
Thought this way, solo travelers travel alone not because they do not care about taking their friends and family members out on an adventure, but because there care about the feelings of the same people. Traveling to new places, for people with different interests, is bound to lead to conflicts. To cover what everyone wants to see and do, there needs to be constant compromises. The only way to avoid having such conflicts is for some people in the traveling party to literally have no opinions about what to do or see. If that is case, it is better if those people do not waste their money traveling in the first place.
Traveling with family members adds a whole new level of complexity into the dynamic. When regular friends are travel companions, the opinions of individual members of the traveling party, most likely, matter equally. There can be (and will be) plenty of debates on what to do on the go. Sometimes the opinions of one person will win, while other times, the opinions of another person will be dominant. If constant conflicts cannot be reconciled, the party can split up, during the travel or after, without too much long-term consequences aside from shakier foundations for future friendship.
But the same is untrue of family members. More often than not, the opinions of some members will always weigh heavier than those of others. These "some members" are likely to be those who initiated the trip, paying for the trip (since family members are not likely to split the bill like friends do), or are older, supposedly more knowledgeable members of the family. Additionally, families may travel for special occasions, and the persons at the center of the special occasions may have the final say on travel plans as the unwritten reward for occasions' celebration.
When travel preferences of some matter more than others, the group dynamics become much more suppressed, certainly not in a good way. If there is any opposition to the preferences that matter, there are not enunciated in ways that incite debate. Instead, for the sake of one person enjoying the trip, everyone else simply plays props to the enjoyment, going along and seemingly enjoying the same thing even if they do not. Even if they started off the trip expecting their own opinions to matter less than those of others, when they actually go on the trip, they will become increasingly unhappy about their opinions not heeded.
Why? Because traveling, at the end of the day, is about surprise. One may plan a trip after researching some sights, restaurants, and hotels on the Internet, in guidebooks, or via advice from others, but when one actually lands at the destination and walk around, one may find things that are much more interesting than what was researched. And if one was content to stick to the plan before starting the trip, when more interesting things are found at the destination, one would no longer be content on just going with the original itinerary.
And that desire to not stick to the plan is where the root cause of conflict within a traveling party lies. The group might have meticulously planned out every minute of every day during the trip, and received explicit approval from the most important people in the party, but once the trip starts, the meticulous plan would no longer be the best. Some sites require more time, while others are just plain boring. Everyone in the party, including the persons that matter the most, will start to have their own opinions about how to "improve" the original travel plan.
Yet, for those changes to be freely made and in ways that are satisfying, the only true way is to not consult and compromise with other people when making the changes. It is by getting rid of other people in the travel party. Traveling, at the end of the day, is about enjoying the destination to the fullest, not about cementing better friendships or family relationships. The latter can be done much more effectively at home, with social gathering and nearby outings of a few hours. When on the road for many days and weeks, conflicts will surely weaken social relationships.