This bug was partly fixed in Patch 1.90.375. It looks like it’s still occurring for festivals.
I only get a small amount of Sims to choose from when trying to travel to another lot. Everyone is greyed out.
The selection screen only shows only the Sims that are already in the neighborhood. As a bonus you get Sims you don’t even know.
The bug affects travelling via phone or if you click on a lot and try to travel there this way.
It works if you travel through the map instead of the specific neighborhood.
To do so press the ‘M’ button on your keyboard and pick a lot to travel to.
You should now have the Sims you know as an option to travel.
It can happen that Sims from your household tag along when using this method. Just send them home if they do.