Party Members Help

[homersan]

I am in the House of Samhain. When I walk up to a someone it asks...Do you want to keep (name) in the game., I say yes and now I have following in the game:

Rowain
Baba
Samantha
Raven

When I walk up to Tyber he says something like....yes, better to go in smaller groups. I cannot add him to my party. Are you only allowed 4 persons at a time? Other games I have played have an option to switch players in or out. Am I missing something? According to the guide there are more players I still have to find.

[Indinera]

You can talk to a semi-transparent character (semi-transparent means they are already in your party) and choose to remove them this way.