Folder permissions in a document library.

Hi

I want to assign permissions to 3 people on a specific folder in my document library.

Library has 10 other folders and these 3 users should not be able to see other folders.

I have assigned this folder unique permissions by stopping inheriting from parent but these users can also see other folders as they are member of site visitors group.

If i delete them from site visitors group then they can't access my site.

Is that possible if i allow these user to access the site and document library but they should be able to see a specific folder only.?

Secondly I have added a user in my group with contribute permissions, I logged in to the site with that user, then i removed the group but user still can log in to site, confused here how security modal works...