Follow these steps to set restrictions:
1.Adding users to your VSTS (https://XXX.visualstudio.com/_user) (turn off the feature of Streamlined User Management)
2.On to your Git team project admin page and choose version control (https://XXX.visualstudio.com/[teamproject]/_admin/_versioncontrol)
3.Select a repository
4.On the Security page, Add -> Add user to add members
5. and then Grant these users Contribute and Read permission
View instance-level information (For connection in Visual Studio) and View project-level information are needed too.
you can refer to this article for more info: Permissions