Yes, anybody with download rights (which you establish in the user or group permissions) will be able to see whatever data is associated with it. If it's connected to a database like SQL, the user will be required to input credentials that allow them to connect to it. If the data is in the form of an excel document, the only way to get it is to download it.
It appears to me that all you need to do is restrict download access to users, which can be done in the user/group settings. Allow them to "view only."
I hope this information is useful.