So I am only familiar with Socket.IO in Node but it's fairly easy to scale websockets horizontally with Socket.IO.
Sockets can come with Sessions, so each session is managed by a specific server. This makes it easy to save state for each socket that is open, and load balance across all of your servers.
Here is SocketIO for Python:
Here is a really good read on how to attach sessions to a redis-store to make it even faster and load balancing across servers more manageable.
How to share sessions with Socket.IO 1.x and Express 4.x?
I know this doesn't answer your question about aiohttp, but hopefully this will give you a better idea about how sockets can work.
Edit: Written in Node-
In Socket.IO this is really easy, it has a ton of functions to broadcast messages in a variety of different ways.
For your example if you would like to emit a message across to everyone in each chat-room. Example everyone that has a socket open you can easily just write.
socket.broadcast.emit('WARNING', "this is a test");
Let's say you have rooms open you can broadcast messages only to people in that room with a simple function called .to(). Example I have a room named 'BBQ':
socket.broadcast.to('BBQ').emit('invitation', 'Come get some food!');
This will message everyone in channel BBQ - Come get some food!
This is a fantastic write for how Socket.IO works, make sure you read the second answer for the updated versions of the functions. It is much easier to read than their documentation.
Send response to all clients except sender (Socket.io)
As far as I know this is how it all works in the python implementation as well. For ease of use I would certainly use it for websockets. The aiohttp seems really powerful but either doesn't have this functionality, buried in the documentation, or written only in the code without any documentation yet.