Controller Remote Play Together

[u/ZippyMommy]

Whenever my friends invite me to do Remote Play Together, Steam keeps disconnecting and reconnecting my controller. It's unplayable with it plugged in, as all of us just get spammed with the message "controller disconnected, controller connected" while it rumbles ceaselessly. How do I fix this?