How do I limit the number of phones that connect to my speaker?

I own a JBL Charge 3 Bluetooth Speaker. Aside from using an auxiliary cord to play music from electronic devices, I can connect the speaker to devices such as phones and pcs wirelessly using bluetooth.

The issue I have came up when i was playing music on it with friends. All of a sudden, the music I played stopped abruptly and was overrided with music of a different genre. I looked at my phone and decided to play my music again, and the speaker switched back to playing my music once I pressed the play button on my phone. Minutes later, the music switches back to different music that I did not want to.

What I concluded from this is that multiple phones are capable of connecting to my speaker via bluetooth and each phone can give instructions on what audio to play, which takes control over the speaker from other devices.

What I want to know is that is there any way that I can prevent my speaker from connecting to multiple devices, and instead be connected to only one device at a time? (Something like, maybe, a passcode that phones or pcs have to input to be allowed to connect to the speaker)