You run into issues around AudioContext and resumption etc... it's a PITA to have to handle all those corner cases :(
Having just had to tackle this again for my own startup, I'm reminded about what you would lose by ditching WebRTC - the audio DSP pipeline, transmit side VAD, echo cancellation, noise suppression, NAT traversal maturity, codec integration, browser ubiquity etc.
Had a nice chuckle.