It depends on the particular model/make radios you have.
A radio has an audio output. If you simply switch that output to an open circuit (because you have only homemade switching devices)...then the possibility exists that the audio output of your unused radio may be injured and blow it's audio output amplifier. (Some radios may not be able to accept high volume audio output without a load such as a speaker, headphones, or audio panel to accept the load.)
Not all radios require an external audio amplifier, but many do. Without an audio panel (with an integral audio amp) you will not be able to drive a speaker, you'll only be able to drive some, repeat some..., headsets.
In other words, audio panels serve several purposes....(and some of them are regulatory. Example: Does your homemade switching circuitry allow for listening in on NAV/VOR Identification Morse/ID's? No? Then it may not be legal for certain operations.) One purpose is to simply direct audio output to the single speaker/headset you may wish to use to listen. A different purpose is to amplify that audio to a level your speaker/headset can utilise and still deliver acceptable volume. A third purpose is to cushion un-used audio outputs to prevent radio audio circuitry damage. A fourth purpose is to match impedance of differing makes/models of avionics. A fifth purpose is to filter out unwanted noise or audio features (such as when identifying NAV stations. A sixth purpose is to allow reception of TWO radios at the same time...without causing audio feedback....such as listening in on the ATC frequency as well as the FSS frequency during a weather report.
Just because YOUR simple switching panel does not/has not yet caused you operational problems....it does not mean you are not causing stress to your un-switched radio's audio output circuit....and it does not mean your installation is legal. There are other issues involved.
So it's not a simple answer that your question requires. It depends upon WHICH equipment you have installed...and whether or not it has internal audio amps and/or internal circuitry to accept an open audio circuit without damage.
This is not to imply that simple installations cannot be made, and made to work very well,...but most
good simple installations involve at least a modicum of engineering fore-thought and expertise. Not all avionics require external audio amplifiers such as found in audio panels. And some audio panels are nothing more than switching panels with audio dummy-loads to prevent stress to un-switched radios. But few home-made switching units provide that protection (If your particular radios need that.) (And what about those momentary pops/crackles that indicate common switches are allowng momentary open circuits? Not good for your expensive radios.) In any case, just because you've gotten by without an audio panel in the past because your home-made switching panel has seemed to take care of your needs....doesn't mean you're not abusing your radios.
(Unless you only HAVE one radio of course. One of the very reasons I've managed to convince myself I didn't need a second transmitter is in order to avoid having to spend the bucks to also buy and install an audio panel. And MY radios are Narco units.....which don't require external audio amps.)
Anyway, I hope that's informative so you can make a good choice.
