After 83539eb7 it was a bit complicated.  It allowed to save a memcpy
of the address in case no port was specified, but we only need to do the
split when changing configuration, so it was overzealous optimization.