From https://puppet.com/blog/speed-up-ssh-by-reusing-connections:
One way to enable this feature is to add the following to your ~/.ssh/config:
Host * ControlMaster auto ControlPath ~/.ssh/sockets/%r@%h-%p ControlPersist 600
In a quick test with a particular host running ssh user@host whoami
takes about 0.8s without that setting and takes about 0.1s with the setting above. Really speeds up bash-completion with scp.