Sounds like packet loss or

Sounds like packet loss or latency issues. What network card do you have?

Can you go over your network topology and Hypervisor setup?  For instance, you really should never share a network card between the host and its virtuals.