multiplex connection terminated [linux][answer]

For those of you out there who have seen and/or reported the error "multiplex connection terminated" I may have an answer for what causes the behavior, at least for a linux implementation. It should be equally applicable for Windows, assuming you are using a load balancing application.

The following table is from ipvs:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  cqvip:5678 rr persistent 50
  -> cq01:5678                    Route   1      0          0
  -> cq02:5678                    Route   1      0          0

IPVS, in order to ensure a destination system is up, will create an open-air connection to each host:port periodically. If it is unable to connect, that system is marked offline and is removed from the routing table you see above.

This creates, as dgate sees it, hits for service. It will log what it can but, since it is simply a tcp connectivity validation, nothing makes it up to application and therefor nothing meaningful logs.

Hope this helps someone out there!