We might fight about the device with polling processes, or other users who probe the device. Retry a few times if the other one goes away in the meantime. Based on a patch from Harald Hoyer.