@bat01 The Twiddler is an HID device. This means it sends HID codes directly to your host. There are no drivers that are required and running to slow things down.
If it happens during Bluetooth operation, it could be a connectivity issue between the Twiddler and the Host. With USB, there is a direct, high speed connection so there should not be an issue.
With the Twiddler connected via USB and its in the "slow mode", what effect does using a regular keyboard have on the system? Does a keypress have an immediate effect or is it slow as well?
Also, what's your CPU load look like? Is it tapped out?