How we tested
To test the efficacy of the devices we developed the following methodology.
First, two files with different keyword contents (to make it easier to confirm search results) are copied across, and then one of them deleted. We then run the following tests:
• Secure delete test. While logged in with the password we raw-read the disk using disk-analytical software to detect the presence of both the live file and the deleted one -- if the contents of the deleted file are found, the device doesn't pass this test. This is designed to see what happens if someone gets access to your key and knows the password, to see if they can recover deleted information. Ideally, a device would wipe the affected areas when deleting to prevent this (or more likely, fill the file with 0s or random junk then delete it, to negate any shenanigans with Windows' Recycle Bin).
• Secure wipe test. This tests what happens when the password is changed, such as when the device is given to another to use or if a lost/stolen device has the incorrect password entered too many times (if this feature is supported). After resetting the device like this, we search for the contents of both files raw-reading the device sector by sector. A good result here is a complete and unrecoverable wipe of the encrypted area, ensuring that no data can be recovered.
• Readable without password. This is a simple test to see if the device can be read when plugged into another machine (in this case a Linux box, with access to its extensive disk tools) and without access to the password. In most cases secure USB drives don't even show up if the password hasn't been entered (technically they can appear to the OS as a device with no media, much like a DVD drive with no disk inserted, which prevents attempts at even raw-access to the device). If the drive is readable, we check whether encrypted files can be read and if so the drive fails this test -- as encrypted files, once copied off, can be susceptible to brute-force attacks.
• Transfer speeds. Encryption can be CPU intensive, and while hardware-based devices do this on-board this doesn't necessarily mean it will be fast. While there are numerous tools to measure disk throughput, we settled on the sequential read and write results from an average of three runs of 64-bit Crystal DiskMark on our 64-bit Windows 7 test bench for a comparable score. Again, these are results when encryption is active and aren't necessarily comparative with USB drives when not using encryption.
• Auto-destruct. Whether the device initiates a forced secure wipe if the incorrect password is entered too many times. This is a feature that may be over the top for some users (don't go entering passwords if you're drunk!), but for some organisations this level of prevention may also be considered a necessity. It's worth noting that devices can provide secure wipe functionality (see above) while not necessarily providing an auto-destruct function as well.