written by ff123
Last Updated 6 May 2004 (Version 1.0)
This is a win32 application written to allow for blind comparion of multiple audio files. The program has been placed under the GNU Lesser General Public License. The source and the binaries can be downloaded from the links below.
Source, version 1.0 stable 387 kB (Ready to compile using Microsoft Visual C++ 6.0)
Binary, version 1.0 stable 182 kB (win32)
Binary, version 1.1 beta 2 189 kB (win32): Splits ABX into normal and practice modes. Results not displayed until the end when using normal mode. Practice mode displays in-progress ABX results but does not save them to file. Program now includes options for setting the type II error risk as well as the type I error risk, with presets defined for typical codec comparison scenarios. If ABX is successful in normal mode, ABC/HR will automatically unhide the reference.
Releases and Tools by other authors
Advantages of ABC/HR-Java (written by schnofler):
Disadvantages of ABC/HR-Java:
ABC/HR-Java, written by schnofler (source code available)
Java 2 Platform, Standard Edition, download page (needed to run ABC/HR-Java)
ABC/HR for GNU/Linux, written by Carsten Haese
Chunky, ABC/HR result parser, by phong (Thomas Schumm)
Here is a screenshot (reduced scale) of the application's main dialog box:
The listener first fills out a setup dialog box or loads a configuration file, a sample of which has been provided in the download.
Each group (there are eight groups) consists of two files: the reference file and a sample. The listener doesn't know which is which because the program has shuffled them within the group. The groups themselves have been shuffled, so that the listener doesn't know which sample has been assigned to which group.
For each group, the listener listens to both the left and right sides, and to the reference (labeled and colored blue). Then he rates what he thinks is the sample. Once he has moved the slider, he is allowed to write a comment describing the degradation he heard. The bar at the bottom allows the listener to choose a portion of the file to listen to.
After rating all files in the test, the listener saves the results to file. The program reveals which sample was assigned to which group inside the results file.
Here is a screenshot of the ABX dialog box:
The ABX mode is meant to allow the listener to identify subtle differences between any two samples. Typically, one of the samples will be the original file. One of the samples is assigned to be 'A' and the other is assigned to be 'B'. Either sample A or Sample B is randomly assigned to be 'X'. The listener must determine whether X is the same as A or B. By performing this test multiple times, the listener can show with a specified level of confidence that his results are not by chance alone.
Known issues:
I've encountered one WAV file which reported an incorrect position for the end of file, causing abchr to misbehave. Editing the file within a sound editor by snipping out a small section of the beginning or ending (using Cool Edit, in this case) and then resaving fixed the WAV file (probably in the header section). A more robust version of abchr should probably check for mangled WAV headers.
Update: Older versions of an AAC decoder, FAAD (dated before about the early part of 2003) had a bug which produced the WAV files described above. It has since been fixed in more recent versions of FAAD2.
Future enhancements:
Other suggestions made, but which probably won't happen in the near future (if at all):
Other pages:
Time Offsets for Various Codecs
References:
1. Sensory Evaluation Techniques, 3rd Edition, by Meilgaard, Civille, and Carr, 1999, CRC Press. This is an indispensible text for the theory and methods of subjective evaluation. The test method used in this application is most closely described in this book as a "duo-trio test in constant reference mode" (p. 68). ABX is a "duo-trio test in balanced reference mode."
2. ITU-R BS.1116-1, This Recommendation is intended for use in the assessment of systems which introduce impairments so small as to be undetectable without rigorous control of the experimental conditions and appropriate statistical analysis. The preferred test method described in this paper is called the "double-blind triple-stimulus with hidden reference" and is similar to the "duo-trio test in constant reference mode" described in Sensory Evaluation Techniques, except with the addition of a grading scale. The ABC/HR application follows the test method and impairment scale recommended by this paper.
3. EBU Listening Tests on Internet Audio Codecs, Application software called "MUSHRA" is described, allowing comparison of multiple codecs. MUSHRA does not force the listener to choose between each sample and the reference because it is designed to test medium to large impairments. The ABC/HR application retains MUSHRA's side-by-side layout while adding a hidden reference for each sample. With proper test setup, ABC/HR can be used to test both large and small impairments.
4. PC-ABX is Arny Krueger's application for testing differences (small impairments); as such it does not include a grading mechanism. Performs multiple trials and statistical analysis of "duo-trio tests in balanced reference mode." With the addition of an ABX mode into the ABC/HR application, very subtle differences between two audio files can be verified for an individual listener by employing multiple trials.
Version History:
1.0, May 6, 2004: fixes bug in which general comments are not cleared in between tests. Added option to attach nickname to the results file.
0.9b, Aug 30, 2002: changed the random number generator. Previously used rand(). One sequence of ABX runs sent to me by a listener showed moderate evidence against randomness when put through a runs test.
0.9a, Aug 23, 2002: bug fix: setup menu was setting all the time offsets to 0. Negative offsets are disallowed from the setup menu. Testname is now put into the results file.
0.9, Aug 21, 2002: fixes bug when changing the portion select box while performing quick switching. Hopefully cures simultaneous playing of samples (not reproduceable on my machine). The portion select box now has a double-click feature (selects the entire sample). Comment boxes now do not steal the focus (they are modeless) and they have word-wrapping. A general comments box has been added. Now able to set up a test from within the program instead of from a configuration file. The program name and version is now added to the top of the results file.
0.8e, July 22, 2002: bug fix for the bug fix -- Aargh! Fortunately the error was not fatal, and only affected the listing of the answers at the top of the results file (would always list just the right side of each group).
0.8d, July 17, 2002: bug fix -- Fixed problem in which program would incorrectly pay attention to the case of the filenames. Only manifests itself when the same file is listed more than once, but with different cases
0.8c, July 16, 2002: bug fix -- some ABX comparisons were not being written to file
0.8b, July 14, 2002: Now write ABX results to file
0.8a, July 13, 2002: Added ABX probability calculator
0.8, July 12, 2002: Added ABX capability; probability calculator, status window, and logging to results file not added yet
0.7b, June 12, 2002: bug fix -- cursor was changing in areas not inside the file portion selection box.
0.7a, June 12, 2002: Added ability to tweak file portion selection.
0.7, May 31, 2002: Able to correct for time misalignment by specifying a correction in the configuration file.
0.6b, May 30, 2002: bug fix -- program was not resetting all values properly when loading another configuration file.
0.6, May 26, 2002: Allows user to select a portion of the files to listen to. Left click and drag in the rectangle at the bottom to select part of the file to listen to (default is to play the entire file).
0.5c, May 23, 2002: bug fix -- program was failing to report some file-open errors.
0.5b, May 22, 2002: Program now prompts user to save results file when exiting or when reloading configuration file.
0.5a, May 20, 2002: Results file now displays the correct answers even if the user rates all files 5.0. Reset sliders if configuration file is reloaded.
0.5, May 20, 2002: Added ability to switch between the different files without having to start back at the beginning ("fast switching"). When the user presses "play" of any sample while the music is playing, the program will fast-switch. If the user presses "stop" or if the sample finishes, the music will start back at the beginning.
0.4b, May 11, 2002: Pressing the tab key chooses a different group. Pressing the arrow keys chooses the play buttons or stop. The sliders and comments button must be chosen by mouse.
0.4a, May 10, 2002: Initial release