Semi-Blind Benchmarking Methodology

The Austin Benchmark Suite for Computational Bioelectromagnetics consists of four key ingredients that we consider necessary for a good computational benchmark [2]:

• a precisely defined set of problems that are intended to represent frequently-encountered bioelectromagnetic (BioEM) problems
• clearly defined quantities of interest and reliable reference solutions for them that are relevant to BioEM applications
• error and cost measures that enable performance comparisons
• a database of results that publicize the performance of various methods/tools for solving the problems in the benchmark suite

We adopt a semi-blind benchmarking methodology, which is performed in two stages [2]:

Stage I (Define, initialize, and distribute):
We solve the benchmark problems using several methods, and report the errors, costs, and method descriptions for these initial solutions at this website. Then we distribute the benchmark suite (the problem set, quantities of interest, performance measures, and results/reports for sample methods) and part of the reference solutions to participants.

Stage II (Collect, quantify, and publicize):
We collect solutions for the quantities of interest, descriptions of the methods used to find these solutions, error measurements for the publicly available portion of the reference solutions, and computational cost measurements from the participants. We evaluate the errors in the solicited solutions first relative to the disclosed reference solutions (confirming the integrity of the participant submissions by comparing these results to those submitted) and then relative to the undisclosed reference solutions. We then publish the errors, computational costs, and descriptions of the solution methods.

If you would like to submit an entry to the benchmark, please see how to participate in the benchmark.