SciCombinator

Discover the most talked about and latest scientific content & concepts.

Concept: Benchmark

168

BACKGROUND: Secondary use of large scale administrative data is increasingly popular in health services and clinical research, where a user-friendly tool for data management is in great demand. MapReduce technology such as Hadoop is a promising tool for this purpose, though its use has been limited by the lack of user-friendly functions for transforming large scale data into wide table format, where each subject is represented by one row, for use in health services and clinical research. Since the original specification of Pig provides very few functions for column field management, we have developed a novel system called GroupFilterFormat to handle the definition of field and data content based on a Pig Latin script. We have also developed, as an open-source project, several user-defined functions to transform the table format using GroupFilterFormat and to deal with processing that considers date conditions. RESULTS: Having prepared dummy discharge summary data for 2.3 million inpatients and medical activity log data for 950 million events, we used the Elastic Compute Cloud environment provided by Amazon Inc. to execute processing speed and scaling benchmarks. In the speed benchmark test, the response time was significantly reduced and a linear relationship was observed between the quantity of data and processing time in both a small and a very large dataset. The scaling benchmark test showed clear scalability. In our system, doubling the number of nodes resulted in a 47% decrease in processing time. CONCLUSIONS: Our newly developed system is widely accessible as an open resource. This system is very simple and easy to use for researchers who are accustomed to using declarative command syntax for commercial statistical software and Structured Query Language. Although our system needs further sophistication to allow more flexibility in scripts and to improve efficiency in data processing, it shows promise in facilitating the application of MapReduce technology to efficient data processing with large scale administrative data in health services and clinical research.

Concepts: Medicine, Clinical trial, Data, Hadoop, Amazon.com, Benchmark, Amazon Elastic Compute Cloud

62

In recent years, wide deployment of automatic face recognition systems has been accompanied by substantial gains in algorithm performance. However, benchmarking tests designed to evaluate these systems do not account for the errors of human operators, who are often an integral part of face recognition solutions in forensic and security settings. This causes a mismatch between evaluation tests and operational accuracy. We address this by measuring user performance in a face recognition system used to screen passport applications for identity fraud. Experiment 1 measured target detection accuracy in algorithm-generated ‘candidate lists’ selected from a large database of passport images. Accuracy was notably poorer than in previous studies of unfamiliar face matching: participants made over 50% errors for adult target faces, and over 60% when matching images of children. Experiment 2 then compared performance of student participants to trained passport officers-who use the system in their daily work-and found equivalent performance in these groups. Encouragingly, a group of highly trained and experienced “facial examiners” outperformed these groups by 20 percentage points. We conclude that human performance curtails accuracy of face recognition systems-potentially reducing benchmark estimates by 50% in operational settings. Mere practise does not attenuate these limits, but superior performance of trained examiners suggests that recruitment and selection of human operators, in combination with effective training and mentorship, can improve the operational accuracy of face recognition systems.

Concepts: Measurement, Test method, Face perception, Face, Faces, Benchmark, Facial recognition system, Identity document

27

There has been a widespread world-wide use of flathead mullet, Mugilcephalus, in fish biomonitor studies within the coastal zone. This review summarises this research field, focusing on heavy metals, and considers the implications of the accumulated data. Differences in sampling methodology, tissues analysed and units of reported data provide challenges in assessing and benchmarking these biomonitor studies. The benthic feeding strategy of M.cephalus invariably increases exposure risk relative to middle or upper water column feeders, nevertheless contaminant accumulation via direct and indirect pathways was regulated sufficiently such that toxicants were below food guidelines in most coastal regions (32 of the 49 examined). Human health issues can arise if fish are consumed from heavily industrialised regions. Recommendations are provided for future biomonitoring studies, based on the results for M. cephalus but relevant for fish species more broadly, to provide more comparable data so that managers can benchmark against local conditions.

Concepts: Heavy metal music, Heavy metal, Mugilidae, Benchmark, Benchmarking, Flathead mullet

23

To utilize functional status (FS) outcomes to benchmark outpatient therapy clinics.

Concepts: Benchmark, Benchmarking

17

Organic mixed conductors have garnered significant attention in applications from bioelectronics to energy storage/generation. Their implementation in organic transistors has led to enhanced biosensing, neuromorphic function, and specialized circuits. While a narrow class of conducting polymers continues to excel in these new applications, materials design efforts have accelerated as researchers target new functionality, processability, and improved performance/stability. Materials for organic electrochemical transistors (OECTs) require both efficient electronic transport and facile ion injection in order to sustain high capacity. In this work, we show that the product of the electronic mobility and volumetric charge storage capacity (µC*) is the materials/system figure of merit; we use this framework to benchmark and compare the steady-state OECT performance of ten previously reported materials. This product can be independently verified and decoupled to guide materials design and processing. OECTs can therefore be used as a tool for understanding and designing new organic mixed conductors.

Concepts: Function, Chemistry, Thermodynamics, Plastic, Design, Organic semiconductor, Benchmark, Benchmarking

15

Since initial reports regarding the impact of motion artifact on measures of functional connectivity, there has been a proliferation of participant-level confound regression methods to limit its impact. However, many of the most commonly used techniques have not been systematically evaluated using a broad range of outcome measures. Here, we provide a systematic evaluation of 14 participant-level confound regression methods in 393 young adults. Specifically, we compare methods according to four benchmarks, including the residual relationship between motion and connectivity, distance-dependent effects of motion on connectivity, network identifiability, and additional degrees of freedom lost in confound regression. Our results delineate two clear trade-offs among methods. First, methods that include global signal regression minimize the relationship between connectivity and motion, but unmask distance-dependent artifact. In contrast, censoring methods mitigate both motion artifact and distance-dependence, but use additional degrees of freedom. Importantly, less effective de-noising methods are also unable to identify modular network structure in the connectome. Taken together, these results emphasize the heterogeneous efficacy of proposed methods, and suggest that different confound regression strategies may be appropriate in the context of specific scientific goals.

Concepts: Regression analysis, Linear regression, Evaluation, Effectiveness, Statistical terminology, Degrees of freedom, Errors and residuals in statistics, Benchmark

11

The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources, but their organization and adoption as standards have been inconsistent. As such, selecting and curating specific benchmarks remains an unnecessary burden on machine learning practitioners and data scientists.

Concepts: Natural selection, Data, Machine learning, Learning, The Target, Selection, Benchmark

7

To pilot benchmark measures of health information and communication technology (ICT) availability and use to facilitate cross-country learning.

Concepts: Health care, Information technology, Benchmark, Benchmarking

5

Our previously presented method for high throughput computational screening of mutant activity (Hediger et al., 2012) is benchmarked against experimentally measured amidase activity for 22 mutants of Candida antarctica lipase B (CalB). Using an appropriate cutoff criterion for the computed barriers, the qualitative activity of 15 out of 22 mutants is correctly predicted. The method identifies four of the six most active mutants with ≥3-fold wild type activity and seven out of the eight least active mutants with ≤0.5-fold wild type activity. The method is further used to screen all sterically possible (386) double-, triple- and quadruple-mutants constructed from the most active single mutants. Based on the benchmark test at least 20 new promising mutants are identified.

Concepts: Scientific method, Enzyme, Test method, Enzymes, Hebrew numerals, Benchmark

3

We present a continuous benchmarking approach for the assessment of RNA secondary structure prediction methods implemented in the CompaRNA web server. As of 3 October 2012, the performance of 28 single-sequence and 13 comparative methods has been evaluated on RNA sequences/structures released weekly by the Protein Data Bank. We also provide a static benchmark generated on RNA 2D structures derived from the RNAstrand database. Benchmarks on both data sets offer insight into the relative performance of RNA secondary structure prediction methods on RNAs of different size and with respect to different types of structure. According to our tests, on the average, the most accurate predictions obtained by a comparative approach are generated by CentroidAlifold, MXScarna, RNAalifold and TurboFold. On the average, the most accurate predictions obtained by single-sequence analyses are generated by CentroidFold, ContextFold and IPknot. The best comparative methods typically outperform the best single-sequence methods if an alignment of homologous RNA sequences is available. This article presents the results of our benchmarks as of 3 October 2012, whereas the rankings presented online are continuously updated. We will gladly include new prediction methods and new measures of accuracy in the new editions of CompaRNA benchmarks.

Concepts: Scientific method, Protein, Protein structure, Bioinformatics, Amino acid, RNA, Secondary structure, Benchmark