Concept: Cohen's kappa
BACKGROUND: Systematic reviews have been challenged to consider effects on disadvantaged groups. A priori specification of subgroup analyses is recommended to increase the credibility of these analyses. This study aimed to develop and assess inter-rater agreement for an algorithm for systematic review authors to predict whether differences in effect measures are likely for disadvantaged populations relative to advantaged populations (only relative effect measures were addressed). METHODS: A health equity plausibility algorithm was developed using clinimetric methods with three items based on literature review, key informant interviews and methodology studies. The three items dealt with the plausibility of differences in relative effects across sex or socioeconomic status (SES) due to: 1) patient characteristics; 2) intervention delivery (i.e., implementation); and 3) comparators. Thirty-five respondents (consisting of clinicians, methodologists and research users) assessed the likelihood of differences across sex and SES for ten systematic reviews with these questions. We assessed inter-rater reliability using Fleiss multi-rater kappa. RESULTS: The proportion agreement was 66% for patient characteristics (95% confidence interval: 61%-71%), 67% for intervention delivery (95% confidence interval: 62% to 72%) and 55% for the comparator (95% confidence interval: 50% to 60%). Inter-rater kappa, assessed with Fleiss kappa, ranged from 0 to 0.199, representing very low agreement beyond chance. CONCLUSIONS: Users of systematic reviews rated that important differences in relative effects across sex and socioeconomic status were plausible for a range of individual and population-level interventions. However, there was very low inter-rater agreement for these assessments. There is an unmet need for discussion of plausibility of differential effects in systematic reviews. Increased consideration of external validity and applicability to different populations and settings is warranted in systematic reviews to meet this need.
Purpose: the Thai PPS Adult Suandok tool was translated from the Palliative Performance Scale (PPSv2) and had been used in Chiang Mai, Thailand for several years. Aim: to test the reliability and validity of the Thai translation of PPSv2. Design: a set of 22 palliative cases were used to determine a PPS score on Time-1, and repeated two weeks later as Time-2. A survey questionnaire was also completed for qualitative analysis. Participants: a total of 70 nurses and physicians from Maharaj Nakorn Hospital in Chiang Mai participated. Results: The Time-1 intraclass correlation coefficient (ICC) for absolute agreement is 0.911 (95% CI 0.86-0.96) and for consistency is 0.92 (95% CI 0.87-0.96). The Time-2 ICC for agreement is 0.905 (95% CI 0.85-0.95) and for consistency is 0.912 (95% CI 0.86-0.96). These findings indicate good agreement among participants and also were somewhat higher in the Time-2 re-test phase. Cohen’s kappa score is 0.55, demonstrating a moderate agreement. Thematic analysis from the surveys showed that 91% felt PPS to be a valuable clinical tool overall, with it being ‘very useful’ or ‘useful’ in several areas, including care planning (78% and 20%), disease monitoring (69% and 27%) and prognostication (61% and 31%), respectively. Some respondents noted difficulty in determining appropriate scores in paraplegic patients or those with feeding tubes, while others found the instructions long or difficult. Conclusion: the Thai PPS Adult Suandok translated tool has good inter- and intra-rater reliability and can be used regularly for clinical care.
BACKGROUND: Physical activity is assumed to be important in the prevention and treatment of frailty. It is however unclear to what extent frailty can be influenced, because an outcome instrument is lacking. OBJECTIVES: An Evaluative Frailty Index for Physical activity (EFIP) was developed based on the Frailty Index Accumulation of Deficits and clinimetric properties were tested. DESIGN: The content of the EFIP was determined in a written Delphi procedure. Intra-rater reliability, inter-rater reliability, and construct validity were determined in an observational study (n=24) and to determine responsiveness, the EFIP was used in a physical therapy intervention study (n=12). METHOD: Intra-rater reliability and inter-rater reliability were calculated using Cohen’s kappa, construct validity was determined by correlating the score on the EFIP with those on the Timed Up &Go Test (TUG), the Performance Oriented Mobility Assessment (POMA), and the Cumulative Illness Rating Scale for geriatrics (CIRS-G). Responsiveness was calculated by means of the Effect Size (ES), the Standardized Response Mean (SRM), and a paired sample t-test. RESULTS: Fifty items were included in the EFIP. Inter-rater (Cohen’s kappa: 0,72) and intra-rater reliability (Cohen’s kappa: 0,77 and 0,80) were good. A moderate correlation with the TUG, POMA, and CIRS-G was found (0,68 -0,66 and 0,61 respectively, P< 0.001). Responsiveness was moderate to good (ES: -0.72 and SRM:-1.14) for an intervention with a significant effect (P< 0.01). LIMITATIONS: The clinimetric properties of the EFIP have been tested in a small sample and anchor based responsiveness could not be determined. CONCLUSIONS: The EFIP is a reliable, valid, and responsive instrument to evaluate the effect of physical activity on frailty in research and clinical practice.
BACKGROUND: In Canada, new models of orthopaedic care involving advanced practice physiotherapists (APP) are being implemented. In these new models, aimed at improving the efficiency of care for patients with musculoskeletal disorders, APPs diagnose, triage and conservatively treat patients. Formal validation of the efficiency and appropriateness of these emerging models is scarce. The purpose of this study is to assess the diagnostic agreement of an APP compared to orthopaedic surgeons as well as to assess treatment concordance, healthcare resource use, and patient satisfaction in this new model. METHODS: 120 patients presenting for an initial consult for hip or knee complaints in an outpatient orthopedic hospital clinic in Montreal, Canada, were independently assessed by an APP and by one of three participating orthopaedic surgeons. Each health care provider independently diagnosed the patients and provided triage recommendations (conservative or surgical management). Proportion of raw agreement and Cohen’s kappa were used to assess inter-rater agreement for diagnosis, triage, treatment recommendations and imaging tests ordered. Chi-Square tests were done in order to compare the type of conservative treatment recommendations made by the APP and the surgeons and Student t-tests to compare patient satisfaction between the two type of care. RESULTS: The majority of patients assessed were female (54%), mean age was 54.1 years and 91% consulted for a knee complaint. The raw agreement proportion for diagnosis was 88% and diagnostic inter-rater agreement was very high (kappa=0.86; 95%CI: 0.80-0.93). The triage recommendations (conservative or surgical management) raw agreement proportion was found to be 88% and inter-rater agreement for triage recommendation was high (kappa=0.77; 95%CI: 0.65-0.88). No differences were found between providers with respect to imaging tests ordered (p>=0.05). In terms of conservative treatment recommendations made, the APP gave significantly more education and prescribed more NSAIDs, joint injections, exercises and supervised physiotherapy (p<0.05). Patient satisfaction was significantly higher for APP care than for the surgeons care (p<0.05). CONCLUSION: The diagnoses and triage recommendations for patients with hip and knee disorders made by the APP were similar to the orthopaedic surgeons. These results provide evidence supporting the APP model for orthopaedic care.
Clinical evaluation of scapular dyskinesis (SD) aims to identify abnormal scapulothoracic movement, underlying causal factors, and the potential relationship with shoulder symptoms. The literature proposes different methods of dynamic clinical evaluation of SD, but improved reliability and agreement values are needed. The present study aimed to evaluate the intrarater and interrater agreement and reliability of three SD classifications: 1) 4-type classification, 2) Yes/No classification, and 3) scapular dyskinesis test (SDT). Seventy-five young athletes, including 45 men and 30 women, were evaluated. Raters evaluated the SD based on the three methods during one series of 8-10 cycles (at least eight and maximum of ten) of forward flexion and abduction with an external load under the observation of two raters trained to diagnose SD. The evaluation protocol was repeated after 3 h for intrarater analysis. The agreement percentage was calculated by dividing the observed agreement by the total number of observations. Reliability was calculated using Cohen Kappa coefficient, with a 95% confidence interval (CI), defined by Kappa coefficient ±1.96 multiplied by the measurement standard error. The interrater analyses showed an agreement percentage between 80% and 95.9% and an almost perfect reliability (k>0.81) for the three classification methods in all the test conditions, except the 4-type and SDT classification methods, which had substantial reliability (k<0.80) in shoulder abduction. Intrarater analyses showed agreement percentages between 80.7% and 89.3% and substantial reliability (0.67 to 0.81) for both raters in the three classifications. CIs ranged from moderate to almost perfect categories. This indicates that the three SD classification methods investigated in this study showed high reliability values for both intrarater and interrater evaluation throughout a protocol that provided SD evaluation training of raters and included several repetitions of arm movements with external load during a live assessment.
- International journal of environmental research and public health
- Published 12 months ago
Due to the unpredictable, varied and often physical nature of law enforcement duties, police officers are at a high risk of work-related physical injury. The aim of this critical narrative review was to identify and synthesize key findings of studies that have investigated musculoskeletal injuries sustained by law enforcement officers during occupational tasks. A systematic search of four databases using key search terms was conducted to identify potentially relevant studies, which were assessed against key inclusion and exclusion criteria to determine studies to be included in the review. Included studies were critically appraised and the level of evidence determined. Relevant data were extracted, tabulated and synthesized. The 16 identified studies ranged in percentage quality scores from 25.00% to 65.00%, with a mean score of 41.25% and high interrater agreement in scores reflected in a Cohen’s Kappa coefficient, κ = 0.977. The most common body site of injury was the upper extremity, the most common injury types were soft-tissue sprains and strains and the most common cause of injury was a non-compliant offender, often involving assault. However, there was limited peer reviewed research in this area and the published research had a narrow focus and was of low to fair methodological quality.
Occupational self-coding and automatic recording (OSCAR): a novel web-based tool to collect and code lifetime job histories in large population-based studies
- Scandinavian journal of work, environment & health
- Published about 1 year ago
Objectives The standard approach to the assessment of occupational exposures is through the manual collection and coding of job histories. This method is time-consuming and costly and makes it potentially unfeasible to perform high quality analyses on occupational exposures in large population-based studies. Our aim was to develop a novel, efficient web-based tool to collect and code lifetime job histories in the UK Biobank, a population-based cohort of over 500 000 participants. Methods We developed OSCAR (occupations self-coding automatic recording) based on the hierarchical structure of the UK Standard Occupational Classification (SOC) 2000, which allows individuals to collect and automatically code their lifetime job histories via a simple decision-tree model. Participants were asked to find each of their jobs by selecting appropriate job categories until they identified their job title, which was linked to a hidden 4-digit SOC code. For each occupation a job title in free text was also collected to estimate Cohen’s kappa (κ) inter-rater agreement between SOC codes assigned by OSCAR and an expert manual coder. Results OSCAR was administered to 324 653 UK Biobank participants with an existing email address between June and September 2015. Complete 4-digit SOC-coded lifetime job histories were collected for 108 784 participants (response rate: 34%). Agreement between the 4-digit SOC codes assigned by OSCAR and the manual coder for a random sample of 400 job titles was moderately good [κ=0.45, 95% confidence interval (95% CI) 0.42-0.49], and improved when broader job categories were considered (κ=0.64, 95% CI 0.61-0.69 at a 1-digit SOC-code level). Conclusions OSCAR is a novel, efficient, and reasonably reliable web-based tool for collecting and automatically coding lifetime job histories in large population-based studies. Further application in other research projects for external validation purposes is warranted.
Background. A culture of stringent drug policy, one-size-fits-all treatment approaches, and drug-related stigma has clouded clinical HIV practice in the United States. The result is a series of missed opportunities in the HIV care environment. An approach which may address the broken relationship between patient and provider is harm reduction-which removes judgment and operates at the patient’s stage of readiness. Harm reduction is not a routine part of care; rather, it exists outside clinic walls, exacerbating the divide between compassionate, stigma-free services and the medical system. Methods. Qualitative, phenomenological, semi-structured, individual interviews with patients and providers were conducted in three publicly-funded clinics in Chicago, located in areas of high HIV prevalence and drug use and serving African-American patients (N = 38). A deductive thematic analysis guided the process, including: the creation of an index code list, transcription and verification of interviews, manual coding, notation of emerging themes and refinement of code definitions, two more rounds of coding within AtlasTi, calculation of Cohen’s Kappa for interrater reliability, queries of major codes and analysis of additional common themes. Results. Thematic analysis of findings indicated that the majority of patients felt receptive to harm reduction interventions (safer injection counseling, safer stimulant use counseling, overdose prevention information, supply provision) from their provider, and expressed anticipated gratitude for harm reduction information and/or supplies within the HIV care visit, although some were reluctant to talk openly about their drug use. Provider results were mixed, with more receptivity reported by advanced practice nurses, and more barriers cited by physicians. Notable barriers included: role-perceptions, limited time, inadequate training, and the patients themselves. Discussion. Patients are willing to receive harm reduction interventions from their HIV care providers, while provider receptiveness is mixed. The findings reveal critical implications for diffusion of harm reduction into HIV care, including the need to address cited barriers for both patients and providers to ensure feasibility of implementation. Strategies to address these barriers are discussed, and recommendations for further research are also shared.
The aim of this study was to create and evaluate the validity, reliability and feasibility of the Regional Anaesthesia Procedural Skills tool, designed for the assessment of all peripheral and neuraxial blocks using all nerve localisation techniques. The first phase was construction of a 25-item checklist by five regional anaesthesia experts using a Delphi process. This checklist was combined with a global rating scale to create the tool. In the second phase, initial validation by 10 independent anaesthetists using a test-retest methodology was successful (Cohen kappa ≥ 0.70 for inter-rater agreement, scores between test to retest, paired t-test, p > 0.12). In the third phase, 70 clinical videos of trainees were scored by three blinded international assessors. The RAPS tool exhibited face validity (p < 0.026), construct validity (p < 0.001), feasibility (mean time to score < 3.9 min), and overall reliability (intraclass correlation coefficient 0.80 (95% CI 0.67-0.88)). The Regional Anaesthesia Procedural Skills tool used in this study is a valid and reliable assessment tool to score the performance of trainees for regional anaesthesia.
BACKGROUND: Rater agreement is important in clinical research, and Cohen’s Kappa is a widely used method for assessing inter-rater reliability; however, there are well documented statistical problems associated with the measure. In order to assess its utility, we evaluated it against Gwet’s AC1 and compared the results. METHODS: This study was carried out across 67 patients (56% males) aged 18 to 67, with a mean SD of 44.13 +/- 12.68 years. Nine raters (7 psychiatrists, a psychiatry resident and a social worker) participated as interviewers, either for the first or the second interviews, which were held 4 to 6 weeks apart. The interviews were held in order to establish a personality disorder (PD) diagnosis using DSM-IV criteria. Cohen’s Kappa and Gwet’s AC1 were used and the level of agreement between raters was assessed in terms of a simple categorical diagnosis (i.e., the presence or absence of a disorder). Data were also compared with a previous analysis in order to evaluate the effects of trait prevalence. RESULTS: Gwet’s AC1 was shown to have higher inter-rater reliability coefficients for all the PD criteria, ranging from .752 to 1.000, whereas Cohen’s Kappa ranged from 0 to 1.00. Cohen’s Kappa values were high and close to the percentage of agreement when the prevalence was high, whereas Gwet’s AC1 values appeared not to change much with a change in prevalence, but remained close to the percentage of agreement. A Schizoid sample revealed a mean Cohen’s Kappa of .726 and a Gwet’s AC1of .853 , which fell within the different level of agreement according to criteria developed by Landis and Koch, and Altman and Fleiss. CONCLUSIONS: Based on the different formulae used to calculate the level of chance-corrected agreement, Gwet’s AC1 was shown to provide a more stable inter-rater reliability coefficient than Cohen’s Kappa. It was also found to be less affected by prevalence and marginal probability than that of Cohen’s Kappa, and therefore should be considered for use with inter-rater reliability analysis.