The Use of Meta-Analytic Statistical Significance Testing

Meta-analysis multiplicity, the concept of conducting multiple tests of statistical significance within one study, is an underdeveloped literature (Tendal, Nüesch, Higgins, Jüni, & Gøtzsche, 2011). We address this issue by considering how Type I errors can impact meta-analytic results, suggest how statistical power may be affected through the use of multiplicity corrections, and propose how meta-analysts should analyze multiple tests of statistical significance. The context for this study is a meta-review of meta-analyses published in two leading review journals in education and psychology. Our review of 130 meta-analyses revealed a strong reliance on statistical significance testing without considering of Type I errors or the use of multiplicity corrections. In order to provide valid conclusions, meta-analysts must consider these issues prior to conducting the study

Validation of the Employment Hope Scale: Measuring Psychological Self-Sufficiency Among Low-Income Jobseekers

The Employment Hope scale (EHS) was designed to measure the empowerment-based self-sufficiency (SS) outcome among low-income job-seeking clients. This measure captures the psychological SS dimension as opposed to the more commonly used economic SS in workforce development and employment support practice. The study validates the EHS and reports its psychometric properties. Method: An exploratory factor analysis (EFA) was conducted using an agency data from the Cara Program in Chicago, United States. The principal axis factor extraction process was employed to identify the factor structure. Results: EFA resulted in a 13-item two-factor structure with Factor 1 representing “Psychological Empowerment” and Factor 2 representing “Goal-Oriented Pathways.” Both factors had high internal consistency reliability and construct validity. Conclusions: While findings may be preliminary, this study found the EHS to be a reliable and valid measure, demonstrating its utility in assessing psychological SS as an empowerment outcome among low-income jobseekers

Reporting doping in sport: national level athletes' perceptions of their role in doping prevention.

This paper qualitatively explores national level athletes' willingness to report doping in sport. Following ethical approval, semi-structured interviews were conducted with nine national level athletes from rugby league (n = 5) and track and field athletics (n = 4). Thematic analysis established the main themes within the data. Contextual differences existed around the role that athletes perceived they would play if they became aware of doping. Specifically, track and field athletes would adopt the role of a whistle-blower and report individuals who were doping in their sport. In comparison, the rugby league players highlighted a moral dilemma. Despite disagreeing with their teammates' actions, the players would adhere to a code of silence and refrain from reporting doping. Taking these findings into account, prevention programs might focus on changing broader group and community norms around doping. In doing so, community members' receptivity to prevention messages may increase. Moreover, developing skills to intervene (e.g., speaking out against social norms that support doping behavior) or increasing awareness of reporting lines could enhance community responsibility for doping prevention. In sum, the findings highlight the need to consider the context of sport and emphasize that a one-size-fits-all approach to anti-doping is problematic

The Educators of Educators: An Evaluation of Carnegie Foundation\u27s Very High Research Activity Universities\u27 Schools of Education Professors

The field of education is diverse, its history marred with politically driven problems, and its research ill-funded and often disaggregated. Indeed researchers of the past have questioned education research\u27s purpose (Kaestle, 1993) while researchers of the present doubt the field\u27s ability to evaluate properly its current researchers and research (Hedges & Hans-Martin, 2009). Therefore the purpose of this thesis was to investigate and evaluate critically who education researchers were and what type of research they produced. Using a nationally representative, stratified-random sample technique, this author distributed the Education Research Identity Survey (ERIS) electronically via email to 2,723 Schools of Education professors with the Carnegie Foundation\u27s very high research activity universities. Five hundred and forty three individuals participated (raw response rate = 19.9%, final response rate = 22.2%); most received a PhD (81%) in Education (57.1%) and held tenure (57.0%). All academic positions and epistemological backgrounds were represented. Analyses revealed several findings of interest. First, most education researchers prefer nonexperimental quantitative designs or case study qualitative designs. Second, epistemological training was strongly related to the type of research that an individual conducted as a professor. Third, MANOVA analysis revealed that after controlling for the number of years since doctoral matriculation, number of years at his or her university, and the average number of courses taught per semester a professor\u27s academic position and epistemological quantitative training were both related to the amount of journal articles the professor produced in the last five years. Finally, these conclusions follow broader previous research conducted within the field of higher education, but represent some of the first quantitative analysis within the field of education. However, since this author surveyed researchers from only one specific Carnegie type, future research is required to confirm these findings

Using Web-Based Surveys to Evaluate Field Day Open Houses: The Great Tomato Tasting Evaluation

The Great Tomato Tasting event was developed to expose residents to Rutgers New Jersey Agricultural Experiment Station research and Cooperative Extension and to educate them on gardening, local agriculture, and nutrition. More than 7,400 attendees have participated in this event from 2000 to 2011. A Web-based survey of attendees was created to evaluate the impact of this event, replacing previous on-site paper evaluations. The results of the survey implemented in 2010 and 2011 revealed attendees overwhelmingly agreed the event gave them a better understanding of Extension and that they were more likely to take advantage of programs being offered

Effects of Cultural Awareness Training in Conjunction with an Established Bullying Prevention Program

While many school bullying prevention programs have been designed to thwart school bullying perpetration, victimization, and physical aggression, empirical support is limited. To date, few researchers have addressed bullying within the context of children\u27s cultural systems, and cultural awareness training has not yet been utilized as a mechanism to decrease bullying behaviors in the context of school bullying prevention programs. This study examined the utility of incorporating multicultural training aimed at reducing prejudice within the context of school bullying prevention programming efforts within a diverse educational setting. This researcher hypothesized that the addition of meaningful culture curriculum would further reduce reports of bullying perpetration, victimization, and physical aggression. Students in two fifth-grade classrooms at an urban public school were randomly assigned to either the intervention or delayed treatment control group. Students in the intervention participated in Second Step and culture curricula while other students were included in a control activity. Those in the delayed treatment control group received Second Step after the intervention group completed all programming. Student self-report data was collected at 5 unique time points and included experiences of bullying perpetration, victimization, and physical aggression; empathy, self-control, and engagement social skills; sense of school belonging; subjective well-being; and valuing/acceptance of one\u27s own and others\u27 cultures. Hierarchical Liner Modeling was utilized to analyze data. Results of the multilevel model indicated that, following implementation of the Second Step intervention, a significant slope decrease in the amount of bullying perpetration for students in the intervention compared to control students (β16 = -.05, SE = .02, z = -2.23, p \u3c .05). In addition, data illustrated a significant slope increase in levels of perceived self-control for students in the intervention group (β16 = .12, SE = .06, z = 2.10, p \u3c .05). Notably, two significant classroom by condition interactions were found for bullying perpetration (β17 = -.14, SE = .05, z = -2.69, p \u3c .01) and bullying victimization (β17 = -.50, SE = .25, z = -2.00, p \u3c .05). Thus, invention students in classroom 1 reported significantly lower bullying perpetration and victimization compared to intervention students in classroom 2. Further, after the addition of cultural lessons, results showed a significant slope increase for valuing others\u27 cultures (β16 = .18, SE = .08, z = 2.09, p \u3c .05), acceptance of others\u27 cultures (β16 = .13, SE = .06, z = 1.89, p \u3c .05), and perceived self-control (β16 = .08, SE = .03, z = 2.78, p \u3c .01). Two significant classroom by condition interactions existed for physical aggression (β17 = .18, SE = .06, z = 2.93, p \u3c .01) and subjective well-being (β17 = -.40, SE = .17, z = -2.41, p \u3c .05) wherein intervention students in classroom 1 reported significantly lower physical aggression and higher levels of subjective well-being compared to intervention students in classroom 2. Research findings indicate the viability of incorporating multicultural training with established bullying prevention efforts in order to increase valuing and acceptance of others\u27 cultures while ultimately enhancing students\u27 perceived self-control and reducing bullying acts. Furthermore, these results illuminate the necessity of focusing special attention on classroom variables that may significantly affect the success of bullying prevention programs with an urban, primary educational setting

Addressing the Issue of Meta-Analysis Multiplicity in Education and Psychology

The concept of multiplicity, conducting multiple statistical significance tests in one study, has pervaded primary research over the last 7 decades (Hochberg & Tamhane, 1987). This continued discussion was due to the fact that multiplicity increases the probability of committing a Type 1 error (i.e., deriving a false conclusion). Little attention has been paid, unfortunately, to multiplicity in meta-analysis (Tendal, Nuesch, Higgins, Juni, & Gotzsche, 2011) and calls have been made for meta-analysis methodologists to address this critical issue (Bender et al., 2008). As such, the purpose and significance of this project was to answer these calls by formally quantifying the multiplicity of statistical test in meta-analyses published within education and psychology literature, and to ameliorate the problem of multiplicity errors through the advancement of Type 1 error corrections for meta-analyses. To accomplish this goal, this project screened all citations from Psychological Bulletin and Review of Educational Research from 1986-2011 for quantitative meta-analyses. From the citations that met inclusion criteria, 130 articles were randomly selected to code. The results revealed an alarmingly high number of statistical tests used per study (μ = 70.82, σ = 94.2, M = 46.5). A major contributor to the number of statistical tests utilized was the number of independent syntheses; the average study conducted 12.72 independent syntheses (σ = 21.26, M = 5.0). A multiple regression model predicting the number of statistical tests used per study found that the date of publication, number of studies included in the review, and the number of independent syntheses per review all were linear predictors. A second phase of the project purposively selected four reviews to investigate the potential use of Type 1 error corrections. The results provided by the review authors were compared to the results using the statistical corrections. Using the statistical corrections, an average of 3.33 conclusions would require modification. This project\u27s results indicated a community of researchers becoming more reliant on statistical significance testing while simultaneously ignoring the consequences of multiplicity. Failure to prevent further reliance on statistical significance testing in meta-analysis has the potential to prorogate the progress of cumulative science