Symposium, Paper and Poster Sessions

Symposium and Paper Presentations

Evaluation-Related Experiences of Graduates of the University of Hawaiʻi at Mānoa’s Educational Psychology Doctoral Program [symposium]

Morris Lai, Curriculum Research & Development Group, University of Hawaiʻi at Mānoa; Linda Hufano, Director of Alakaʻi Na Keiki; Alice Kawakami, Professor, College of Education, University of Hawaiʻi; Gerald Knezek, Professor, University of North Texas; Carol Pang, Retired System Director of Human Resources, UH Community Colleges;Xiang (Bo) Wang, Vice President, Relations Development, Asia, The College Board

Many University of Hawaiʻi UH) Educational Psychology PhD graduates are doing diverse evaluation-related work throughout the nation and the world. On the eve of the 45th anniversary of the Educational Psychology program at UH, five graduates from the UH Educational Psychology doctoral program will describe the diverse evaluation-related experiences that they have had since earning their PhDs. The areas they will address include behavioral health, College Board tests, measuring dispositions in science, technology, engineering, and mathematics, decolonizing evaluation methods, and assessment in career and technical education.

Utilization-focused evaluation: What does it mean to stakeholders?

Lehua Choy, MPH, Healthy Hawaii Initiative Evaluation Team, Office of Public Health Studies, University of Hawaiʻi at Mānoa; Kathryn Braun, DrPH, professor, Office of Public Health Studies, John A. Burns School of Medicine, University of Hawaiʻi at Mānoa

Patton has encouraged utilization-focused evaluation, but in what ways do stakeholders actually want to be involved? The purpose of this study was to explore how an external evaluation team could better engage program staff in evaluation and attend to use. A brief online survey was developed from Patton’s framework and administered to Department of Health staff, asking about preferred ways of being involved, desired uses for evaluation, and best ways to present evaluation findings. The results were then shared with the program staff and evaluation team to discuss how evaluation practices could accommodate an increased focus on utilization.

Length of Stay for Youth in Mental Health Services: Developing Evidence-Based Methods at the Child and Adolescent Mental Division (CAMHD)

David Jackson, PhD, Evaluation Specialist, Hawaiʻi Dept. of Health/Child and Adolescent Mental Health Division; Scott Keir, PhD, Research & Evaluation Specialist, Hawai'i Dept. of Health/Child and Adolescent Mental Health Division;Jarrett Ku, Graduate Assistant, University of Hawaiʻi at Mānoa [note: David Jackson will present]

This presentation will focus on the current efforts of the Child and Adolescent Mental Health Division (CAMHD) to develop a more research-based method for determining appropriate lengths of stay (LOS) for the various levels of mental health services available to youth in Hawai'i. Length of stay for youth varies by levels of care, but there has been little research to recommend lengths of stay based on the youth’s improvement trajectory. CAMHD has been developing an evidence-based method of determining LOS standards. We hope to elicit ideas from the audience regarding improvements/refinements of our research methods, including statistical techniques for longitudinal data and children’s mental health implications.

Utilization of Observation, Rubric Measures, and Mixed Methods Program Evaluation in the Kohala Center HI-MOES Program

Ryan Tolman, Graduate Research Assistant, Office for Evaluation and Needs Assessment Services of the Social Science Research Institute at the University of Hawaiʻi at Mānoa

In the 2010-2011 academic year, The Kohala Center implemented the Hawaiʻi Island Meaningful Outdoor Experiences for Students (HI-MOES) program to initiate meaningful science-based outdoor learning opportunities. The HI-MOES program consisted of multiple components, activities, and target audiences. The program utilized several different types of measures, rubrics, and observation to measure program progress and the achievement of goals and objectives. This paper addresses evaluation methods and practices in the use of rubrics, the utilization of observation, and mixed-method approaches that can be applied to similar educational program evaluations.

Poster Session and Networking Ice-Cream Social

Evaluation Capacity-Building: Using Evaluation Results for Program Improvement

Linda Toms Barker, Berkeley Policy Associates; Nada Rayyes, Senior Analyst; Paola Rochabrun Oliveira, Analyst

Too often programs conduct minimal evaluations simply to meet grant funding requirements, with little or no understanding of how to use evaluation results. In building the capacity of 21st Century Community Learning Centers grantees to conduct effective program evaluation, a key component is to use evaluation to support program improvement. Only when programs can actually use their evaluation results, do they invest in effective, meaningful program evaluation. Not only does this involve identifying appropriate outcome measures to identify "what" needs to be improved, but also identifying the qualitative measures needed to identify "how" to improve the program to increase performance.

Infrastructure development in translational research to eliminate health disparities: The Role of Evaluation in the RMATRIX Project

Susana Helm, Assistant Professor, Department of Psychiatry-Research Division, University of Hawaiʻi at Mānoa;Nancy Marker, Academic Support, Social Science Research Institute, University of Hawaiʻi at Mānoa; Stephanie Nishimura, Assistant Professor, Psychiatry, University of Hawaiʻi at Mānoa; Judith Inazu, Associate Director, Social Science Research Institute, University of Hawaiʻi at Mānoa

The RMATRIX project is an NIH infrastructure development grant to the John A. Burns School of Medicine for translational research to eliminate health disparities. The summative evaluation assesses the extent to which short- and long-term RMATRIX goals have been met, while the formative evaluation provides frequent feedback to researchers and administrators for continuous program improvement. This poster will highlight the evaluation design; data collection and analysis tools; findings from the first year of this three year project; and implications for program development across key function areas of the RMATRIX project.

Evaluation of a Community of Practice Model for Faculty Development—C4WARD 

Yao Zhang Hill, PhD, Institutional Analyst, Office for Institutional Effectiveness, Kapiʻolani Community College

Collaborative Circle for Creative Change, or C4WARD has become the main model of professional development at Kapiʻolani Community College since spring 2011. The model is based on the principle of community of practice where faculty of similar interest or goals learn and collaborate together under the coordination of a C4WARD facilitator, called concierge, to achieve their professional development goals. This poster presents the evaluation model of the C4WARD. Data are to be triangulated to evaluate the impact of the concierge’s training, the impact on the faculty professional practice, and ultimately the impact on students’ engagement, learning, and institutional progress.

Healthy Hawai'i initiative model schools 2013: A three-year evaluation of health, nutrition, and physical education in Hawaii public schools 

Rebekah Rodericks, Junior Specialist, Office of Public Health Studies, University of Hawaiʻi; Stephanie Lee, Graduate Research Assistant, University of Hawaiʻi; Cathy Ross, School Health Coordinator, Hawaiʻi State Department of Health; Jennifer Ryan, School Health Specialist, Hawaiʻi State Department of Health; Ann Horiuchi, Educational Specialist, Hawaiʻi State Department of Education; Dr. Jay Maddock, Director of Office of Public Health Studies, University of Hawaiʻi [Note: Rebekah Rodericks and Stephanie Lee will present.]

The Healthy Hawaiʻi Initiative (HHI), funded by the Hawaiʻi State Department of Health, is working in collaboration with the Hawaii State Department of Education to develop a three-year Model Schools program. Nine schools have been selected to provide exemplary education in the content areas of health, nutrition, and physical education. The evaluation methods involve documenting program delivery and assessing the program’s effectiveness at generating positive changes in healthy behaviors. This poster will present the program’s overall methodology and evaluation instruments, which include student and teacher surveys, classroom observations tools, fitness assessment and school performance data.

Evaluating Long-Term Academic Performances of At-Risk Students: An Alternative to Longitudinal Analysis

Malkeet Singh, Pacific Resources for Education and Learning

Evaluating the long-term academic performance of at-risk groups is crucial in understanding how well an educational initiative has worked, especially in closing achievement gaps. Since Native Hawaiian students are the most at-risk group in Hawaii, an effort was made in evaluating their academic performance from the third grade up to the tenth grade across reading and math. A challenge in conducting any longitudinal study is whether the assessment allows for longitudinal analysis. In this poster, an alternative to longitudinal analysis is developed to study the long-term academic performance of Native Hawaiian students in comparison to their White counterparts.

Hawai'i Race to the Top Project: The Evaluation Framework 

Jerry Wang, Analyst, System Planning and Improvement Section, Hawaiʻi Dept. of Education; Clyde Igarashi, Evaluation Specialist, System Planning and Improvement Section, Hawaiʻi Dept. of Education; Donna Fujimoto-Saka, Evaluation Specialist, System Planning and Improvement Section, Hawaiʻi Dept. of Education; John Carroll; Tina Winquist

This poster will cover the first year progress of Hawaiʻi`s Race to the Top Project (RTTT). The following four core reform areas will be covered with emphasis on evaluation components: 1. Adopting standards and assessments that prepare students to succeed in college and the workplace and to compete in the global economy; 2. Building data systems that measure student growth and success, and inform teachers and principals about how they can improve instruction; 3. Recruiting, developing, rewarding, and retaining effective teachers and principals, especially where they are needed most; and 4. Turning around our lowest-achieving schools. A summary report consisting of evaluation activities across 12 RTTT states will be included and comparisons will be made wherever appropriate.