Symposium, Paper, Round Table, and Poster Sessions

2012 H-PEA Conference

September 7, 2012
Koʻolau Ballrooms, Kāneʻohe

Symposium and Paper Presentations (1:15 pm - 4:00 pm)

 

Implementation of Measurement Feedback Systems in a Child Mental Health System  [symposium]

David Jackson, Hawai'i DOH Child & Adolescent Mental Health Division

The Child and Adolescent Mental Health Division (CAMHD) of the State of Hawai’i’s Department of Health is committed to a more data-driven decision-making system for its administrators, caseworkers, providers, and families. Emerging research is beginning to show that Measurement Feedback Systems (MFSs) are a critical component in improving mental health outcomes, but little is known about how to best implement such systems. This presentation reviews the latest research on MFSs and their implementation, and describes CAMHD’s current efforts to develop their own real-time data feedback system.

Implementing A New Mental Health Assessment for Youth: The Ohio Scales Pilot Project[symposium]

Scott Keir, Department of Health-Child & Adolescent MH Division (CAMHD)

The Child and Adolescent Mental Health Division (CAMHD) of the State of Hawai’i’s Department of Health is committed to a more data-driven decision-making system for its administrators, caseworkers, providers, and families. CAMHD has targeted a great deal of resources to staff access to relevant and timely information that will result in improved outcomes for youth on their caseload. In 2012, CAMHD began a pilot project to test the Ohio Scales mental health assessment tool as a potential addition to the information already collected on the youth served by CAMHD. This presentation will share the methodology and results of the pilot project.

Implementing Individual-Level Practice Profile Reports to Support Data-Based Clinical Decisions within Systems of Care [symposium]

Christopher Rocchio, Monitoring Coordinator

The Child and Adolescent Mental Health Division (CAMHD) of the State of Hawai‘i’s Department of Health is committed to a more data-driven decision-making system for its administrators, caseworkers, providers, and families. The CAMHD Research and Evaluation Team (RET) is currently piloting a measurement feedback system using the Monthly Treatment and Progress Summary (MTPS) with Project Kealahou, a six-year federal grant that aims to implement and evaluate innovative, evidence-based services to girls with trauma histories. This presentation will share how practice profile reports are being used to support data-based clinical decisions for better addressing the complex needs of girls with significant trauma.

 

An Educational Program on Hawaiian Elementary School Students’ Teachers’ Effectiveness, Interest in STEM Subjects, and Parental Involvement on Student Achievement

Clifford H. Clarke, GIS, LLC, & UHM; Naomi Takashiro, UHM; Yao Zhang Hill, GIS, LLC, & Kapi'olani Community College,Timothy J. Fulkerson, HFFA Foundation, Wa'akunewa i Ka 'Ikena Program

This Energy Technology program examined the influence of Hawaiian student perceptions of parental involvement, teachers’ effectiveness, and interest in STEM subjects on student academic performance. Five hundred twenty-seven students from 13 schools in Oahu participated in this study. Students learned energy technology concepts during the program and their academic achievement in this area was assessed before and after the program. In addition, students filled in questionnaires about their perceptions of teacher effectiveness, parental involvement, and interest in STEM. The main findings supported the research questions on whether parental involvement, teacher effectiveness, and student interest would significantly predict student academic achievement.

Assessing Quality of Worklife in Higher Education: A Proposed Model

Lynn Inoshita, Honolulu Community College

Assessing worklife perceptions in higher education is important because how people feel about specific worklife issues affects performance, impacts morale, and has implications for the general well-being of the entire organization. This study proposed a quality of worklife model comprised of six factors (using confirmatory factor analysis), tested the generalizability of the model across two groups of support personnel (using test of model invariance) and examined differences in latent means to assess differences in the perceptions of worklife across both groups. The results revealed that the proposed model had construct validity, and was generalizable across both groups of employees.

Insider Evaluation: Using Hawaiian Cultural Ways as Method

Kalei Kanuha, University of Hawai'i at Mānoa

The use of insider, indigenous or "N/native" epistemologies and methods in research has been a growing field of interest over the past two decades. As more N/native and indigenous peoples gain access to academia and to knowledge-generating institutions, we have been challenged to re/consider core methodological concepts such as validity, generalizability, reliability and even confidentiality as now understood in deeply cultural ways. This presentation will focus on the praxis of theory and action as represented in insider or N/native epistemologies and methods of evaluation. These issues will be discussed through a case study of a Hawaiian cultural domestic violence program evaluation conducted by a Hawaiian researcher in her own birth land; a Native being N/native vs. "going Native" in the Malinowski tradition of field study.

What the Federal Human Subjects Regulations Really Say About Educational Evaluation

Morris Lai, University of Hawai'i

The federal regulations regarding research involving human subjects are often inappropriately interpreted by IRBs as well as by educational evaluators. In this presentation I select different parts of the regulations, for which I give examples of such misinterpretations.

Evaluation for Program Development: Investigating the Need for Teaching Experience Within a New Bachelor of Arts Program in Second Language Studies

Sena Pierce, Pacific Center for Changing the Odds

An evaluation into the need for students to gain teaching experience in a new BA program followed a utilization-focused evaluation (UFE) design conducted in two phases with the end goal of evaluation for program development. Data was collected using anonymous web-based surveys, a Delphi technique and focus groups. An unexpected finding revealed that stakeholders held different conceptions of students’ professional identity within the degree. Consequently, the present study explored the use of UFE as a means of program understanding and self-exploration at a time of transition in the field of second language teacher education.

One Strong ‘Ohana: An Evaluation of a Partnership Campaign to Raise Awareness of Child Abuse and Neglect Prevention in Hawai’i

Angela Sy, Gina Cardazone, Ivan Chik, University of Hawai'i at Mānoa; Dana Apo, Department of Health

The One Strong ‘Ohana Campaign (OSO), which aims to increase public awareness of protective factors that can help prevent child abuse and neglect, was launched in January 2012 by the Hawai'i Children’s Trust Fund. OSO accomplishes its goals through coalition action, traditional media activities, social media, community events, and small grant funding. In this presentation, we will describe the evaluation planning process and results of the public awareness campaign in Year 1. Lessons learned and recommendations on conducting an evaluation with often diverse interests of the HCTF Advisory Committee will be described in relation to the Standards of Program Evaluation.

Roundtable Discussions

Scoring Teachers’ Responses to Mini-vignettes for Measuring Professional-Development Effects

George M. Harrison, Lisa M. Vallin, Paul R. Brandon, Curriculum Research & Development Group, University of Hawai'i at Mānoa

One of the outcomes of interest in teacher professional-development programs is the degree to which the teachers improve their understanding of pedagogy. This study describes the development of an assessment for measuring teachers’ understanding of inquiry-based science teaching and presents the pre- to post-project changes. Still under investigation is the degree to which we can place confidence in the scores of teachers who provide very short responses. We ask for discussion on whether these short responses reflect teacher understanding and on how to encourage more substantive responses from teachers on this low-stakes assessment.

Analyzing and Validating Survey Items Using Modern Test Theory IRT

Yao Hill, Kapi'olani Community College

This paper will discuss the advantages and caveats of using item response theory (IRT) or latent trait theory to analyze and validate Likert-scale survey items. The data used was collected from 194 respondents on the Faculty Confidence and Engagement Survey (FaCES) administered at Kapi'olani Community College in spring 2012. The main advantages of employing IRT are its ability to transform the survey ratings from ordinal scale to interval scale, and its ability to align persons' attitude with items endorsability. However, accurate IRT results come from large data sets and fulfillment of all the assumptions.

Development of the Pacific Education Research Resource (PERR) Logic Model Module

Brian Lawton, University of Hawai'i at Mānoa; Wendy Kekahio, McREL; Paul R. Brandon, University of Hawai'i at Mānoa; Krishna Narayan, NSC Partners, LLC.

The PERR Logic Model Module is a technical assistance product, developed under the Regional Education Laboratory Contract for the Pacific Region, to support data driven decision-making in the Pacific region. The product uses guides that aim to provide participants with an understanding of logic models, as well as an interactive method to develop them. The purpose of this round-table presentation will be to present our work to date to local research and evaluation professionals, with the goals of informing them about the forthcoming release of the module and obtaining feedback to increase the usability of this product.

Dealing With Missing Survey Data

Monica Stitt-Bergh, University of Hawai'i at Mānoa

Particularly in longitudinal studies, missing data/values are an issue. Disregarding missing responses can result in biased analyses and incorrect interpretations. The presenter will describe methods to deal with missing data and ask round-table attendees for feedback on her logic and decisions on handling missing survey data in a longitudinal evaluation project (a "work in progress") that began with 251 participants. Attendees do not need to be experts in statistical methods because the round-table discussion will focus on the underlying motivation and principles of dealing with missing data and not on step-by-step statistical processes.

Poster Session and Networking Ice-Cream Social

Developing Tools to Evaluate Teaching & Learning Environments: A Participatory Approach to Gathering Parent Voice

Rozlynd Awa, Kamehameha Schools

In the spring of 2012, the Office of Teacher Effectiveness at Pittsburgh Public Schools (PPS) began a partnership with Carnegie Mellon University (CMU) to create tools that would support their efforts in increasing parental engagement. To this end, PPS and CMU used a participatory evaluation approach to create a parent survey and research of best practices in parent engagement, among other tools. This presentation describes the collaborative approach used to engage key stakeholders in the creation of such tools, as well as the progress made towards improving the districts relationships with parents and increasing parent’s involvement in their children’s education.

Talking Story: Understanding the perceptions of teachers using a culture-based curriculum

Jamie Dela Cruz, Argosy University

This phenomenological study investigated the perceptions of teachers who implemented a culture-based curriculum at an elementary school on Oahu. Aloha ‘Āina is a culture-based curriculum with instruction and student learning grounded in the values, norms, knowledge, beliefs, practices, experiences, and language that are the foundation of the Hawaiian culture. Eight teachers were interviewed after they used the culture-based curriculum in their classrooms during the 2010 spring semester. Data analysis revealed four categories: teachers’ initial experiences, student engagement, challenges and opportunities, and meaningful experiences. Teachers were challenged by the culture-based education program and teachers’ perceptions of the Aloha ‘Āina curriculum were positive, most agreeing that it helped students to learn and improve student engagement through hands-on learning in and outside of the classroom.

Native Hawaiians in the USA, 2010

Justin Hong, Kamehameha Schools

The recent release of 2010 Census data revealed important facts and trends about the Native Hawaiian population across the nation. This poster depicts Native Hawaiian population counts for each state and includes the 2000 counts to indicate growth by state. In addition, the state of Hawai‘i map contains population counts for more detailed levels of geography. Highlights include the following: (a) The total Native Hawaiian population nationwide has increased by 31.4 percent over the past decade. (b) A greater proportion of Native Hawaiians now live on the Continent compared to 2000 (45 percent in 2010, compared to 40 percent in 2000). (c) Nearly 20 percent of all Native Hawaiians live in the West Coast states. (d) In Hawai'i, there was no meaningful change in the distribution of Native Hawaiians from county to county between 2000 and 2010.

The Hawai'i Partnership for Educational Research Consortium (HPERC)

Judith K. Inazu, University of Hawai'i at Mānoa; Christina Tydeman, Hawai'i Department of Education

This poster describes the recently-organized Hawai'i Partnership for Educational Research Consortium (HPERC), a working group comprised of Hawai'i-based organizations that support and conduct research focused on the improvement of educational outcomes for Hawai'i’s keiki. The poster outlines HPERC’s purpose, goals, and membership composition. HPERC works to foster collaboration among members of Hawai'i’s community of educational researchers and build statewide capacity to conduct educational research in Hawai'i. HPERC provides input to the list of educational research priorities developed initially by the Hawai'i Department of Education based on a survey of stakeholders. HPERC is one of the initiatives outlined in the federally-funded “Race To The Top” award.

Integrating a First Time Evaluation of the 25-Year RCMI-BRIDGES Program

Stephanie Nishimura, Jane Onoye, Gina Cardazone, Judith K. Inazu, University of Hawai'i at Mānoa

After 25 years of the RCMI-BRIDGES grant at the University of Hawai'i, a systematic program evaluation was implemented for the first time. The overall aims of the program evaluation are to conduct formative evaluation, develop mechanisms and protocols for communicating evaluation information, and assessment of progress toward project aims. Initial meetings with Core Directors were conducted to develop logic models and establish data collection methods. The dashboard concept was utilized to provide real-time summary of usage and user characteristics. Additionally, user satisfaction, progress reports from each Core Director, and faculty development and achievement information were obtained through web surveys.

Evaluation of the iPad in Orthopaedic Surgery Residency Training

Stephanie Nishimura, University of Hawai'i at Mānoa; Joseph Varcadipane, Orthopaedic Resident; Nick Scarcella, Orthopaedic Resident; Maria Chun, University of Hawai'i at Mānoa; Gary Belcher, University of Hawai'i at Mānoa

As mobile computing technology continuously improves, tablet computers (i.e., Apple iPads) are becoming more prevalent in modern medical practice and education. For the first time, in July 2011, 11 orthopaedic surgery residents were provided an Apple iPad 2 to assist with clinical and educational duties pertaining to their residency. In order to assess the impact of the iPad 2 on residency experiences, both quantitative surveys (at two time points) and qualitative methods (i.e., focus groups) were conducted. General topic areas of information including computer usage, educational activities, clinical duties, patient education, and feedback on the iPad 2 were obtained.

Program Sustainability Evaluation Study: The Story of Public School Dance Education Programs

Amy Schiffner, University of Hawai'i at Mānoa

Dance education programs in public high schools are the most underrepresented of all the performing arts to date. There are many factors that lead to the underrepresentation of dance education such as the lack of funding, appropriate facilities, teacher credentialing, instruction time, and low priority placed on arts education to name a few. In this era of "Arts in Crisis" arts educators and advocates have sought to find ways to sustain the presence of arts programs in our public schools. In this work in progress evaluation study I have begun to evaluate the program goals, curriculum and organizational structure of one public high school dance program that has managed to sustain itself in a community and education system full of adversity. What is revealed in this evaluation study may serve as road map for other dance programs seeking sustainability.