Paper and Poster Sessions

Paper Sessions (2:45-4:00 pm)

Cultural Evaluation in Education
Room: Kaiulani I

Ho‘omau I Nā ‘Öpio: Recognizing Youth Developmental Assets and Hawaiian Cultural Connectedness

Kathy Tibbetts

PowerPoint

Increasingly, evaluators are approaching their work with strengths-based models in mind. In this presentation we discuss the Ho‘omau I Nā ‘Öpio study, a research/evaluation project examining developmental assets and Hawaiian cultural connectedness among local youth, ages 12-18. In partnership with a variety of community stakeholders and the Search Institute, a national leader in the field of youth developmental assets, the project team developed and tested a new instrument for use with local, school-age populations. The audience will hear about the origins of the study, the process of working with a community advisory board to modify the Search Institute's Attitudes and Behavior survey, and the development of a model for gauging youth connection to Hawaiian culture. Data and analyses will be shared from the pilot test (n= ∼400) and field test (n= ∼ 2,800) of the ‘Öpio survey administered in different school settings as well as out-of-school programs.

The Relationship of Evaluation and Research to the Exploration of 'Place'

Kay Fukuda & Jonathan Wang

PALS (Program for Afterschool Literacy Support) is a multi-year study of the impact of using place/culture-based literacy program on student achievement, including reading and school engagement. Participants included 4th, 5th and 6th grade students in public school with predominately Native Hawaiian student populations. This paper presents the relationship between the project evaluation and research processes, as the two informed and strengthened the other, and ultimately the study.

Evaluation Techniques for the 21st Century
Room: Kaiulani II

The Five "R's" of Culturally Responsive Evaluation within a Native Hawaiian Context: Relationship, Rigor, Relevance, Resilience, and Responsibility

Anna Ah Sam, Herb Lee, Darlene Martin, & Verlie Ann Wright

PowerPoint

Culturally responsive evaluation (CRE) in diverse cultures and contexts share one important similarity: the recognition that culture and context matter, and that there are no universally agreed upon rules that are applicable in all contexts (Guba & Lincoln, 2005). CRE acknowledges that culture is an explicit, rather than implicit, criterion of the program being evaluated. The purpose of this paper is to identify the distinguishing features of CRE within a Native Hawaiian context, using evaluations of culture-based curriculum projects conducted by the Pacific American Foundation as examples. The methodological practices and relevant findings of, and benefits to using, CRE are presented.

Addressing Data Collection Challenges in a Complex Community-Based Health Evaluation

Gina Cardazone, Landry Fukanaga, & Christy Nishita

PowerPoint

External evaluators of the Hawai‘i Demonstration to Maintain Independence and Employment (HI-DMIE), a federally funded diabetes and employment project, will discuss data collection challenges encountered when evaluating a community-based health intervention with multiple partners and data sources. Data collection challenges naturally occur during evaluation research. However, strategies for addressing these challenges provide opportunities to strengthen the evaluation. HI-DMIE requires data from participants, intervention staff, and partnering service providers, and has devised strategies for addressing data collection challenges from each of these sources. This presentation will discuss strategies for managing data collection challenges, and implications for analyses, conclusions, and replication.

A Qualitative Meta-Evaluation of Accreditation in Higher Education

Helen Slaughter

Using the "best approaches for 21st Century evaluation", from Stufflebeam’s study Evaluation Models (2004), and concepts from a variety of current evaluation approaches such as utilization focused evaluation (Patton, 2008), an accreditation approach to evaluating higher education programs will be analyzed and critiqued in terms of stakeholder involvement, goal free evaluation questions (Scriven), evaluation capacity building, and potential for contributing to genuine program improvement and reform.

Youth Evaluation in Education and Health
Room: Kaiulani III

Evaluating Tech Together: A Hands-on Science Curriculum Targeting Native Hawaiian Students

Adam Ayers & Judith K. Inazu

Tech Together taught hands-on electronics to sixth-grade students attending public schools in predominantly Native Hawaiian communities on the islands of O‘ahu and Hawai‘i. Tech Together was administered by Partners in Development Foundation, a nonprofit organization in Honolulu. In total, 1087 students from 16 schools received 20 hours of classroom instruction reinforcing 36 DOE standards in physical science. Students completed pre- and post-tests before and after receiving instruction. On average, students significantly increased their scores following the two-week course. Results were consistent across all schools and both islands, suggesting the program was effective in teaching new scientific concepts to students.

Comparing Youth Served by a System of Care in Guam to a National Sample

Bonnie Brandt, Ranilo Laygo, Ye Xu, & Jennifer Dewey

The National Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program has generated one of the most extensive databases in the field of children's mental health. However, there has not been a systematic comparison of a Pacific island population served by a system of care to a national sample of youth and children served by the program. The following paper compares demographic, diagnostic, and outcome data of children and families served by Project I Famagu’on-ta (Our Children) in Guam to a random sample drawn from the remaining 28 sites which were part of the same funding cohort.

Poster Session and Networking Ice-Cream Social (4:00-5:00 pm)

Evaluation of a Two Week Web-based Summer Institute for Non-native-speaking Teachers of Chinese

Ritsuko Iyoda & HeeJin Kim

A two-week web-based language institute for non-native-speaking teachers of Chinese was evaluated for multiple purposes and across multiple stages, from program development through delivery of the institute. The evaluation was set theoretically within a utilization-focused approach (Patton, 2008). The external evaluators' early involvement, along with persistent collaboration by the primary intended user, contributed to the generation of actionable evaluative information as well as a reduction in the instructor's workload, and program improvement was accomplished while it was being developed and during implementation. This evaluation suggests diverse values that program evaluation can offer, beyond serving as a required external accountability mechanism.

Short Stories on Evaluation Projects in the Pacific: Successes and Challenges

Sonja Evensen & Andrew Sahallie

Several evaluation projects, set in a Pacific Island context, have been designed to work under challenging conditions of distance, limited technologies, and cross cultural situations that need to be considered in the evaluation design. It is hoped that this presentation will provide some ideas for practical approaches to culturally responsive evaluation design, as well as a place for participants to share what works for them.

Project Ho‘omohala: Evaluation of Cultural Competency

Stephanie Manley & Angela T. Hoppe-Cruz

The Cultural Competence and Service Provision Questionnaire (CCSP-Q) completed by Project Ho‘omohala youth and caregiver participants, rates the cultural competency of their service provider/s. The instrument inquires of the youth and caretaker how important it is for service provider/s to:

1. Understand practices, customs, and traditions of their child's racial/ethnic group
   
2. Understand the beliefs and practices of their child’s racial/ethnic group included in service planning and provision
   
3. That the person the caregiver and their child have seen most is of the same racial ethnic group as your child.

Results indicate importance of cultural competency to effective service provision.

Guided Action Research as a Culturally-Responsive Evaluation Methodology

Wendy Kekahio & Katherine Tibbetts

Recent research and program evaluation literature (Hood, Hopson, & Frierson, 2005; Smith, 1999; Thompson-Robinson, Hopson, & SenGupta, 2004) strongly documents the need for evaluation methods that are responsive to the values and perspectives of minority and indigenous communities. The subject of this paper is an evaluation approach that is responsive to the cultural context of the Native Hawaiian education community. In addition to being culturally respectful and responsive, the approach extends the conventional purpose of evaluation to prove or improve, by employing a meta-action-research strategy to support the transfer of skills learned at the training and assess their impact on teaching and learning through synthesis of findings across multiple action research projects. The paper reports on the successes and challenges encountered as the approach was implemented and concludes with recommendations for future practice.

Pre-K ACE (Pre-Kindergarten Accelerated Community Empowerment): Young Children from Diverse At-Risk Hawaii Communities Ready to Learn in 20 days

JoAnn Yuen, Peter W. Dowrick, Elisapeta Alaimaleata, Chuan Chang, Yoko Kitami, & Alok Rajouria

Pre-K ACE is a 20-day, 80-hour transition program—offered on O‘ahu, Lanai and the Big Island—designed to get young children ready to learn. Pre-K ACE uses a highly structured kindergarten curriculum, certified kindergarten teacher, educational assistants, and community volunteers (1:4, adult to student ratio). A study conducted during in 2008 documented a 5-month developmental gain, on average (Woodcock Johnson III). Children had little or no preschool experience, and were at-risk (i.e., low-income, SPED, English Language Learners, etc.). Evaluation is designed to confirm the efficacy of such a cost-effective "boost" and inform Hawaii's early education policy.

Capturing and Telling Our Stories of Success: The Native Hawaiian Education Council’s Culturally Relevant System for Better Assessing Impacts of Native Hawaiian Programs

Colin Kippen & Reshela DuPuis

The Native Hawaiian Education Council’s (NHEC) mission is to assess, evaluate, and coordinate federally-funded Native Hawaiian Educational Programs and to report and make recommendations to federal, state and local governments in order to improve educational outcomes for Native Hawaiians. To fulfill this mission, a multifaceted matrix was developed called the Indicators System, which enables the NHEC to collect and tell the stories of success of indigenous education programs focused on promoting the well-being of Native Hawaiians, from keiki to kupuna. The matrix allows programs to capture information about the different services they provide and the participants whom they serve.

One Project, Multiple Approaches: Evaluating the Hawai‘i Gatekeeper Training Initiative Across Multiple Agencies, Settings, and Contexts

Cory Cook, Nancy Marker & Judith Inazu

As a result of the Garrett Lee Smith Memorial Act, the Substance Abuse and Mental Health Administration (SAMHSA) of the Department of Health and Human Services funded the Hawai‘i Gatekeeper Training Initiative (HGTI). The HGTI seeks to enhance state level infrastructure for youth suicide prevention efforts and enhance youth suicide prevention efforts in three systems: Public Schools, Law Enforcement, and Alcohol/Substance Abuse Treatment and Prevention in the state Department of Health. This poster highlights multiple evaluation strategies used to assess the procedures and outcomes of the HGTI and serves to demonstrate successful cross-site evaluation approaches with other SAMHSA evaluation teams and grantees.