Paper, Demonstration, Symposium, and Poster Sessions

2014 H-PEA Conference

September 5, 2014
Koʻolau Ballrooms, Kāneʻohe

Paper & Symposium Session (1:15 – 2:30)

Creating Meaningful Measures for Native Hawaiian Health Programs: Decolonizing Our Metrics [Symposium]
Sharon Kaiulani Odom, RD, MPH, Kokua Kalihi Valley Comprehensive Family Services; Mele Kalama-Kingma, RD, MPH candidate Ehuola 'Ohana Health Program Practitioner; Kat Burke, MPH Internal Evaluator

Native Hawaiian children are experiencing early weight gain in unprecedented numbers. Because obesity is so difficult to treat the Ehuola ‘Ohana Health Program takes an innovative approach, working through ‘ohana, family, as the natural and most powerful place to plant seeds of transformation. Ehuola is an alternative to Western approaches to obesity intervention that are well-conceived but often perceived as autocratic and negating by Native Hawaiian patients, that have failed to reduce the incidence of obesity, diabetes, heart disease and cancer. An evaluation system for such a program should focus on documenting and nurturing that transformation to achieve healing, reconciliation and the alleviation of suffering caused by health disparities.

Design process for a Hawaiian culture-based program evaluation [Symposium]
Katherine A Tibbetts, Kamehameha Schools; Karen Umemoto, Professor University of Hawai'i, Department of Urban and Regional Planning; Wayde Lee, Wahi Kana'aho

The Wahi Kana'aho program is a core component of an innovative diversion system for youth charged with status offenses and first time misdemeanors. Wahi Kana'aho is a 21-day residential program grounded in Native Hawaiian cultural values and practices that focuses on healing of the mind, body, and spirit. This workshop will introduce participants to key elements of the Wahi Kana'aho program then walk them through the evaluation decisions intended to reflect those elements. The session will start with a brief presentation followed by a "talk story" time during which the presenters and the audience will explore applications of this approach to other contexts.

 

CANCELLED Lessons learned from the evaluation of an ethnomathematics teacher professional development project[Paper]
Joanna Philippoff, Curriculum Research & Development Group; Lisa M. Vallin, Graduate Assistant; Paul R. Brandon, Professor; Linda H. L. Furuto, Associate Professor

We present findings from an evaluation of an ethnomathematics and STEM teacher professional development program. One definition of ethnomathematics is the relationship between mathematics and culture. This program was designed to increase participant’s knowledge and use of culturally-responsive STEM pedagogies. We will focus our presentation on our evaluation process, sharing the challenges of evaluating a program with a complex theoretical framework, multiple stakeholders, and limited budget. We hope by sharing our lessons learned, and how we approached building evaluative thinking among program facilitators, others can learn from our experiences.

“Hawaiian Place of Learning”: College Students’ Perceptions Over Time [Paper]
Monica Stitt-Bergh, Assessment Office, University of Hawaiʻi at Mānoa; Jenna Caparoso, Graduate Assistant, University of Hawaiʻi at Mānoa

The University of Hawai‘i at Mānoa’s strategic plan includes being a “Hawaiian Place of Learning.” In this presentation we describe our longitudinal study (2010-2014) that captured student perceptions related to Hawaiian Place of Learning (HPL). We present results such as students’ HPL definitions, HPL’s importance to them, extent to which they view the university as a HPL, and their self-reports of amount learned about Native Hawaiian culture and where that learning took place. In addition, we discuss the strengths and challenges of a longitudinal research design and next steps regarding use of results.

Some Implications for Evaluators of Recent Developments in Psychometric Validity Theory [Paper]
Paul Brandon, Professor, Curriculum Research & Development Group, College of Education, University of Hawaiʻi at Mānoa

Over the past 35 years, evaluators and psychometricians have borrowed from each other’s notions about test and evaluation validity and about test and evaluation use. Cronbach borrowed from House’s notion of argumentation and applied it to test validity theory; similarly, some evaluation theorists such as Kirkhart have drawn upon Messick’s notion of consequential validity to support notions about evaluation theory. In this presentation, I will provide an overview of the literature on key aspects of psychometric theory and evaluation theory, show the overlap between them, and present implications for evaluation practitioners.

Paper & Demonstration Sessions (2:45-4:00)

Evaluators' Perspectives on Research on Evaluation [Paper]
Nicole Lewis, Department of Educational Psychology, University of Hawaiʻi at Mānoa; George M. Harrison, Assistant Professor, Curriculum Research and Development Group, College of Education, University of Hawaiʻi at Mānoa; Anna Ah Sam, Associate Specialist, Office of Student Equity, Excellence & Diversity, University of Hawaiʻi at Mānoa; Paul Brandon, Professor, Curriculum Research & Development Group, College of Education, University of Hawaiʻi at Mānoa

Research on Evaluation (RoE) has been a growing endeavor in recent years. For example, in a review of the articles in the American Journal of Evaluation (AJE) from 1998 (when the journal took its current name) through 2012, 219 (37.4%) articles were classified as research on evaluation--a number that might surprise some evaluators. Even though the body of RoE is growing, there is a lack of empirical research on evaluators' degree of attention to the research, their attitudes toward it or beliefs about it, and their application of its findings. This presentation will describe the methods and results of an online questionnaire investigating evaluators’ perceptions of RoE. In addition, we will identify the challenges and successes we encountered in undertaking our study. Our intent is to increase the understanding of those attending our presentation of how the body of RoE is perceived and potentially put into use by evaluators and teachers of evaluation. We will also illustrate how such a study was conceived and implemented, including the work that went into developing and administering the questionnaire, completing the IRB process, and analyzing the data.

Collaborative Planning for the Actionable Evaluation of Cultural and Place-Based STEM Education: The Kamehameha Schools Kupa ‘Āina Program [Paper]
Genevieve Manset, Literacy Services; Brandon C. Ledward, Director, ‘Āina-Based Education Department, Kamehameha Schools

Stakeholder engagement is central to evaluation that is relevant and useful. In this presentation, we will discuss collaborative planning for actionable evaluation in the early stages of the Kamehameha Schools (KS) Kupa ‘Āina program. Kupa ‘Āina is a Native Hawaiian cultural and place-based STEM education program designed in partnership with Hawai‘i DOE and the University of Hawai‘i. The presentation will be framed by the following questions: How does collaborative planning contribute to actionable evaluation? How were the multiple stakeholders engaged both within and between organizations? How can we address the challenges involved in engaging stakeholders in evaluation planning?

Program Velocity: A New Tool for Academic Program Evaluation [Demonstration]
Delia N. Acevedo, PhD, University of Hawaiʻi at Mānoa; Gary Rodwell, STAR Director

Academic programs are in need of data that can enable them to identify constraints and strengths and take steps to improve how they serve their current students. Using “real time” STAR data and visual technology, we introduce the audience to the concept of student velocity towards degree completion, and how it can be aggregated into a new indicator of student progress at the academic program level known as Program Velocity. When presented in this manner, Program Velocity becomes a springboard for critical thinking, conversation, and hypothesis-testing about the programmatic factors that may be affecting student progress.

Synthesizing Results to Demonstrate Program Outcome Achievement – A Showcase [Demonstration]
Yao Hill, University of Hawaiʻi at Mānoa

This session will demonstrate how to synthesize results from both qualitative and quantitative types of data, collected at multiple time points from the participants in the Assessment Leadership Institute offered by the Assessment Office at University of Hawaiʻi at Mānoa. Specifically, the presenter will demonstrate how to code qualitative data from daily open-ended feedback forms and participant presentations, how to present closed-ended survey question results, how to align different pieces of evidence with each program outcome, and how to conclude on the achievement or lack of achievement of each outcome using result synthesis.

Poster Session & Ice Cream Social (4:00 – 5:00)

Implementing WASC’s Core Competencies in Coordination with Institutional Learning Objectives: Five Steps
Dawne Bost, University of Hawai‘i at Mānoa

This poster describes how the University of Hawai‘i at Mānoa implemented Institutional Learning Objectives and incorporated WASC’s five core competencies. The presenter will detail implementation strategies, give an overview of the assessment plan, and indicate what processes worked well. Attendees will leave understanding how the history of program assessment on this campus and appropriate processes led to minimal resistance and wide-spread support.

Research Into Practice at REL Pacific
Nitara Dandapani, Research Specialist, McREL International's Pacific Center for Changing the Odds; Jean Isip Schneider, Consulting Manager, McREL International's Pacific Center for Changing the Odds

This poster will present a representative overview of REL Pacific's evaluation work (including tools, trainings, and studies) conducted under a large federal contract, presented in a brief, graphically pleasing format. The poster will feature the following tools, reports, and studies in progress:

• A downloadable computer app that runs locally on Google Chrome that which guides users through a series of questions and provides opportunities to enter and connect program resources, actions, and outcomes. The end result is a printable logic model.
• A study of education management information systems in the Federated States of Micronesia: Education management information systems are designed to provide comprehensive, integrated, relevant, reliable, unambiguous, and timely data to education leaders, decision makers, planners, and managers.
• Data analysis training in Palau: This series of trainings for data teams in Palau focuses on Developing a Research Question and Exploring Data Quality; Choosing and Conducting Appropriate Analyses; Exploring Patterns and Trends; Identifying Possible Explanations for Patterns and Trends; and Communicating Results to Different Stakeholders.
• A guide that provides a framework and the tools and vocabulary needed to support data-informed conversations and action in education. The guide walks users through five key steps in using data for decisionmaking and strategic action: setting the stage, examining the data, understanding the findings, developing an action plan, and monitoring progress and measuring success.

Livable Income in Hawaiʻi
Justin Hong, Kamehameha Schools

Livable income is the income required to sustain an “adequate but modest” living. This measure may be a more accurate depiction of the income required to support a family’s basic needs compared with other measures of economic wealth (e.g., federal poverty threshold). Hawaiian households, who are already less likely to have a livable income than non-Hawaiian households, showed a fast decline in the proportion of households with a livable between 2003 and 2012. The trend is disconcerting as studies show strong relationships between access to financial resources and educational outcomes.

Coding Research Along the Translational Continuum--A Work in Progress
Judith Inazu, University of Hawaiʻi; Ryan Tolman, MA Graduate Resaerch Assistant; Susana Helm, PhD, Associate Profesor

PURPOSE. Clinical/translational research (CTR) infrastructure supported studies are evaluated using Institute of Medicine classifications (T0-basic biomedical/epidemiological research, T1-human translation, T2-patient translation, T3-practice translation, T4-community translation) to document changes in research emphases over time across CTR classifications and to assist decision makers in stimulating CTR. METHOD. Expert reviewers independently code studies, and inter-rater reliability (weighted kappa) and consensus coding are used. RESULTS. It is expected that as the Hawai`i CTR infrastructure expands, T3-T4 studies will increase. DISCUSSION. Coding will improve transparency for decision makers to increase investments in Hawai`i-Pacific CTR, particularly community-based approaches to working with Hawaiian and other Pacific people.

Toward Actionable Feedback Systems: Using CAFAS to Predict Treatment Success in Youth Mental Health Services
David Jackson, Dept of Health, Child & Adolescent Mental Health Division; Scott Keir, PhD; Sonia Izmirian, MA; Max Sender, BA; Charles Mueller, PhD

This poster presents findings from a study that aims to inform the development of an actionable feedback system, which is part of efforts to increase success rates for youth in public mental health services. Based on longitudinal Child and Adolescent Functional Assessment Scale (CAFAS) outcome data, a trajectory based on population percentiles was created for each level of baseline severity and service type. Youth who ultimately showed success at service discharge was examined for each percentile and timepoint. The results provide information about individual risk for unsuccessful treatment, which can be used to guide treatment in its early stages.

Utilization of Reflection as An Evaluation Method
Lu Leng, University of Hawaiʻi at Mānoa; Duan Wei, Chief Executive Officer of Lixin More Fun Children Development Center

Surveys, questionnaires, test, interviews, and observations have been used extensively for data collection in evaluation projects. Based on evaluation objectives and resource constraints, the evaluators utilized reflection as an evaluation tool in a seven-day intensive Philosophy for Children Hawaiʻi (p4cH) professional development workshop. In the poster presentation, the presenters will provide a rationale for the introduction of reflection into workshop evaluation, report on how they used this method to collect data for their workshop evaluation, and discuss the evaluation results. The findings suggest that reflection is an effective approach and method for improving the quality and depth of workshop evaluation.

Measuring Our Success
Jessica Miranda, College of Education, University of Hawaiʻi at Mānoa

At the University of Hawaiʻi at Mānoa’s College of Education, faculty select key program assessments to systematically measure, collect and analyze data on student learning. In the 2012-13 academic year, the college began creating visuals of aggregate student performance on key assessments and posting these data on our website. The data allow us to test our beliefs about our programs and make adjustments based on actual candidate performance and stakeholder feedback. The use of data visuals provides not only a more engaging means for faculty to analyze student performance, but also allows us to share this information with multiple audiences.

15 to Finish: Towards a New Normal for Full-Time Enrollment
David Mongold, University of Hawaiʻi System

The "15 to Finish" media campaign promotes a 15 hour credit load to finish on-time in four-years. This poster describes the analytical study conducted to provide a solid foundation for the campaign. Results of the media campaign are assessed by comparing cohorts of first-time freshmen from before and after the campaign. On-going efforts include the development of a decision tree to identify indicators of student success. Viewers can see videos from the campaign and learn about on-going assessment efforts.

CANCELLED Evaluation as a Tool to Guide and Support System of Care Advancement
Edward Suarez, PhD, Project Kealahou, DOH CAMHD; Kathleen McGeehan, PhD, Data Analyst/Program Monitor; Tamara Whitney, MBA

Project Kealahou (PK) provides gender-responsive, trauma-informed services and supports to female youth and their families and coordinates among Hawaiʻi’s mental health, juvenile justice, education and child welfare systems to promote System of Care principles. Local and national evaluations include collection of comprehensive longitudinal data from youth, caregivers and service providers. PK utilizes data in its continuous quality improvement (CQI) program to inform the development of tools for educating and motivating SOC stakeholders to improve the system of care. Results promote sustainability by demonstrating that Project Kealahou offers a distinct and promising alternative to the currently available service array in Hawai`i.

Native Hawaiians' Perceptions of Cultural Safety in Community Based Research
Suresh Tamang, University of Hawaiʻi at Mānoa; Dr. Lana Sue Ka‘opua, PhD, DCSW, LSW

Objectives: To explore the cultural safety from the perspective of Native Hawaiians and describes its relevance in the context of community-based research (evaluation). Theoretical Framework: Cultural Safety that focuses on the potential power differences between the researcher/service providers and the communities/participants. Methods: Five focus groups and five key informants interviews were conducted in three major homestead communities on the island of O‘ahu. A content analysis using a priori coding was performed. Data were audio-recorded, transcribed and coded to identify primary themes. Results: Upstream factors influence perceptions of cultural safety; attention to ethical values of Hawaiian culture and homestead community promotes cultural safety; culturally safe research reflects “culture” as multi-dimensional; community empowerment is intricately linked to cultural safety, cultural safety is relationally-based; and safety is holistic, with systemic and community factors influencing personal perceptions. Conclusion: Cultural safety is a relevant framework that honors culture and promotes active engagement of Native Hawaiians. Scientific Importance: Results can inform researchers (evaluators) how to design and implement culturally safe research evaluations among Native people.

 

 

 

updated 8/27/2014