Abstracts

A Meta-Evaluation of an Afterschool Program: Catalyst for Change [roundtable]*
A. Ah Sam, K. Morris, M. Greaney

The objective of this session is to highlight the rationale, process and outcomes of conducting a meta-evaluation study of a community-based after school program. The study became the catalyst for revamping the organization’s existing evaluation system. The results of the study helped to refine the internal and external evaluation system, and aided in identifying the requirements for establishing a longitudinal evaluation system. Through hearing about how the program’s evaluation evolution occurred, session attendees will learn how meta-evaluation can be useful (e.g., to identify strengths and concerns about evaluation methodology that need to be discussed), the general steps in conducting a simple meta-evaluation, and possible ways that findings can be used. It is hoped that attendees will share either similar experiences or similar insights.

When, Why and How to use Propensity Score Matching Techniques in Evaluation Research [demonstration]*
J. Barile

Evaluation researchers are often charged with comparing program outcomes between two or more non-randomized groups. In most cases, simply comparing outcomes of individuals that participated in a program to non-participants can introduce selection bias. Propensity score matching is a quasi-experimental technique that is used to statistically match cases between two or more non-randomized groups. This presentation will demonstrate when, why and how an evaluator can use propensity score matching techniques to compare non-randomized groups. This presentation will show how matching techniques can be carried out using popular software packages and what to look out for when examining results.

Best Practices for Including Temporary, Student and Volunteer Staff in the Evaluation Process [paper]
D. Bost

Academic, non-profit and government agencies alike are often faced with few permanent full-time staff members, so to meet staffing needs these entities often turn to undergraduate students assistants, interns, volunteers, temporary agency employees and even friends and family to complete essential tasks such as evaluation. These persons, though willing to work, often arrive with little training and interest in the practice of evaluation. The presentation will share tools for maximizing the talents and interests of workers that are unfamiliar with evaluation and are very likely using evaluation only in the context of a temporary employment and short term assignments.

Phone Surveys in Evaluation: Effective Planning, Implementation, and Tracking [demonstration]
J. Caparoso

Tired of online surveys? A phone survey is personable, adaptable, and brings the client voice into an evaluation. This demonstration will guide attendees through the process of planning, conducting, and tracking phone surveys. Specifically, the presenter will discuss how to design useful phone surveys, establish speaking scripts, create a respondent tracking sheet, and effectively communicate with respondents during the survey. Attendees will leave with practical tips and techniques that address new and evolving challenges such as caller ID and voice mail. The presenter will use examples from an evaluation that included phone surveys and cold-calling over 1,000 clients.

Got Data? Building Local Capacity to Use Publicly Available Data [poster]
F. Frank, S. Dalton

REL Pacific recently conducted a project to promote the use of publicly available education data among the Hawai'i education research community by creating Guidance Materials to Support Use of Hawai‘i-Specific, Publicly Available Data Sources. The Guidance Materials explain how to access publicly available data sources, identify questions that can be supported by the data sources, and discuss the limitations of the data sources. This poster will provide attendees with a step-by-step tutorial on how to navigate and use the Guidance Materials as a tool for using publicly available data.

Keeping it Simple: Straightforward Strategies and Resources to Build Evaluation Capacity [roundtable]
D. Fujimoto-Saka

Evaluation Capacity Building (ECB) has been defined as the “intentional work to . . . make quality evaluation and its uses routine.” ECB can seem like a daunting task, especially if it is something new to your organization and/or to you as an evaluator. As with most new and encompassing tasks, it helps to identify a conceptual framework from which to visualize how to approach ECB in your organization and then to try out some of those ideas on a small scale. This round table will discuss ECB frameworks and experiences as well as to share resources and brainstorm simple strategies to facilitate ECB in organizations.

Building an Evaluation Culture to Promote Effective and Continuous Improvement within a Student Services Unit [roundtable]
K. Gushiken Jernigan

The intent of this roundtable presentation is to create a time for discussion and sharing of ideas about ways to build an evaluation culture within a student services unit in a university setting. A Career Counselor from the University of Hawaiʻi at Mānoa Career Center will share strategies on building an evaluation culture that empowers individual counselors (faculty) to reflect on their practices while assisting students with accomplishing their career goals. As an internal evaluator, the counselor will present on the development of the evaluation process, successful strategies, lessons learned through the process, and challenges encountered throughout this initiative.

Using the Theory of Planned Behavior to Model Evaluators’ Research-on-Evaluation Beliefs and Use [paper]
G. Harrison, N. Lewis, P. Brandon

This paper presents the methods and results of a structural equation model of 1093 evaluators’ responses to a questionnaire about research on evaluation. We present the theory that informed the design of the questionnaire and how well a model based on this theory fits to the data. The results support utilizing the theory of planned behavior, along with background and demographic variables, when examining evaluators’ self reports of their research-on-evaluation beliefs, perceptions, and use.

Impact Evaluation of Grass-Root Leadership Development [roundtable]
Y. Hill

This roundtable discussion presents the current impact evaluation model of a grass-root assessment leadership development project in a higher education setting. The model examines participant achievement of knowledge and skill learning outcomes, assessment plans produced, implemented, and presented as a result of the participation in the project, coordination of collaborative assessment activities in their respective programs, and changes in program assessment activities. The presenter is seeking feedback on other meaningful and useful measures of impact of leadership development and suggestions for data collection and reporting.

Opportunity Youth [poster]
J. Hong

The well-being of our communities is linked to the success of the youth. Opportunity youth consist of young individuals who have become disengaged from both school and work. As a result, this population is often not well-understood. This study helps describe the size, characteristics, and implications of the Native Hawaiian opportunity youth population in Hawai'i.

Field Testing the Native Hawaiian Educational Council’s Common Indicators System and Framework [symposium]
S. Hussey, I. Luning

In 2014, the Native Hawaiian Education Council (NHEC) completed a Study of Common Culturally-Aligned Evaluation Measures in which evaluation measures and tools used by former and current Native Hawaiian Education Program funded program grantees were identified, inventoried, and categorized into its Common Indicators System and Framework (CISF). In May 2015, NHEC kicked off its CISF Field Testing Project to (1) evaluate how participating programs incorporate cultural measures in their evaluation activities; and (2) evaluate the accessibility, reliability, and utility of the CISF to measure the culture-based outcomes of Native Hawaiian education and culture-based programs in a systemic manner. The Pacific Policy Research Center is assisting NHEC with the field testing project. The H-PEA symposium will provide a status of the project’s process in organizing field testing cohorts and the process by which field testing elements were chosen among cohort members.

Sources and Quality of Data and Research Available to the Pacific Education Community [poster]
J. Isip

This poster will present study findings from a collection and analysis of survey data about how Hawaii's education community access current information and applies research findings to school improvement and policy development initiatives.

Re-Thinking Data Among Marginalized Communities: Comprehensive Research in the Social Sector [roundtable]*
J. Lee-Ibarra

While data and research can advance efforts in the social sector, they can also be used—often unintentionally—to perpetuate inequity and the status quo among marginalized communities. The purpose of this roundtable is twofold: (1) to examine limitations of data in serving the needs of marginalized populations, and (2) to discuss strategies for approaching research and data collection more comprehensively. Drawing on examples from education and public health, we will consider the use of data to reinforce deficit-based approaches, potential concerns with comparison groups, the value of qualitative data, and the critical need for community engagement in research.

Hō‘ike Ana: A Cultural Evaluation Framework [roundtable]*
M. Lloyd, L. Benson, S. Tokunaga-May, S. Wada

Organizations serving the needs of indigenous peoples are challenged to develop an evaluation framework reflecting the epistemology of these populations. What spaces do culture and spirituality occupy within such a framework? What outcomes and methodologies play a dominant role when considering a culturally-relevant process of evaluation? What are the challenges in translating the value of outcomes to a wider audience such as funders? In 2012, the Queen Lili‘uokalani Children's Center developed Hō‘ike Ana, a cultural evaluation system. This presentation describes the organization’s efforts to define and implement Hō‘ike Ana as a journey of co-learning with our staff, managers, and communities.

The Importance of Culture in Assessment: Developing Culturally Responsive Assessments in Support of Hawaiian Culture-Based Charter Schools [symposium]
L. Makekau-Whittaker, C. Hoe, M. Kellig

In this session, presenters will share information about a multi-year project to develop culturally responsive tools for Hawaiian-focused charter schools to assess, enhance, and document cultural connectedness, readiness for the next level (including college, career, and community membership as contributing adult)s, and growth in academics. The assessment development work is grounded in the schools’ philosophies. Presenters will share the impetus for the work, its current status, and lessons learned to date. After brief presentations, audience members will engage with the presenters in small groups to ask questions and share their mana‘o and connections to work they are doing.

Applying Portfolio Assessment to the Authentic Evaluation of Knowledge Gained in Hawaiian Cultural and ´Āina-based Project-Based Learning [roundtable]*
G. Manset, S. Evensen, R. Lee

The objective of this roundtable is to present our “work-in-progress” as we develop a portfolio evaluation system to assess the knowledge gained in a multi-site program that focuses on Hawaiian cultural and ´āina project-based learning along with college/career guidance with secondary students. In this discussion, we will review the theoretical support for portfolio assessment, the technical issues related to developing a reliable and valid assessment tool, and lessons learned of how to pragmatically apply portfolio assessment in this context.

Culturally Responsive Evaluation for Native Hawaiian Contexts [roundtable]
J. Matsuoka, H. Lee, C. Kauhane Lupenui, L. Watkins-Victorino, S. Maunakea, K. Tibbetts

This roundtable session will share information about ongoing work to support culturally-responsive evaluation for Native Hawaiian contexts. The work is an outgrowth of the evaluation hui that brought together Kanaka Maoli and Māori in 2003 and of current hui participants’ experiences with the Center for Culturally Responsive Evaluation and Assessment (CREA). Planned work includes the formal creation of CREA-HI (an affiliate of CREA hosted by the Consuelo Foundation) and weaving of a Native Hawaiian Evaluation Framework. Participants will be invited to share their perspectives on the roles of CREA-HI and key elements to be incorporated in the Framework.

Culture-based and Community Accountable Evaluation in Action: Community Learning Exchange ʻOhana Series[demonstration]
S. Maunakea, I. Ruelas

The program coordinators and external evaluator of the Community Learning Exchange ʻOhana Series—partnership between Kamehameha Schools Ka Pua Initiative, UHM Hawaiʻinuiākea School of Hawaiian Knowledge, INPEACE, and MAʻO Organic Farms (WCRC) within moku ‘o Waianae—will share their arrival at a community-based participatory evaluation framework. The evaluator will share her experiences in developing relationships with ʻohana and how this translates into data. Presenters will walk the audience through their dynamic evaluation design that is action-oriented, scholarly, inspired, and purposed towards the benefit of everyone involved from developing instruments to presenting findings and always remaining accountable to community.

Remediation within Higher Education [poster]
A. Morehouse

Higher Education stakeholders need to evaluate the current state of remediation within its Higher Education institutions. Specifically, streamlining the community college to four-year University pipeline within the state institutions. The purpose of this poster is to evaluate the current remediation statistics according to race, age and socioeconomic status and to examine the current programs being implemented to combat remediation. Finally, it discusses ways for the two-year state institutions to use best evaluation practices to streamline the remediation process and create clearer pathways for students to go on to four-year state institutions.

Where do you Draw the Line? Ethical Dilemmas in External Evaluation [roundtable]
J. Philippoff, L. Vallin

Evaluators make a commitment to produce ethical work that includes making value-based judgments. However, program evaluators are often faced with ethical dilemmas in their work. These ethical dilemmas may sometimes present evaluators with moral challenges about how to most appropriately make a decision or present or disseminate evaluation findings to stakeholders. The purpose of this roundtable is to discuss some of the ethical dilemmas that may arise while conducting evaluations, and how to take appropriate action in supporting the needs of stakeholders while being true to the evaluation.

Data Visualization: A Step-by-step Demonstration of How to Make Engaging and Interpretable Charts in Microsoft Excel [demonstration]*
S. Pierce

In May 2014, Stephanie Evergreen and Ann Emery published the Data Visualization Checklist. The missing link for many evaluators, however, is the nuts and bolts of how to format a chart to meet guidelines for good reporting. The purpose of the proposed demonstration is to walk attendees step-by-step on how to format a default graph or chart in Microsoft Excel into a data visualization masterpiece. The demonstration will specifically cover, a) choosing the right graph, b) labeling, c) formatting grid lines, tick marks, and axes, and d) how to put your company’s colors into a graph or report.

If You Build It, Why Don’t They Come? Results from a First Year College Needs Assessment [paper]
S. Pierce

To keep students enrolled, University of Hawai‘i (UH) campuses provide a variety of programs and services to support students, particularly in their first year when students are most likely to drop out. In order to improve services to students, GEAR UP Hawai‘i partnered with eight UH campuses to conduct a year-long first year needs assessment. The project followed a mixed methods qualitative design guided by a project advisory committee. The presentation will overview the project highlighting stakeholder involvement, evaluation design, data collection, analysis, and results with particular attention to the development and involvement of the project advisory committee.

Introduction to Database Management Using Microsoft Access [demonstration]
R. Tolman

Often, programs deal in large quantities of data and information that is complex, interconnected, and ever-changing. Microsoft Access is a database management system that can be utilized for organizing and evaluating complex data. The purpose of this demonstration is to introduce members of the evaluation community to fundamental concepts of database management and demonstrate basic functions using Microsoft Access. The demonstration will cover applications including how to create relational data tables, data entry forms, how to analyze data with queries, and construct dynamic evaluation reports.

Coding RMATRIX Translational Research [poster]
R. Tolman, S. Helm, J. Inazu

The RMATRIX project in the School of Medicine is funded by NIH, with Dean Jerris Hedges as the PI. RMATRIX’s mission is to promote the rapid transfer of research findings to treatment settings to reduce minority health disparities. The evaluation was designed to document the type of translational research being conducted by RMATRIX. Classification of research projects could be useful in documenting current levels of translational research and how this changes over time and helping decision makers assess intervention strategies to further stimulate translational research. A framework for classifying research into translational categories was developed, based on a literature review. A cadre of biomedical researchers were recruited and trained to code RMATRIX research into the specified categories.

Rigorous Evaluation of a Culturally-Responsive Teen Pregnancy and STI Prevention Program for Hawai‘i Middle Schoolers [poster]
L. Toms Barker

This poster will present the study design and findings of a randomized controlled trial (RCT) of a culturally responsive Teen Pregnancy and Sexually Transmitted Infection prevention program targeting youth in the state of Hawai‘i. Pono Choices was developed by the University of Hawai‘i, Planned Parenthood and Alu Like, explicitly incorporating the values and perspectives of the Hawaiian host culture, emphasizing Hawaiian cultural values to promote positive character development and “pono” (right) choices. The poster will present findings from an analysis of the impact of Pono Choices at one-year follow-up and present some of the challenges in conducting the evaluation.

Evaluating Learning Community in Higher Education: Practical Implications for Educators [poster]
K. Van Duser

Access to College Excellence (ACE) at University of Hawaiʻi at Mānoa provides a series of first year programs that serve approximately 25% of incoming freshmen a year. By introducing the major ACE program: Learning Communities, the poster will help participants learn successful strategies of running a first-year learning community program, such as cohort registration, freshman seminar course, and peer mentoring. Participants will also learn how to analyze and present closed-ended and open-ended survey responses from pre- and post surveys to assess program outcomes and improve the program and how to track retention rate to evaluate the program impact.

* = offered twice

updated 8/30/2015