Paper Presentations (1:15 – 2:30)

Balancing Outcomes Measurement Demands with the Complexities of Program Development
George Harrison, Curriculum Research & Development Group (CRDG)/University of Hawaiʻi at Mānoa (UHM); Lisa M. Vallin, CRDG/UHM; Joanna Philippoff, CRDG/UHM; Paul R. Brandon, CRDG/UHM

In programs that are developing, definitions of the intended outcomes may also shift or become more refined. This presents a challenge for evaluators tasked with creating instruments to measure effects during the final stage of program development, as psychometrically sound instruments require iterative editing based on program objectives and empirical data. This study describes the steps an evaluation team took to maximize content validity throughout a program’s development while also achieving measurement equivalence between two forms of an instrument used at different stages in the program development to measure the developed program’s effects on the outcome measure.

Creating an Evidence Based Agency: Strategy for Branding a Quality Non-Profit
Scott Ray, Scott Ray & Associates, Inc.; Jan E. Hanohano Dill, Partners in Development Foundation (PIDF); Alison T. Masutani, PIDF; Laura R. Dang, PIDF

This paper presents a strategy for documenting an agency-wide array of evidence-based programs. Partners in Development Foundation’s (PIDF) Integrated Evaluation Model (IEM) applies a common model of process and outcome evaluation to a variety of programs. While evaluation plans and tools are tailored to each specific program, some common outcome measures used to assess changes in client attributes can also support comparisons between similar programs with similar populations. A common approach is also used in assessing client satisfaction with services and the extent to which clients see the programs in which they are participating as supporting the agency mission.

The Foreign Language Instructors’ Perspective, Understanding, and Implementation of the “Third Culture” in Classroom Instruction
Xiu Zhang, Argosy University

While Kramsch and others have defined and discussed the “third culture” concept in language teaching, there has been little research into the foreign language instructors’ perspective, understanding, and implementation of the “third culture” in classroom instruction, particularly as it applies to Chinese language and culture.  The researcher gathered data from a qualitative phenomenological study in order to provide insight into this research question. [Focus will be on methodology]

 

Demonstrations (1:15 – 2:30)

Program Planning, Monitoring, and Evaluation: The Pacific Education Research Resources (PERR)
Lauren Mark, McREL; Krishna Narayan, Island Innovation, LLC; Nolan Malone, McREL

The PERR Education Logic Model (ELM) and Program Outcomes, Measures & Targets (POMT) modules are technical assistance tools developed under the REL Pacific contract for the Pacific educational community. These interactive computer-based modules can provide participants with an understanding of the specific content that goes into a logic model and how these models are useful in supporting short- to long-term program planning, monitoring and evaluation. The purpose of this demonstration will be to share with Pacific educators the most current versions of these modules and to obtain valuable feedback on the usability of these products.

Using Social Network Analysis to Visualize and Quantify Collaborative Networks
Gina Cardazone, University of Hawaiʻi at Mānoa (UHM); Ryan Tolman, UHM; Dan Rempala, UHM; Judith Inazu, UHM

Social network analysis (SNA) can be a useful means of evaluating collaborations between individuals, organizations, or other entities. This two-part demo of social network analysis will include an overview of SNA and three examples of how SNA has been used in evaluation, as well as a guided introductory step-by-step “tour” of the SNA software program UCINET. The examples will highlight varied approaches to data collection and analysis, with an emphasis on how evaluation studies can benefit from SNA. The guided tour will introduce attendees to the basics of preparing and importing data, and quantifying and visualizing networks.

 

Roundtables (1:15 – 2:30)

What the Literature Says About Involving Stakeholders in Program Evaluation: Discussing a Conceptual Model for Stakeholder Involvement
Landry Leimalani Fukunaga, Kamehameha Schools; Paul Brandon, Curriculum Research & Development Group (CRDG)/University of Hawaiʻi at Mānoa (UHM)

Stakeholder involvement is a key aspect of effective evaluation as evidenced by the inclusion of stakeholder perspectives in the evaluation competencies and standards, the focus on stakeholder participation in many evaluation theories, and the growing collection of literature on the topic. The objective of this roundtable discussion is to present a conceptual model of stakeholder involvement in program evaluation based on a systematic review of the research on evaluation literature. Attendees will have an opportunity to discuss factors that affect decisions to involve stakeholders in their own practice and the extent to which resources are present and challenges overcome.

CANCELLED: Using micro-analytic tools to examine qualitative data: How do we make it feasible?
Hanbyul Jung, University of Hawaiʻi at Mānoa

As part of a larger ethnographic evaluation of an in-service EFL teacher-training program in the U.S., this study focuses on the participants’ perceptions of the teacher-training curriculum delivered within a focus group. This study will highlight how implementing micro-analytic tools to examine the interactions in the focus group allows for a more in-depth discussion of the findings (e.g., participants’ collaboration in constructing responses to the focus group protocol), while complementing the results of the evaluation at hand. Furthermore, as a work-in-progress, the question will be posed as to how such micro-analytic tools can be made feasible for other evaluation practices.

An Evaluation Plan for Philosophical Inquiry Project
Lu Leng, University of Hawaiʻi at Mānoa

With continued collaboration between the Department of Education and the Uehiro Academy for Philosophy and Ethics in Education, this report is designed to make an evaluation plan to determine the implementation and effectiveness of a newly developed Hawai’i State Department of Education high school social studies course – Philosophical Inquiry. This report will introduce the philosophical inquiry course, and the University of Hawai’i Uehiro Academy for Philosophy and Ethics in Education, identify evaluation questions and evaluation goals, design plans to collect and analyze the data, and share lessons learned while designing this evaluation plan.

Who and How Long? Adventures in Capturing Participation and Dosage Data
Sena Pierce, Hawai‘i P-20 Partnerships for Education

The roundtable will focus on developing data collection systems to capture participation and dosage within a state-wide federally funded program, implemented in different forms across multiple and diverse contexts. The presentation will include background about the GEAR UP Hawai‘i statewide program, including its unique approach to programming and roadblocks to the collection of participation and dosage data. Current strategies to manage these challenges will also be presented. Key issues for discussion will include; (a) the collection of participant data across diverse contexts and projects, and (b) measuring outcomes without individual-level participation and dosage data.

 

Paper Presentations (2:45 – 4:00)

An Evaluation of Public Middle School Tour Event
Lu Leng, University of Hawaiʻi at Mānoa (UHM)

In 2013, four middle schools with a Parents for Public Schools organization held middle school tours for parents. This evaluation is designed to evaluate the middle school tours’ effectiveness in changing parents’ perceptions of public middle schools. Survey results suggested that participants had positive overview regarding the school tours. Most viewed the school tour as a learning experience. The middle school tour event and organization will be introduced, evaluation questions and goals, evaluation methodology, evaluation results, and recommendations will be presented in this evaluation report.

Culture-based STEM Curriculum – Can you bridge the gaps between individuals and between schools?
Yao Hill, University of Hawaiʻi at Mānoa (UHM); Naomi Takashiro, UHM; Clifford Clarke, Pathways to Learning

Using hierarchical linear modeling, this study investigated the effects of student characteristics and school characteristics on the 6th graders’ achievement in a locally developed culture-based STEM curriculum. The findings showed that the curriculum resulted in equivalent achievement between Native Hawaiian (NH) and non-NH students, between both genders, and among students of different social economic status. Academic preparation, ESL status, and disability significantly affected achievement. At the school level, school type, restructuring status, and low income status did not affect achievement. However, the lower performing schools had a faster growth rate, thus bridging the gaps of STEM achievement.

Evaluation Findings From the First Two Years of a State-Wide Coalition Campaign to Raise Awareness of Child Abuse and Neglect in Hawaiʻi: One Strong ʻOhana
Angela Sy, University of Hawaiʻi at Mānoa (UHM); Ivan Chik, UHM; Gina Cardazone, UHM

The One Strong ʻOhana (OSO) campaign aims to prevent child abuse and neglect (CAN) through increasing knowledge of protective factors against CAN. It features PSAs, social media, and collateral material dissemination. Process and outcome evaluation has been a core component since the campaign inception.  Methods involved a general public survey throughout Hawaiʻi (N=1524), collaboration network analysis with key informant interviews (N=11) and collaboration surveys (N=43), and tracking of media reach.  Results indicated that OSO reached a wide audience with statistically significant changes in awareness of CAN, and protective factors that can help to prevent them.

 

Symposium (2:45 – 4:00)

Evaluating Race to the Top: Who, What, Where, When, How
John Carroll, Hawai‘i Department of Education (HI-DOE); Jerry Wang, HI-DOE; Donna Fujimoto-Saka, HI-DOE; Dale Tanouye, HI-DOE, Tina Winquist, HI-DOE

Race to the Top is a federal government grant initiative aimed at transforming K-12 education by encouraging the implantation of strategic reforms related to curriculum, data use, leadership, teacher effectiveness, teacher induction and mentoring, and organizational efficiency.  Hawaii Department of Education won a grant and is moving into its third year of implementation (4 year grant).  American Institutes for Research (AIR) evaluated the first year of implementation and is in the midst of preparing a report on the second year.  HIDOE internal evaluators facilitate the evaluation work.  This presentation describes the work done to date with an emphasis on design, data collection, analyses, and reporting.

 

Poster Presentations & Ice Cream Social (4:00 – 5:00)

An Evaluation of Infrastructure Development in Support of Emerging Investigators in Health Disparities Research in Hawai`i.
Susana Helm, University of Hawaiʻi at Mānoa (UHM); Ryan Tolman, Social Science Research Institute (SSRI)/UHM; Stephanie Nishimura, UHM; Nancy Marker, SSRI/UHM; Adam Ayers, SSRI/UHM; Judith Inazu, SSRI/UHM

John A. Burns School of Medicine at the University of Hawai`i is one of five Research Centers in Minority Institutions (RCMIs) funded by the National Institutes of Health for its infrastructure development for translational health research to eliminate health disparities.  The JABSOM initiative is referred to as RMATRIX, an acronym for RCMI Multidisciplinary and Translational Research Infrastructure Expansion.  A cross-cutting emphasis among RCMIs nationally is to increase the number of minority and other scientists with research capacity in the biomedical sciences.  This poster describes an evaluation of RMATRIX in terms of its infrastructure development in support of such emerging investigators.

Evaluation of a Mentoring Program for Developing a Critical Mass of Biomedical Researchers
Jane Onoye, University of Hawaiʻi at Mānoa (UHM); Stephanie Nishimura, UHM; Judith Inazu, UHM; Gina Cardazone, UHM

Mentoring is key in developing a critical mass of biomedical research scientists. The mentoring component of an ongoing grant was evaluated to determine if program activities were implemented as planned to achieve goals and objectives. Mentee and mentor surveys examined the nature of paired relationships and concordance between mentee-mentor perceptions and experiences. Despite the lack of formal mentoring plans and variable mentee participation in required activities, mentors were generally positively rated. Both mentors and mentees agreed that scientific and career mentoring was greater than psychosocial aspects; however, mentors tended to have more confidence in mentees regarding their future career accomplishments.

Effects of a Summer Ethnomathematics Institute for Undergraduates in Hawai'i
Joanna Philippoff, Curriculum Research and Development Group (CRDG)/University of Hawaiʻi at Mānoa (UHM); Lisa M. Vallin, CRDG/UHM; Linda Furuto, UH-West Oʻahu

We present findings from an evaluation of an ethnomathematics–the study of the relationship between mathematics and culture– summer institute, a two-week course for undergraduates throughout the University of Hawaii system. The program was designed to address issues of retention, support, and advancement in STEM majors. We used a mixed-methods approach to examine the association between program participation and student outcomes. In addition to presenting the study’s findings, we will elaborate on the evaluation process, sharing the challenges and successes of evaluating a transdiscipline program that links history and culture to 21st century problems and skills.

The Planning and Implementation of an Evaluation of the Teaching Science as Inquiry (TSI) Aquatic Professional Development Program
Joanna Philippoff, Curriculum Research and Development Group (CRDG)/University of Hawaiʻi at Mānoa (UHM); George Harrison, CRDG/UHM; Kanesa Duncan Seraphin, CRDG/UHM; Paul B. Brandon, CRDG/UHM

The Teaching Science as Inquiry (TSI) Aquatic professional development program is a series of four modules presented over the course of a year. The goals of the program are to improve student understanding of the process and content of science by increasing teachers’ content and pedagogical knowledge. We will share our evaluation plan, describe the instruments used to assess program goals, and share the results of our analyses of teachers’ content knowledge, teachers’ understanding and implementation of inquiry-based science teaching in the classroom, and changes in students’ content and nature of science knowledge.

Race to the Top Conceptual Framework - The Application of Logic Model in Evaluation
Jerry Wang, Hawai`i Department of Education (HI-DOE); John Carroll, HI-DOE; Donna Fujimoto-Saka, HI-DOE; Jerald Plett, HI-DOE

The poster illustrates how the logic model can serve as the conceptual framework for Hawaiʻi's Race to the Top project. The following three major components and their interactions will be included: 

1. Improved student learning,
2. Increased college and career readiness, and  
3. Improve student outcomes.

 

updated 8/28/2013