A case study of metrics for assessing STEM scholarship programs

As the challenges in Science, Technology, Engineering and Mathematics (STEM) education become more prominent, resources for promoting STEM education are more available. STEM scholarship programs, such as the S-STEM program funded by the National Science Foundation (NSF), can be an important tool for computing educators to meet the challenges of recruiting, retaining and preparing more computing undergraduates. These scholarship programs usually require a rigorous assessment plan in the proposals for evaluating program effectiveness. Although there is an abundance of literature on assessment metrics on student learning outcomes, especially those related to accreditation, similar papers on scholarship program assessment are lacking. This paper is intended to contribute toward filling this gap. Based on our experience on managing two consecutive NSF S-STEM grants over eight years, the paper proposes a simple set of criteria for designing good assessment metrics for scholarship programs: that they need to be essential, measurable, sensible and simple. The paper presents our evaluation plan and results of the evaluation that ascertained the program effectiveness. It discusses how the metrics are refined and evolved in light of the proposed criteria. It then elaborates on lessons learned and our future directions. The paper can thus be considered as a case study for computing educators interested in submitting scholarship proposals and managing scholarship programs.