AI Tool Evaluation Rubric
The NIU AI Tool Evaluation rubric was developed by a working group of the Task Force on AI in Academic Affairs to guide the selection of academic student support services AI tool(s), reflect NIU's values, and meet the needs of students, faculty and staff.
Values
- Curiosity and Creativity: Innovation and commitment to active learning and enhancing critical thinking.
- Equity and Inclusion: diversity and inclusion, environmental sustainability, bias
- Ethics and Integrity: Transparency, making sources visible, clarity with users about AI implementation, accuracy, IP considerations. data collection and protection, user ability to opt-out
- Service and Stewardship: integrity of the service unit and their ability to deliver quality services
- Fiscal Responsibility: budget considerations, efficiency
- Alignment: Alignment with institutional goals, student needs, and the needs of support units
- Human-centeredness: Usefulness of the AI to human endeavor, labor considerations, including people in AI—i.e., human-in-the-loop, concern for harms
Rubric
Adapted from Mannheimer, S., Clark, J., Rossmann, D., Sheehey, B., Young, S. W. H., Bond, N., Shorish, Y., & Kettler, H. S. (2025, February 1). Viewfinder Toolkit.
| Category | Criteria | Not appropriate for NIU (0) | Meets some requirements but lacks in functional needs (1) | Meets some requirements with minor issues (2) | Meets all requirements + value-added benefits (3) |
|---|---|---|---|---|---|
| Fiscal Responsibility | Cost | Cost is prohibitively expensive | Cost is high, but there are benefits that may outweigh the cost | Cost is low and/or the benefit outweighs the cost | Cost is free and/or the benefit greatly outweighs the cost |
| Efficiency | Tool adds significant workload without reduction | Tool adds moderate workload, and somewhat frees up staff time for more impactful tasks | Tool adds some workload, but frees up staff time for more impactful tasks | Tool adds little-to-no workload and frees up staff time for more impactful tasks | |
| Ethics and Privacy | Transparency | Documentation on tool's functions and policies are not available | Documentation on tool's functions and policies are somewhat available or available with limitations | Documentation on tool's functions and policies are readily available | Documentation on tool's functions and policies are readily available; tool is open source |
| Explainability | Tool's decision making for outpot is neither inherently understandable nor logically explained within or outside of the system | Tool's decision making for output is neither inherently understandable nor logically explained within the system, though some external explanation is available | Tool's decision making for output is either somewhat inherently understandable or is somewhat explained within the system | Tool's decision making for output is either inherently understandable or is logically explained within the system | |
| Data Collection and Sharing | Does not list data that is collected and/or shared or collects and/or shares user data without explanation | Lists data that is collected and/or shared but provides little-to-no explanation | Lists data that is collected and/or shared but explanations are not always clear | Only essential data is collected and/or shared with explicit consent | |
| User Control | Users have no control over how their data is collected or shared | Users have limited ability to opt out of data collection and/or sharing | Users have the ability to opt out of non-essential data collection and/or sharing | Users have the ability to opt out of data collection and/or sharing | |
| Safety | Not compliant with NIST CSF, or lacking enough information to make a risk assessment. | Somewhat comopliant with NIST CSF. Limited risk assessment practices. | Mostly compliant with NIST CSF. Regularly audits the compliance documentation are available, though there may be gaps. | Compliant with NIST CSF. Regular audits and compliance documentation are available. | |
| Functionality | Support | Support is not available | Support is available via documentation only | Limited support is avialable from the vendor and via documentation | Robust suppot is available from the vendor and via documentation |
| Accessibility | Conformance level WGAG 2.2 AA "Partially Supports" and/or "does not support" for criteria | Conformance level WGAG 2.2 AA "Full support" for some criteria, and "Partially Supports" for others | Conformance level WGAG 22 AA "Full support" for most criteria | Conformance level WGAG 2.2 AA "Full support" for all criteria | |
| Operating Systems | Works with few operating systems and/or does not work on mobile or other devices | Works with most operating systems and devices, but functionality is impacted by operatingh system and/or device | Works with most operating systems and across devices | Works with all major operating systems and across devices | |
| Human-centeredness | Removes or discourages human interaction | Maintains human interaction, but neither supplements nor detracts from it | Supplements human interaction | Enhances human interaction | |
| Environmental and Social Responsibility | Environmental impacts | No commitment to sustainability, demonstrated negative environmental impact | No commitment to reducing environmental impact | Commitment to reducing environmental impact | Commitement to Green AI with investment in sustainability |
| Bias | No consideration for bias | Minimal acknowledgement of bias | Some acknowledgement and mitigation of biases | Active mitigation of bias through design elements and education | |
| Equity and Inclusion | No evidence of support or features designed for diverse users | Little evidence of support or features designed for diverse users | Some inclusive elements designed for diverse users | Actively addresses diverse user needs through design features such as support for non-English languages, flexibility and choice | |
| Notes: Describe how well the tool could meet the needs of students and student support units | |||||
This framework is shared under a Creative Commons Attribution-NonCommercial 4.0 International License.