Learning Management System Review Summary Report and Recommendation

Northern Illinois University (NIU) conducted a comprehensive Learning Management System (LMS) review during the 2020-2021 academic year. While faculty usage and overall satisfaction with the university’s current LMS (Blackboard) remains high, the university hadn’t conducted a formal evaluation of other LMS systems in a number of years.

The review was a thorough, comprehensive effort to evaluate the primary learning management systems in the higher education market. This evaluation was broad in scope and sought input from faculty, staff and students on not only their current satisfaction with NIU's current LMS but also what features are desired in a LMS moving forward. Through NIU’s partnership with Internet2, the top three leading LMS platforms in Higher Education – Blackboard Learn, D2L Brightspace, and Instructure Canvas – were reviewed to ensure that we select a platform that will serve the core interests of the university for years to come. The evaluation supplied the data to prepare a recommendation on whether an alternative LMS should be considered at this time.

Overview and History of Blackboard at NIU

Blackboard Learn was originally licensed at NIU in 1999 to facilitate the delivery of web-based courses, but it has been used much more broadly as a critical system for facilitating teaching and learning.

Over time, other components have been added (initially individually priced but now bundled in our institutional license):

  • Blackboard Community System in 2003 – facilitates non-academic use of Blackboard
  • Blackboard Content System in 2007 – file repository for storage and sharing, within and outside of courses
  • Blackboard Collaborate in 2011 – two-way audio/video web conferencing (upgraded to Collaborate Ultra in 2016)
  • Blackboard Mobile Learn in 2011 – mobile app for student and faculty access to Blackboard on smartphones and tablets

NIU hosted Blackboard locally from 1999 to 2018, when it was migrated to the cloud to reduce costs, increase performance, and provide a path to the Ultra Experience, Blackboard’s modern and completely redesigned interface. The Ultra Base Navigation and Course View were introduced in Spring 2019. Blackboard was relied upon heavily in the pivot to emergency remote instruction in response to the pandemic.

Usage of Blackboard

Academic Courses

Blackboard is used by 95% of faculty and instructors, 99% of students, and for 73% of all course sections (fall 2020). A majority of NIU courses each semester make extensive use of the capabilities of the system. Blackboard integrates with other cutting-edge tools for teaching and learning including video applications, publisher platforms, exam proctoring, and student engagement systems.

Figure 1: NIU Students & Faculty with at Least 1 Course in Blackboard, Fall Semester, 2002-2020

percentage of NIU students and faculty with at least 1 course in Blackboard, 2002-2020

Figure 2: Use of Blackboard by Individual Students, 2002-2020

number of individual students using Blackboard at NIU, 2002-2020

Figure 3: Use of Blackboard by Individual Faculty/Staff/TAs to Teach, 2002-2020

number of individual instructors of record using Blackboard to teach at NIU, 2002-2020

Figure 4: Course Sections on Blackboard, 2002-2020

number of NIU course sections on Blackboard, 2002-2020

Figure 5: Percentage of Course Sections Taught Using Blackboard (since 2010)

percentage of NIU course sections taught using Blackboard since 2010

Figure 6: Top 10 Tools Used in Blackboard Courses

top 10 tools used in Blackboard courses

Additional Blackboard use stats are available.

Other Uses

In addition to scheduled academic courses, Blackboard is used in a variety of ways to support the mission of the university:

  • Departmental communication to students by major/minor on events and program deadlines
  • Development and storage of course materials for upcoming and previous semesters
  • Student portfolios for program assessment and supporting job searches
  • Department and programmatic assessment across courses, including general education assessment
  • Storage and delivery of electronic reserves using a process custom-developed by the University Libraries and Division of Information Technology
  • Annual compliance training for students, faculty, and staff, including state-mandated ethics and Title IX training
  • Delivery and tracking of training completion for employees including Card Services, Division of Information Technology, Recreation, and Holmes Student Center

Satisfaction with Blackboard

NIU faculty and instructors were surveyed in 2016 and 2019 to gauge satisfaction and comfort with Blackboard. In 2016, 83% of faculty strongly agreed or agreed that they were satisfied with Blackboard and 82% responded that they were confident using Blackboard (these questions were not repeated in 2019). From 2016 to 2019, other measures of confidence and satisfaction were relatively consistent or increased, including finding that Blackboard enhanced teaching effectiveness and helped students to be more successful.

2016 2019
I am generally satisfied with the quality of Blackboard 83% N/A
I feel confident using Blackboard features in general 82% N/A
I have a generally favorable attitude toward using Blackboard 80% 79%
Using Blackboard increases my efficiency when teaching 72% 75%
I find Blackboard to be easy to use 67% 68%
Using Blackboard enhances my teaching effectiveness 64% 69%
Blackboard helps students to be more successful in their courses 62% 65%

For more details on reported faculty use and satisfaction with Blackboard, see complete reports of past Blackboard usage surveys.

Review Task Force

In October 2020 Executive Vice President and Provost Ingram authorized the LMS review and for it to be jointly coordinated by the Center for Innovative Teaching and Learning (CITL) and the Division of Information Technology (DOIT). Given the widespread and longstanding use of the LMS, a task force with broad and inclusive representation was formed with members nominated by colleges and stakeholder units from across the university, consisting of the following:

Executive Committee

Responsible for execution of the review, including the solicitation of feedback, development of a communications plan, and presentation of a final report. The executive committee met weekly and consisted of the project co-chairs, project manager, application services senior director, one member from the technical committee, two members from the advisory committee, and procurement representative.

  • Project Co-chair – Matthew Parks, Associate Vice President and Chief Information Officer, DOIT
  • Project Co-chair – Jason Rhode, Executive Director, CITL
  • Project Manager – Susan Smith, Projects Coordinator, Institutional Effectiveness
  • Application Services Representative – Nick Choban, Senior Director, Application Services, DOIT
  • Technical Committee Representative – Daniel Ihm, Director, Application Development and Support, DOIT
  • Co-chair of Advisory Committee – Therese Arado, Acting Director of the Law Library and Professor
  • Co-chair of Advisory Committee – Stephanie Richter, Director of Teaching Excellence and Support, CITL
  • Procurement Representative – Antoinette Bridges, Director, Procurement Services and Contract Management

Advisory Committee

Representatives from across university stakeholder groups who provided input into the evaluation process and establishment of evaluation review criteria. The advisory committee met on an as-needed basis and was responsible for seeking and incorporating campus feedback on the process, responding to draft reports, and supporting the communications plan. The technical committee served as a subgroup of the advisory committee.

  • College of Business Faculty Representative – Gina Nicolosi, Professor and Chair, Department of Finance
  • College of Education Faculty Representative – Wei-Chen Hung, Professor and Chair, Department of Educational Technology, Research and Assessment
  • College of Engineering and Engineering Technology Faculty Representative – Gary Chen, Professor, Department of Industrial and Systems Engineering
  • College of Health and Human Sciences Faculty Representative – Hamid Bateni, Associate Professor, School of Allied Health and Communicative Disorders
  • College of Liberal Arts and Sciences Faculty Representative – Artemus Ward, Professor, Department of Political Science
  • College of Law Faculty Representative – +Therese Arado, Acting Director of the Law Library and Professor
  • College of Visual and Performing Arts Faculty Representative – Kelly Gross, Assistant Professor, School of Art and Design
  • University Libraries Faculty Representative – Alissa Droog, Assistant Professor, Education and Social Sciences Librarian
  • Instructor Representative – Federico Bassetti, Instructor, Department of Operations Management and Information Systems
  • Instructor Representative – Joyce Laben, Instructor, Department of Leadership, Educational Psychology and Foundations
  • Instructor Representative – Candace Medina, Instructor, Department of World Languages and Cultures
  • Student Success Representative – LaBrian Carrington, Academic Advisor/Counselor, Academic Advising Center
  • Student Affairs Representative – Amanda Newman, Associate Director, Disability Resource Center
  • Staff Representative – Kelsey Johnson, Senior Director, Operations and Financial Strategy, College of Business
  • Staff Representative – Deirdre Young, Acting Associate Director, Registration and Records
  • Student Representative – Antonio Johnson, President, Student Government Association
  • Student Representative – Marissa Dordick
  • Student Representative – Gaylen Rivers
  • Graduate Student Teaching Assistant – Bailey Zo Kreager, Department of Geology and Environmental Geosciences
  • Communications Representative – Holly Nicholson, Web Team Manager, Division of Enrollment Management, Marketing and Communications
  • Division of Outreach, Engagement and Regional Development Representative – Amy Jo Clemens, Director, Center for P-20 Engagement

+ = Advisory Committee Co-chair

Technical Committee

Responsible for operationalizing the review of systems, engagement with key stakeholders, and collection of evaluation data.

  • DOIT Representative – Daniel Ihm, Director, Application Development and Support
  • DOIT Representative – Shawn Staggs, Applications Programmer Analyst
  • CITL Representative – +Stephanie Richter, Director of Teaching Excellence and Support
  • CITL Representative – Jason Underwood, Director of Instructional Design and Development
  • CITL Representative – Tracy Miller, Assistant Director

Additional Internal Engagement

Beyond the aforementioned individuals who served collectively as the LMS review task force, the following internal stakeholder groups were also engaged throughout the review process and provided with status updates along with opportunities for feedback.

Review Criteria and Scoring

Review Scoring

The following scoring system was established, with scores weighted according to the following maximum scores:

Review Elements  Points Possible
Responses to questions from task force 100
Technical review of the criteria, based on technical committee's testing in a pilot server 200
Campus feedback (direct surveys, focus groups) 250
Pricing, contracted 200
Pricing, transition (to include maintaining multiple licenses, training, etc.) 50
Vendor support for transition and training 100
Roadmap 100
Total Weighted Points Possible 1,000

Technical Review Criteria

The LMS Review Task Force identified and established technical review criteria, organized into the following categories:

Additional Criteria and Scoring

The following additional criteria and scoring elements were similarly developed for use during the review and shared:

Review Process and Timeline

The review process was comprehensive and transparent, with detailed criteria and metrics for evaluation and scoring. The executive committee met with each vendor to provide an overview of NIU’s review process and topics for inclusion in campus demonstration presentations. Subsequent steps in the review included:

  1. Each vendor met with the executive committee to provide background into their history as a company, general overview of solution, and roadmap ahead -- information that provided additional context in advance of campus demonstrations.
  2. Each vendor provided a test instance of their platform for technical review and access to existing documentation; technical committee then reviewed the systems and rated each according to the full list of technical criteria.
  3. Executive committee and technical committee requested written responses to additional questions based on the technical review.
  4. Each vendor provided two virtual demonstrations for faculty, staff, and students to attend on functionality for teaching and learning. Demonstrations were recorded and made available for NIU users to view on the LMS review website. Participants completed follow-up surveys to rate each system’s functionality.
  5. Each vendor provided separate virtual demonstrations of shopping cart features for outside learners and external programming. Participants similarly completed follow-up surveys to rate the system’s functionality.
  6. Technical committee built sample courses in each system with identical content and conducted focus group sessions with advisory committee and other NIU constituent groups. Focus groups compared the courses from each system and rated the systems’ functionality.
  7. Each vendor provided references of client institutions comparable to NIU for the executive committee to reach out to for their institutional experience and feedback.

Recommendation

Of the three LMS platforms reviewed (Blackboard Learn, D2L Brightspace, Instructure Canvas), Blackboard Learn received the highest review score and is recommended for renewal for another 5-year contract term.

Frequently Asked Questions

Why was Blackboard selected as our LMS to use going forward?

While each system has strengths and weaknesses, Blackboard received the highest review score, taking into account all the review elements. The overall scoring included feedback from faculty, instructors, staff and students; technical review; pricing (both for transition and multi-year contract); roadmap of new feature development; and vendor support for transition and training.

Continuing with Blackboard allows faculty to be thoughtful and deliberate about transitioning to a new system on a longer timeframe. Blackboard Learn Ultra has a robust feature set that is being added to rapidly over the next year, and Blackboard provides enterprise-level security and reporting options we need as a large institution. Staying with Blackboard is also the most affordable solution to the university and provides the most value to our students and faculty.

In choosing to stay with Blackboard as our institutional LMS, does that mean we can continue using Blackboard Original?
Yes, those who today use the Blackboard Original course experience for their teaching at NIU are able to continue using Blackboard Original into the future. There will however come a point and time when Blackboard Original will no longer be supported or receive any future feature enhancements and all Blackboards users will eventually have to transition to Blackboard Ultra. We do not have a fixed date when this transition would need to happen, but it will need to happen in the future. The Original Course View will be available at NIU through May 2023, at a minimum.
How long will NIU faculty be able to continue teaching with the Original Course View in Blackboard?

We strongly recommend faculty transition to the Ultra Course View as soon as they are comfortable doing so, with the support of the Center for Innovative Teaching and Learning (CITL). Faculty can learn more by attending a workshop or taking advantage of a CITL partnership to work one-on-one with a member of their team. Blackboard course templates are also available that provide a quick start for building your course in Ultra.

The Original Course View will become unavailable at some point, and all NIU faculty and instructors will have to adopt the Ultra Course View at that time. Right now, that date has not yet been determined. The Original Course View will be available at NIU through May 2023, at a minimum.

The Division of IT and the Center for Innovative Teaching and Learning will be working closely with their advisory and governance committees as well as with Blackboard to identify an appropriate timeline for sunsetting Blackboard Learn Original.

What is Blackboard Ally?

NIU’s new contract with Blackboard includes adoption of Ally, a revolutionary product that integrates seamlessly into Blackboard and focuses on making digital course content more accessible. As the instructor adds course content, the content passes through an accessibility checklist and is scored, and machine learning algorithms perform a full structural and visual analysis. Alternative formats are automatically generated for students and the instructor is provided feedback on how to ensure future content is as accessible as possible.

Download Printable Report (PDF)

Creative Commons License
Northern Illinois University LMS Review Criteria and Scoring are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Back to top