This report describes the spring 2014 evaluation of the University of Minnesota
Duluth (UMD) home page. The report and its recommendations are endorsed by the ITSS Web Team and the Campus Accessible Technology Team.
Executive Summary
People with disabilities were more effective and successful than people without disabilities in task completion. Attitudes of people with disabilities toward the home page were more favorable than those of people without disabilities. On average people with disabilities viewed nearly the same number of pages per task than people without disabilities. However, the time-on-task level of people with disabilities was three times more than for people without disabilities.
Findings are:
- CSS luminosity contrast WCAG AA failures exist.
- Paragraph text in the carousel on top of textured background image fails WCAG AA luminosity contrast ratio.
- Small font size was problematic for test participants with motor impairments.
- Cluttered design negatively impacted task completion for people without disabilities.
- Excessive amount of scrolling was problematic for people without disabilities.
- The carousel is ineffective in presenting important content.
These issues can be resolved by increasing contrast, increasing font sizes, reducing clutter, reducing scrolling, and not using the carousel for important information or removing it.
For the task of locating the admissions form, people without disabilities took on average 28 seconds with a failure rate of 42%. They simply gave up or failed. People with disabilities had a much lower failure rate of 14% because of accessibility coding improvements made after the October 2013 test, which resulted in reduced clutter for that user group. A 42% failure rate is unacceptable as usability guideline 6.1 suggests avoiding cluttered displays in order to increase success rate. The majority of web users today are notoriously impatient and fickle - if they are frustrated, they will
quickly go somewhere else and likely not return. For a university site trying to increase enrollment, this is a serious concern.
Methodology
Two main groups tested the UMD home page: people with disabilities (Test Group A) and
people without disabilities (Test Group B).
14 people with disabilities completed the study. They were derived from Knowbility's Accessworks
database. 100% of participants described themselves as having
intermediate to advanced Assistive Technology (AT) proficiency. Table 1 provides details of this group.
26 people without disabilities completed the study. They were derived from Admissions, the ITSS TechCenter, and the Beta Bulldogs group. Table 2 provides details of this group.
Testing took place February 13 to March 17, 2014 via an accessible
online remote testing tool, Loop11. The same eight task/scenarios were presented to each participant, for example: locate the "Register for Classes link". Participants were asked the same likert scale questions to obtain ease of locating information and satisfaction levels. The testing tool recorded task completion rate, time on task, common fail pages, paths taken, and answers to questions.
On average people with disabilities, Group A were more successful (72%) than people without disabilities, Group B (56%) in task completion. Group A had abandon or fail rates of 28% whereas Group B had abandon or fail rates of 44%.
On average people with disabilities viewed nearly the same number of pages for each task than people without disabilities (3.83 versus 3.48 page views).
On average people with disabilities expended approximately 3 times more effort (168 seconds versus 52 seconds) than people without disabilities per task.
Upon completing all of the tasks, participants answered likert scale questions to rate how well phrases described the UMD home page. This subjective assessment helps to reveal how the users feel about the home page. This rating is distinct from how efficiently or effectively users actually performed tasks. As Dr. David Travis has explained,"participants tend to rate an interface highly on a post-test questionnaire even when they fail to complete many of the tasks...Once you ask participants to assign a number to their experience, their experience suddenly becomes better than it actually was."
The following are percentages of how participants answered each
question:
"Considering only the home page, for the items listed below, please indicate how well it
describes the UMD home page".
As the Department of Justice
and the Department of Education's Office of Civil Rights, stated in their Resolution Agreement South Carolina Technical College System OCR Compliance Review No. 11-11-600, "Accessible means
a person with a disability is afforded the opportunity to acquire the same information, engage in the same interactions, and enjoy the same services as a person without a disability in an equally effective and equally integrated manner, with substantially equivalent ease of use. The person with a disability must be able to obtain the information as fully, equally and independently as a person without a disability. Although this might not result in identical ease of use compared to that of persons without disabilities, it still must ensure equal opportunity to the educational benefits and opportunities afforded by the technology and equal treatment in the use of such technology."
Usability is the extent to which the page can be used by specified
users to achieve specified goals with effectiveness, efficiency
and satisfaction in a specified context of use. [ISO 9241]
Appendices
Definitions
- Accessible
- A person with a disability is afforded the opportunity to acquire the same information, engage in the same interactions, and enjoy the same services as a person without a disability in an equally effective and equally integrated manner, with substantially equivalent ease of use. The person with a disability must be able to obtain the information as fully, equally and independently as a person without a disability. Although this might not result in identical ease of use compared to that of persons without disabilities, it still must ensure equal opportunity to the educational benefits and opportunities afforded by the technology and equal treatment in the use of such technology. - [The Department of Justice and the Department of Education's Office of Civil Rights, Source - Resolution Agreement South Carolina Technical College System OCR Compliance Review No. 11-11-600.]
- Effectiveness
- The accuracy and completeness with which users achieve specified goals. - [ISO 9241]
- Efficiency
- The resources expended in relation to the accuracy and completeness with which users achieve goals. - [ISO 9241]
- Perceivable
- Information and user interface components must be presentable to users in ways they can perceive. This means that users must be able to perceive the information being presented (it can't be invisible to all of their senses). - [WCAG2]
- Operable
- User interface components and navigation must be operable. This means that users must be able to operate the interface (the interface cannot require interaction that a user cannot perform). - [WCAG2]
- Satisfaction
- Freedom from discomfort, and positive attitudes towards the use of the product. - [ISO 9241]
- Usability
- The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use. - [ISO 9241]
- WCAG
- The W3C Web Content Accessibility Guidelines 2.0 (WCAG), level AA, serve as Web accessibility standards for the University of Minnesota. For more information consult, Effective January 1: New University Web Accessibility Standard.
Participants
Table 1: Participant Profile (people with disabilities test group A)
Participant |
Browser |
Operating System |
Assistive Technology (AT) |
AT Proficiency |
Disability |
Participant 1 |
Safari |
Apple |
Voice Over |
Advanced |
Blind |
Participant 2 |
Firefox |
Windows |
JAWS 15 |
Advanced |
Blind |
Participant 3 |
Internet Explorer |
Windows |
JAWS 14 |
Advanced |
Blind and 30% hearing loss |
Participant 4 |
Internet Explorer |
Windows |
JAWS 14 |
Advanced |
Blind |
Participant 5 |
Internet Explorer |
Windows |
JAWS 15 |
Advanced |
Blind |
Participant 6 |
Chrome |
Windows |
Specialized keyboard/mouse |
Intermediate |
Motor Impairment: Lack of dexterity of the hands |
Participant 7 |
Internet Explorer |
Windows |
JAWS 15 |
Advanced |
Blind |
Participant 8 |
Internet Explorer |
Windows |
JAWS |
Intermediate |
Blind |
Participant 9 |
Internet Explorer |
Windows |
JAWS 14 |
Advanced |
Blind |
Participant 10 |
Internet Explorer |
Windows |
A communication device with an internal tracker |
Advanced |
Motor Impairment |
Participant 11 |
Chrome |
Windows |
On Screen Keyboard |
Advanced |
Motor Impairment |
Participant 12 |
Firefox |
Windows |
Zoomtext screen magnifier/reader |
Intermediate |
Low Vision |
Participant 13 |
Internet Explorer |
Windows |
JAWS |
Advanced |
Low Vision |
Participant 14 |
Internet Explorer |
Windows |
N/A |
Intermediate |
Low Vision |
Table 2: Participant Profile (people without disabilities, test group B)
Participant |
Browser |
Operating System |
Group |
Participant 1 |
Chrome |
Apple |
Prospective Student |
Participant 2 |
Firefox |
Windows |
TechCenter Student, BetaBulldog |
Participant 3 |
Internet Explorer |
Other |
TechCenter Student, BetaBulldog |
Participant 4 |
Firefox |
Windows |
TechCenter Student, BetaBulldog |
Participant 5 |
Chrome |
Apple |
TechCenter Student, BetaBulldog |
Participant 6 |
Firefox |
Apple |
Prospective Student |
Participant 7 |
Internet Explorer |
Windows |
TechCenter Student, BetaBulldog |
Participant 8 |
Chrome |
Windows |
TechCenter Student, BetaBulldog |
Participant 9 |
Firefox |
Apple |
TechCenter Student, BetaBulldog |
Participant 10 |
Chrome |
Windows |
TechCenter Student, BetaBulldog |
Participant 11 |
Firefox |
Windows |
TechCenter Student, BetaBulldog |
Participant 12 |
Internet Explorer |
Windows |
TechCenter Student, BetaBulldog |
Participant 13 |
Chrome |
Apple |
TechCenter Student, BetaBulldog |
Participant 14 |
Safari |
Apple |
TechCenter Student, BetaBulldog |
Participant 15 |
Chrome |
iOS |
TechCenter Student, BetaBulldog |
Participant 16 |
Chrome |
Windows |
TechCenter Student, BetaBulldog |
Participant 17 |
Firefox |
Windows |
TechCenter Student, BetaBulldog |
Participant 18 |
Chrome |
Windows |
TechCenter Student, BetaBulldog |
Participant 19 |
Firefox |
Apple |
TechCenter Student, BetaBulldog |
Participant 20 |
Chrome |
Apple |
TechCenter Student, BetaBulldog |
Participant 21 |
Safari |
iOS |
TechCenter Student, BetaBulldog |
Participant 22 |
Chrome |
Apple |
Prospective Student |
Participant 23 |
Safari |
Apple |
TechCenter Student, BetaBulldog |
Participant 24 |
Chrome |
Windows |
TechCenter Student, BetaBulldog |
Participant 25 |
Safari |
Apple |
TechCenter Student, BetaBulldog |
Participant 26 |
Firefox |
Apple |
BetaBulldog |
Group A: Perceptions of What is Especially Good
The following are participant responses to the question (q32): "What did you find especially good about the UMD home page?"
it worked great with my technology Group A, Participant1
easy to find headings for the different parts of the universityGroup A,, Participant 2
Could navigate by headingsGroup A, Participant 3
Easy to find general info on applying, costs, housing, campus life.Group A, Participant 4
I could use the search engine to locate most of the tasks I was asked to complete.Group A, Participant 5
Home page looks clean and simple. Easy to navigate links.Group A,, Participant 6
The information was presented in a very clear manner. Links went where they indicated that they would, and topics were well organized. Someone put a lot of thought into designing this for best practices...Group A, Participant 7
the search feature makes it easy to find what you wantGroup A, Participant 8
It was extremely accessible. Screen elements were labelled logically and concisely. The interface was not at all cluttered, had very clear and concise information. The best part was, all required information was easy to get to, within one to two steps, starting from the homepage. That, in my opinion, makes the website appealing and that is something that most college/university websites lack. The homepage had just the right blend of links and text. It is very well-organized and has a good structure.Group A, Participant 9
Pretty picturesGroup A, Participant 10
Everything was labeled good.Group A, Participant 11
Lots of direct links to useful information or if not directly to info needed, clear where info would be found within the section selected.Group A, Participant 12
There is a plethora of information relative to multiple campus areas that is easy to locate and access.Group A, Participant 13
I thought having links like the email was exceptional because usually we don't have tons of time to check email and instead, it's right there as soon as you open the site.Group A, Participant 14
Group B: Perceptions of What is Especially Good
The following are participant responses to the question (q30): "What did you find especially good about the UMD home page?"
It had many quick links to important information, especially because you can choose what you're looking for (newly admitted, current students etc.)Group B, Participant 1
Since i am a current student i could find everything with easy, and since i work in admissions. i would not expect prospective students and parents to navigate it wellGroup B, Participant 2
It is appealing to people who may be interested in the school. Also it looks very professionalGroup B, Participant 3
There's many different boxes and buttons for different areas including newly admitted students and current students, along with faculty & staff. Also, it has a very young and modern look to it which appeals to prospect students.Group B, Participant 4
I think that it works well for prospective students to find information.Group B, Participant5
The home page is very well-organized! The other pages are less organized once you leave the home page, however.Group B, Participant 6
I like the links at the very bottom of the page. Those are the ones I use.Group B, Participant 7
The site seems to be more clean in style.Group B, Participant 9
Certain tabs are clear about where information can be found. Definitely not all tabs though.Group B, Participant 10
Color schemeGroup B, Participant 11
It was easy to navigate around.Group B, Participant 12
I was able to find most things relatively easilyGroup B, Participant 13
I love the colors, it makes it appealing and having everything listed on the bottom.Group B, Participant 14
The links at the bottom can lead to every important websiteGroup B, Participant 15
all the links at the bottom. quick links are awesome.Group B, Participant 16
The search bar almost always elicited the results that were desired.Group B, Participant 17
The topGroup B, Participant 18
The easy linksGroup B, Participant 19
It gives off a more complete vibe of the website overall, as it gives easy links to topics that most people will be looking for (i.e. Admissions, Academics etc.). Through this, the easy flow/transitions between webpage and pictures, it makes it seem more complete than other sites.Group B, Participant 22
attempting to make things easier to findGroup B, Participant 23
It's better than the old homepage, I like the pictures, and how everything is a little more open and spread out.Group B, Participant 24
Inclusive by showcasing diverse groups of people and events, creating a welcoming feel for people from non-dominant culturesGroup B, Participant 26
Group A: Perceptions of What is Frustrating
The following are participant responses to the question (q33): "What did you find frustrating about the UMD home page?"
nothingGroup A, Participant 1
not muchGroup A, Participant 2
The Sochi Olympics information in the middle of the page.Group A,, Participant 3
Finding the housing application, not possible to find database info.Group A, Participant 4
nothing that I can think of.Group A, Participant 5
I wish buttons "myU" and One Stop" will be a little large or focused. Hard to see them.Group A, Participant 6
In my opinion, I found nothing frustrating about this page. I look at pages all day for a living, and this one really ROCKS!!!Group A, Participant 7
the fact that it's hard to find what you need by navigating the links on the page Group A, Participant 8
Nothing in particular.Group A, Participant 9
The web pages loaded very slowly and the font is too smallGroup A, Participant 10
Small fontsGroup A, Participant 11
Lots of info, scrolling to see everything was a bit of a pain. Seemed like there was duplication of information with links at the bottom of the page the same as links in the middle of the page.Group A, Participant 12
Some of the links that I accessed took a few attempts to activate using the screen reader. I used the shortcut keystroke to access the links list and ended up having to resort to an alternative access method of locating the link on the page and pressing enter.Group A, Participant 13
Group B: Perceptions of What is Frustrating
The following are participant responses to the question (q33): "What did you find frustrating about the UMD home page?"
The search function is frustrating to use and often doesn't help.Group B, Participant 1
directory is in the very bottom corner. this to focused on the design, not the information people are actually trying to find.Group B, Participant 2
The main information is all at the bottom so every time i need to get somewhere i have to scroll all the way down.Group B, Participant 3
To acquire the "departments" button, you have to click on a page first, and then it will pop up.Group B, Participant 4
It is not set up as well for current students. I think there could be a little bit more of a balance between the two sides, such as not having to scroll all the way to the bottom for current student links or remembering the URLs because of the some times confusing and convoluted ways currents students have to access what they need.Group B, Participant 5
After leaving the home page, the website become much less organizedGroup B, Participant 6
The email button never works for me.Group B, Participant 7
If you do not know where to exactly look, it can be a bit confusingGroup B, Participant 8
The audience for the page is aimed at everyone. It needs to be solidified more to one group otherwise it does its job more poorly.Group B, Participant 9
You have to click several links in order to register for classes. Takes too much time. Also, to apply at UMD, you have to search for the link instead of one being right on the homepage.Group B, Participant 10
I don't find the majority of the pictures and interest stories relevant as a student or prospective student. Solution: band-aid style, would be to add a link on upper banner with "Most Common Links" brings to new page which is a series of the most common links used.Group B, Participant 11
Nothing.Group B, Participant 12
Some sections are just too large - especially the top piece which is nothing but advertising and of no real use.Group B, Participant 13
There was only one thing I couldn't find and I probably wouldn't have known how to find it on any site.Group B, Participant 14
Some things were hard to findGroup B, Participant 15
the split second it takes for everything to load and forces me to scroll back to the top of the page. Normally I just want to scroll to the bottom to get a quick link, but the homepage loads all the applets and everything which takes an extra second or two and then the page snaps me back to the topGroup B, Participant 16
If you searched for Gmail, only ITSS links showed as the top results.Group B, Participant 17
So many different linksGroup B, Participant 18
Sometimes hard to find certain thingsGroup B, Participant 19
ScrollingGroup B, Participant 20
Way too cluttered.Group B, Participant 21
Not much, the whole site flows together rather smoothly.Group B, Participant 22
long, due to having lots of info. Not sure this is good or bad.Group B, Participant 23
The 6 buttons at the top don't always help me find what i'm looking for.Group B, Participant 24
I had a hard time finding certain statistics regarding different ethnicities.Group B, Participant 25
Not intuitive, cluttered, randomly organized/layed out. While I am a visual person and like pictures, these dominant without serving a useful purpose. For example, the "Those who can" banner and "Skinny on Fat" image of squirrel take up the whole of my screen (along with the list of "about UMD, admissions, academics" etc. The list is helpful the images and the large banner are not. The links that are more useful are at the bottom of the page and require scrolling quite a lot to get to them.Group B, Participant 26
Group A: Participant Suggestions for Change
The following are responses to the question (q34): "If you could make one significant change to the UMD home page, what change would it be?"
put certain things in columns such as costs and who to contact.Group A, Participant 1
add a link for housing and put departments under collegesGroup A, Participant 2
Did get rid of the olympics carouselGroup A, Participant 3
More straightforward links to tuition and housing applications.Group A, Participant 4
nothing at this time.Group A, Participant 5
I would make large or more attractive buttons "My U" and "One Stop". Hard to see them. I also would make shorter drop down menu, looks weird empty area under links. Also need better divide areas for example under drop down menu "Admissions" words "Graduate" & Under Graduate etc.Group A, Participant 6
More pizza places listed!!Group A, Participant 7
I think the page is quite organized and accessible as it is.Group A, Participant 9
Font sizeGroup A, Participant 10
Font sizesGroup A, Participant 11
Fit everything on the page so you didn't have to scroll and change the order that sections are listed on the home page for the most accessed at the top.Group A, Participant 12
I would put financial aid/tuition links on the page for easy access.Group A, Participant 13
I think the page is very clear as is.Group A, Participant 14
Group B: Participant Suggestions for Change
The following are responses to the question (q32): "If you could make one significant change to the UMD home page, what change would it be?"
The home page doesn't need that much work. It would be nice to add a mobile ability but the desktop version does a very good job.Group B, Participant 1
start from scratchGroup B, Participant 2
Making it easier to get to the major pages. The main ones are at the bottom and i do not want to scroll all the way down every time.Group B, Participant 3
To add a "current students" button to the top row of buttons on the page.Group B, Participant 4
Make it more balanced between the current and prospective students.Group B, Participant 5
It was much easier to find information when the homepage was one screen long. Now I have to scroll like 5 times until I reach the bottom.Group B, Participant 7
Make it more compact and have it fit into a standard viewable screen instead of scrolling down (I'm sure many people miss that you can scroll down)Group B, Participant 9
Do not scatter current student links below. We are the ones using the page the most to register, check financial aid, etc.Group B, Participant 10
Get rid of the "Those who can" banner, a link will suffice, I want my home page to have almost all the information needed/accessed at the top without scrolling.Group B, Participant 11
Maybe making some more important things such "Register for classes" a little more easy to find just because it is a pretty important aspect to students.Group B, Participant 12
Compress itGroup B, Participant 13
Nothing.Group B, Participant 14
More helpful linksGroup B, Participant 16
Link a MyU Login to the search term, 'Gmail.'Group B, Participant 17
nothingGroup B, Participant 18
Making housing easier to find.Group B, Participant 19
Make it smaller or more compactGroup B, Participant 20
More user friendly.Group B, Participant 21
Have it focus more on letting families and students know about the college and academic life rather than upcoming events for students at the college. Though this may go against some things the site might be trying to highlight, it is a personal preference of mine.Group B, Participant 22
drop-down menus for tabs as you mouse-overGroup B, Participant 23
I'm not really sure.Group B, Participant 24
To move all of the small writing/links that are below the "UMD News" area, to the top of the page rather than the bottom. Those small links have a lot of useful information but I doubt many people take the time to scroll to the bottom where they are less noticeable.
Group B, Participant 25
Most important info and links at the top of the pageGroup B, Participant 26
Overall the page was okay. Some of the things like student email was difficult to locate. I think the carosel with the Sochi Olympic athletes and other random announcement should not appear in the screen reader navigation of the page.Group A, Participant 3
Need to check that links to department pages are working (I got thrown out of the system when I tried to access the social work program page.Group A, Participant 4
thanks for giving me the opportunity to help test the page.Group A, Participant 5
Again, someone put much thought into this design. I will recommend this page as one of the better ones that I have tested. Great job, and keep up the good work!!Group A, Participant 7
It was fun!Group A, Participant 11
Text colors and contrast is good and attractive.Group A, Participant 12
None.Group A, Participant 13
I feel the calendar should a little clearer, I was a little lost at times trying to find a specific date on the calendar.Group A, Participant 14
Some of the more obscure links need updating (housing particularly with more pictures) and clubs and stuff like that.Group B, Participant 1
Designing a website is hard and frustrating, especially with a big audience... keep up the good work!Group B, Participant 9
Nothing more.Group B, Participant 17
yeah, why are there two banners at the top of the page? It takes up too much space.Group B, Participant 24
Thanks for doing this!Group B, Participant 26
Test Material
Loop 11 Data