Vendor claims of accessibility are not reliable without being confirmed through written documentation and validation. The commonly accepted method of reporting accessibility is through Accessibility Compliance Reports (ACRs). There are also called Voluntary Product Accessibility Templates (VPATs). A VPAT is a generic template that companies use to draft reports on their product’s accessibility, based on either Section 508 (American) or EN 201 549 (European) standards.
For most commercial off-the-shelf products, VPATs are available online. Most major vendors whose customers include post-secondary institutions in the United States or Europe will have previously completed VPATs and will be able to share them.
It is important to note that a VPAT is not an audit report, though the vendor must have performed an audit to complete the VPAT. While useful, VPATs cannot be assumed to be accurate. Since they are written by the vendor, VPATs may exaggerate the accessibility of any given product.
Red Flags
The following are some items for concern or indications that a VPAT may not be accurate:
- An old version of a VPAT (VPAT 1.X) indicates that the vendor has not updated its documents. Similarly, a VPAT dated more than 12 months ago would suggest that the product has not been updated.
- The VPAT is not completed or is completed with non-standard language (e.g., it uses “Passes” and “Fails” rather than “Supports”, “Partially Supports”, or “Does Not Support”). This indicates a lack of expertise in accessibility by the author.
- One VPAT for multiple products or inaccurate/ unclear descriptions of the product. A VPAT needs to be product-specific to be accurate.
- Evaluation methods are unstated or indicate that only automated testing was completed. Automated testing can only test for approximately 30% of conformance, but it cannot be relied upon alone.
- The VPAT states that it “Supports” all accessibility specifications. While this is possible, it is incredibly rare for a product to support every criterion. It may indicate that the author did not understand or confirm compliance. Note that a VPAT may say “Supports” instead of “Not Applicable” where a criterion is not relevant.
- VPATs that consistently state “Supports” and often includes minimal content in the “Remarks” column. The Remarks column is not intended to be blank. It should provide information on how the authors arrived at this conclusion. If most responses in the VPAT paraphrase the criterion, question the authors on how they reached those conclusions.
- Authors may also put “Not Applicable” for many criteria. While this may be an appropriate response, it should be accompanied by a brief explanation. “Not Applicable” responses without explanation are suspect as they make it challenging to determine why the criterion does not apply.
If you encounter red flags
If any of these red flags are raised when reviewing a VPAT, you can do some quick tests to confirm the accuracy of the claims in the VPAT. This is not a substitute for a full accessibility audit but it can validate some claims in a VPAT. For example, you could do the following tests and check that the results are accurately described in the VPAT:
- Test the product with the keyboard (see 4.5.2 if you’re not sure what this entails).
- Are all functional items available by keyboard?
- Do all active items have a highly visible keyboard focus indicator?
- Can you activate hidden content like menus?
- Can you complete and submit forms?
- Is there movement on the site? Does it have an accessible (mouse, keyboard, touch) pause feature, or does it stop within 5 seconds?
- If you submit an empty or incorrect form (e.g., a log-in), are the errors described appropriately?
If the results are not consistent with the VPAT, the VPAT is not reliable. In that case, you cannot know if the product is accessible without testing it yourself.
If questions arise from the VPAT, contact the vendor. You may want to ask questions like:
- Why is this VPAT dated 18 months ago, has the product not been updated?
- Why did this criterion say Not Applicable when the product has that feature?
- What testing was done to reach these results?
Signs of a Reliable VPAT
In contrast, the following are positive indications of the reliability of a VPAT:
- It was authored by a third party, with greater expertise in the accessibility criteria and are more likely to have an objective view.
- There is substantial content on the VPAT’s cover page, including detailed information about the exact type of testing employed. For example, listing multiple assistive technologies and types confirms the rigor of the testing. Similarly, a note about the scope of testing is reassuring.
- Examples of how the product meets specific success criterion demonstrates an understanding of the success criteria.
- The VPAT is publicly available, which provides an indication of the vendor’s confidence in the report.
