VAST Annotation Usability Evaluation

– Step 1 of 3

Demographics

Please indicate your gender *

Please indicate your age: *

25 years or less
26-35 years
36-45 years
46-55 years
56-65 years
66 years or more
Prefer not to specify

Please indicate your highest completed education degree from this list: *

Pre-university
Bachelor or equivalent
Master or equivalent
PhD or equivalent

Please indicate your field of expertise in your own words: *

Do you have any previous annotation experience: *

None
I’ve done it once or twice
It happens to me to do it occasionally
I regularly engage in text annotation

Please indicate your technical knowledge: *

I can search information on the Internet
I can write and format a document with MS Word
I can identify the sources in a piece of news I am reading, or confirm that these are not present
I can look up vocabulary that I don’t understand in online dictionaries or thesauri
I can upload an image on a social network

Check all that apply

Did you feel that your level of English was satisfactory for the needs of the task: *

Adapted System Usability Scale

Likert Scale *

	Strongly disagree	Somewhat disagree	Neither agree nor disagree	Somewhat agree	Strongly agree
I think I would like to use this annotation tool	Strongly disagreeI think I would like to use this annotation tool Strongly disagree	Somewhat disagreeI think I would like to use this annotation tool Somewhat disagree	Neither agree nor disagreeI think I would like to use this annotation tool Neither agree nor disagree	Somewhat agreeI think I would like to use this annotation tool Somewhat agree	Strongly agreeI think I would like to use this annotation tool Strongly agree
I found the tool unnecessarily complex	Strongly disagreeI found the tool unnecessarily complex Strongly disagree	Somewhat disagreeI found the tool unnecessarily complex Somewhat disagree	Neither agree nor disagreeI found the tool unnecessarily complex Neither agree nor disagree	Somewhat agreeI found the tool unnecessarily complex Somewhat agree	Strongly agreeI found the tool unnecessarily complex Strongly agree
I though the tool was easy to use	Strongly disagreeI though the tool was easy to use Strongly disagree	Somewhat disagreeI though the tool was easy to use Somewhat disagree	Neither agree nor disagreeI though the tool was easy to use Neither agree nor disagree	Somewhat agreeI though the tool was easy to use Somewhat agree	Strongly agreeI though the tool was easy to use Strongly agree
I think that I would need further technical support to be able to use the tool	Strongly disagreeI think that I would need further technical support to be able to use the tool Strongly disagree	Somewhat disagreeI think that I would need further technical support to be able to use the tool Somewhat disagree	Neither agree nor disagreeI think that I would need further technical support to be able to use the tool Neither agree nor disagree	Somewhat agreeI think that I would need further technical support to be able to use the tool Somewhat agree	Strongly agreeI think that I would need further technical support to be able to use the tool Strongly agree
I found the various functions of the tool well integrated	Strongly disagreeI found the various functions of the tool well integrated Strongly disagree	Somewhat disagreeI found the various functions of the tool well integrated Somewhat disagree	Neither agree nor disagreeI found the various functions of the tool well integrated Neither agree nor disagree	Somewhat agreeI found the various functions of the tool well integrated Somewhat agree	Strongly agreeI found the various functions of the tool well integrated Strongly agree
I thought there was too much inconsistency in this tool	Strongly disagreeI thought there was too much inconsistency in this tool Strongly disagree	Somewhat disagreeI thought there was too much inconsistency in this tool Somewhat disagree	Neither agree nor disagreeI thought there was too much inconsistency in this tool Neither agree nor disagree	Somewhat agreeI thought there was too much inconsistency in this tool Somewhat agree	Strongly agreeI thought there was too much inconsistency in this tool Strongly agree
I would imagine that most people would learn to use this tool very quickly	Strongly disagreeI would imagine that most people would learn to use this tool very quickly Strongly disagree	Somewhat disagreeI would imagine that most people would learn to use this tool very quickly Somewhat disagree	Neither agree nor disagreeI would imagine that most people would learn to use this tool very quickly Neither agree nor disagree	Somewhat agreeI would imagine that most people would learn to use this tool very quickly Somewhat agree	Strongly agreeI would imagine that most people would learn to use this tool very quickly Strongly agree
I found the tool very cumbersome to use	Strongly disagreeI found the tool very cumbersome to use Strongly disagree	Somewhat disagreeI found the tool very cumbersome to use Somewhat disagree	Neither agree nor disagreeI found the tool very cumbersome to use Neither agree nor disagree	Somewhat agreeI found the tool very cumbersome to use Somewhat agree	Strongly agreeI found the tool very cumbersome to use Strongly agree
I felt very confident using the tool	Strongly disagreeI felt very confident using the tool Strongly disagree	Somewhat disagreeI felt very confident using the tool Somewhat disagree	Neither agree nor disagreeI felt very confident using the tool Neither agree nor disagree	Somewhat agreeI felt very confident using the tool Somewhat agree	Strongly agreeI felt very confident using the tool Strongly agree
I needed to learn a lot of things before I could use this tool	Strongly disagreeI needed to learn a lot of things before I could use this tool Strongly disagree	Somewhat disagreeI needed to learn a lot of things before I could use this tool Somewhat disagree	Neither agree nor disagreeI needed to learn a lot of things before I could use this tool Neither agree nor disagree	Somewhat agreeI needed to learn a lot of things before I could use this tool Somewhat agree	Strongly agreeI needed to learn a lot of things before I could use this tool Strongly agree

Please indicate your level of agreement

Imrovement Feedback

List one or more aspects that positively impressed you in using the VAST annotation tool: *

List one or more aspects that negatively affected your annotation experience: *

Do you have suggestions how to improve the usability of the tool? *

Do you have suggestions about possible functionalities that can be added to the tool? *

Were the annotation objectives that you received adequate for you to perform the activity? Was it easy or difficult to find the required number of values? *

Were the annotation guidelines that you received adequate for you to perform the activity? How could they be improved? *

Is there any question that you had and remained unanswered or difficult to answer about the annotation activity? *

How do you evaluate the list of value labels used for annotation? Do you have suggestions about possible missing labels to include? *

Cookie	Duration	Description
cookielawinfo-checkbox-analytics	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics".
cookielawinfo-checkbox-functional	11 months	The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional".
cookielawinfo-checkbox-necessary	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary".
cookielawinfo-checkbox-others	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other.
cookielawinfo-checkbox-performance	11 months	This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance".
viewed_cookie_policy	11 months	The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data.