ANNEX F (INFORMATIONAL) APPLICATION PROFILES This standard is aimed at any individual, group, institution, administration, or company involved in the processes of creation, use, and assessment of digital educational materials. The application of the standard depends on the user's role and the scenario for application. In this regard, four types of users are distinguished: Author/Creator, Consumer/User, Reviewer/Evaluator, and Supplier/Distributor (Chapter 1. Objective and scope of application). The quality assessment tool (Annex A) is aimed at all four types, as the quality model on which it is based is a general, global model. However, the scenarios for application of the standard are very diverse, and in some of them it may be advisable to adapt the tool in order to facilitate and optimise application. One of these recommended scenarios is that of a teacher (or teaching team) who wishes to create quality DEMs for a highly specialised knowledge domain. In this scenario, in order to use the quality tool as a guide for production, the author of the DEMs must have significant knowledge both of the DEM disciplines and of Educational Technology and Accessibility to use the tool. This means that a teacher who is a specialist in his discipline but not a specialist in Educational Technology or Accessibility will find it hard to apply certain parts of the tool and may not benefit full from it to create or select the DEMs needed. It is also the case that, when creating highly specialised DEMs (e.g. Health and Environment), the tool is too generic and its usability would improve if it is specialised by specifying some of its items with respect to the knowledge domain (e.g. item 1.3. can exactly specify the minimum academic syllabus, and items 3.2 and 3.3. can specify the topic for reflection and criticism of the domain). The adaptation of the quality tool to a specific application scenario in order to facilitate or improve application is called an application profile. The creation of an application profile is justified when there really is a need to facilitate or improve the standard in a specific scenario. Application profiles do not guarantee fulfilment of 100% of the quality criteria, so they cannot be used as a tool to certify DEM quality. However, they can be used to ensure/verify more easily certain quality aspects. If the application profile keeps the minimum compliance level items, compliance with it guarantees the basic quality conditions. This Annex provides two application profiles for non-specialist users: the teacher profile (table F.2) for those who are specialists in their disciplines but not necessarily specialists in Educational Technology and Accessibility, and the student profile (table F.1), for someone who is initially a non-specialist user. Both profiles have adapted the tool, simplifying or explaining the criteria or items that constitute it. The teacher profile maintains the minimum level items, while the student profile does not. They are based on profiles created and tested in two research projects for the creation of DEMs in the fields of: 1) Health Sciences. To evaluate the quality of digital educational materials in this field of knowledge, the research team led by Dr Carmen Álvarez Nieto at the University of Jaén has created one profile for teachers and one for students, so that both profiles can be used to evaluate the educational materials that are being created as part of the NurSusTOOLKIT European project (http://nursus.eu). 2) Educational apps. To assess the quality of this kind of specific application, the ATLAS research group at UNED, as part of the SO-CALL-ME national project, has created an application profile using the COdA initial model. In these projects, the student profile was used to obtain students’ views after using the DEM (post-use evaluation). The teacher profile was used by project participants (teachers-researchers) as a guide to create DEMs (pre-use evaluation) and to obtain teachers’ views about DEM quality (post-use evaluation). The student profile (table F.1) and the teacher profile (table F.2) are given below. Table F.1 – Adaptation of the tool to the STUDENT profile1 Identification of the Digital Educational Material (DEM) SOURCE REPOSITORY ID IN THE REPOSITORY URL OTHER IDS (IF ANY) DEM TITLE 1 Does not maintain minimum compliance items Criteria/items Score (from 0 to 10)* Remarks 1. Didactic description: didactic value and consistency 1.1. I clearly understand the goals to be achieved with this material 1.2. It is specified that the material is aimed at my educational level and/or age; it can achieve the didactic goals proposed 1.3. I am clear from the start about the competences and skills which I will develop by means of these materials 1.4. I know about the instructions and suggestions for possible ways to use the material (in the classroom with a teacher or in self- learning) 1.5 I know how much time approximately I will need to study/complete the DEM content 1.6. I know if I need to have previous knowledge before using the material and what this knowledge is TOTAL CRITERION 1 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 60 points 2. Content quality 2.1. The contents include all the learning objectives and are suitable for my knowledge level 2.2. The contents are clear and I understand what the key ideas that I must learn are 2.3. The contents are presented in an objective, respectful, and non-biased way TOTAL CRITERION 2 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 30 points 3. Ability to generate learning 3.1. This material helps me to connect the new knowledge with my previous knowledge 3.2. The material helps me to be critical and make questions 3.3. This material helps me to generate new knowledge 3.4. I can apply this material to practice TOTAL CRITERION 3 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 40 points 4. Adaptability 4.1. This material covers my learning expectations and needs and increases my interest in the topic 4.2. I feel comfortable with this material because it offers different content levels and formats that facilitate learning TOTAL CRITERION 4 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 20 points 5. Interactivity 5.1. The material is interactive (I learn actively) 5.2. The material contains interactive activities for key ideas 5.3. The material enables me to monitor and manage my learning I can choose what and how to learn 5.4. I can check my progress 5.5. The types of activities and exercises are varied TOTAL CRITERION 5 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 50 points 6. Motivation 6.1. I feel that what I learn from this material is important for my training 6.2. I feel that this material encourages self-learning 6.3. The estimated learning time is adequate to achieve by expected goals 6.4. The contents are appealing and innovative 6.5. The material encourages communication and collaboration among students TOTAL CRITERION 6 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 50 points 7. Format and design 7.1. The material has an easy, clear, and well-organised design 7.2. Texts, images, audios, and videos are of good quality 7.3. The audiovisual contents facilitate and reinforce learning 7.4. Different multimedia formats (text, image, audio, and/or video) are included 7.5. The material is easy to use 7.6. The material is appealing and supports study TOTAL CRITERION 7 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 60 points 8. Portability 8.1. I can view and use the materials in all the devices I use (computer, tablet, mobile, etc.) 8.2. The material can be used on any device with or without an Internet connection TOTAL CRITERION 8 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 20 points 9. Robustness; technical stability 9.1. The material does not fail during operation 9.2. The material has a fast response when I interact with it 9.3. I can find assistance to use it if I have a problem TOTAL CRITERION 9 Maximum 30 points 10. Navigation 10.1. The names of the links say where they go 10.2. Links work properly 10.3. I am clear at all times about what part of the material I am in, how much I have done, and how much is still left 10.4. I can exit the material at any point TOTAL CRITERION 10 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 40 points 11. Operability 11.1. The material works with a keyboard and mouse. It also works with a tactile screen or other support devices 11.2. I have no problems seeing and moving the cursor 11.3. I have enough time to read and do the activities TOTAL CRITERION 11 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 30 points 12. Accessibility of the audiovisual content 12.1. I can see or read image descriptions 12.2. I can see, listen, and read video contents. I can listen and read video contents. 12.3. I can play the videos and audios TOTAL CRITERION 12 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 30 points 13. Accessibility of the text content 13.1. I can read well the text and can adjust its size 13.2. If there are any forms, they are easy to complete 13.3. If there are any tables, they are easy to read and understand 13.4. If there are any lists, they are easy to read and understand TOTAL CRITERION 13 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 40 points 14. Competences After completing the session with the learning materials, now I am able to… 14.1. Competence X (e.g. ... explain how X is used) Add the list of competences listed in the didactic description of the material. In this way, the actual didactic effectiveness of the material after use can be evaluated Open questions: Is there anything else you would like to tell us about regarding your experience using these materials? Is there any way in which these materials can be improved? If so, please specify the number of the item you would improve, how, and why. TOTAL CRITERION 14 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 510 points Table F.2 – Adaptation of the tool to the TEACHER profile2 Identification of the Digital Educational Material (DEM) SOURCE REPOSITORY ID IN THE REPOSITORY URL OTHER IDS (IF ANY) DEM TITLE 2 Maintains minimum compliance items Criteria/items Score For each item a value between 0 (it does not fulfil) and 1 (it fulfils) with just a decimal figure or NA (not applicable). Criterion´s score is the addition of the items Remarks 1. Didactic description: didactic value and consistency 1.1. The didactic objectives are clearly and precisely specified in the DEM (what I am going to teach) 1.2. The recipients are specified; the didactic objectives can be achieved by the recipients according to the profile required in the DEM 1.3. The competences and/or skills to develop are clearly specified; they are consistent with the objectives and the recipients 1.4. There are instructions and suggestions about the possible didactic use for teachers and/or students (self-training) 1.5 The estimated learning time is specified 1.6. The student's previous knowledge is specified TOTAL CRITERION 1 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 6 points 2. Content quality 2.1. The content is consistent with the didactic objective (each of the objectives is worked on). 2.2. The contents are presented in a clear and understandable way. The key ideas are highlighted and clear instructions are given in the activities 2.3. Ideas and concepts are presented in an adequate number, in an orderly and balanced way throughout the DEM 2.4. The content is scientifically correct, non-biased, objective, and contains true information 2.5. The content is up to date or else is atemporal (there is no need to update it) 2.6. The content respects intellectual property rights if other materials are used (texts, images, etc.) Annex G to the Standard can be viewed for help 2.7. The content declares the conditions for use of the educational material TOTAL CRITERION 2 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 7 points 3. Ability to generate learning 3.1. The DEM promotes significative learning (connecting new concepts to those they already have) 3.2. Reflection is encouraged 3.3. Critical thinking is encouraged 3.4. Creativity and innovation are encouraged, the generation of new ideas and ways to apply them by students TOTAL CRITERION 3 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 4 points 4. Adaptability 4.1. The content is adapted to students’ previous knowledge and their learning needs 4.2. The DEM content/activity can be easily modified to adapt them to different student groups/types (e.g. different levels of knowledge or learning pace). 4.3. Different contents/activities or different content/activity itineraries are proposed depending on the level of knowledge and/or learning possibilities and capacity. 4.4. The DEM respects different learning styles. Annex H to the Standard can be viewed 4.5. Contents can be used independently from the teaching and learning method TOTAL CRITERION 4 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 5 points 5. Interactivity 5.1. The DEM encourages student participation during its reading or viewing or during interaction with it 5.2. The DEM contains interactive activities for key ideas 5.3. Monitoring and management of their learning by students is facilitated 5.4. A student’s activity performance history can be obtained 5.5. The types of interactive activities are varied TOTAL CRITERION 5 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 5 points 6. Motivation 6.1. There is a connection between what is learnt and the DEM recipient’s life environment (professional and/or social) 6.2. Students’ self-learning is encouraged 6.3. The estimated learning time is adequate to achieve the didactic objectives and is in accordance with students’ expectations and possibilities 6.4. Contents are presented in an appealing or innovative way 6.5. Communication and collaboration are encouraged TOTAL CRITERION 6 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 5 points 7. Format and design 7.1. The DEM design is well organised and is clear, concise, and intuitive 7.2. Images, audios, and videos are of good quality 7.3. The audiovisual contents facilitate and reinforce learning They are not decorations that hinder or slow down 7.4. The DEM includes multimodal formats: text, image, audio, and/or video 7.5. Use of the interface is intuitive (e.g. contents and instructions are easily located), and, if not, there are very clear instructions for use 7.6. Aesthetics is compatible and adequate for study of the DEM. It does not display visual noise or an unnecessary information overload 7.7. Consistency is maintained in the appearance (shape, size, colour, location, etc.) of those elements that have the same function (links, icons, buttons, etc.) across the DEM 7.8. There is a "preferences” option that makes it possible to personalise the interface (font, colour, font size, background colour, menu, etc.) and these are maintained in following sessions. TOTAL CRITERION 7 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 8 points 8. Reusability 8.1. The DEM is organised in a modular way so that it is scalable 8.2. The DEM or any of its modules can be used to create new DEMs 8.3. The DEM or any of its modules can be used in more than one discipline or group of students (this may not be desirable if the material is highly specialised. In this case, score as Not Applicable). TOTAL CRITERION 8 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 3 points 9. Portability 9.1. The DEM has been created using widespread or de facto standard formats (e.g. txt, word, pdf, html, xml, wav, mp3, mp4, jpeg, gif, etc.) If the DEM has not been created in an official standard format or in a widely used format, the computer requirements are described and the software required to use it is provided 9.2. Students can use the material on any device with or without an Internet connection 9.3. The DEM is associated with a metadata record that describes it 9.4. The DEM metadata are created in accordance with international standards (e.g. LOM-ES, Dublin Core) 9.5. The DEM is exported using international standards for the exchange of educational contents (e.g. SCORM, IMS Content Package) TOTAL CRITERION 9 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 5 points 10. Robustness; technical stability 10.1. The DEM does not fail during operation 10.2. The DEM is not affected by user errors 10.3. The DEM responds quickly, in a visible and audible way, to user actions 10.4. Assistance is provided about common user problems and their solutions TOTAL CRITERION 10 Maximum 4 points 11. Structure of the learning scenario(*) (*) A learning scenario is the DEM space where the user works with the content. E.g., software application screens, presentation “slides", and webpages 11.1. Every learning scenario has a unique and significant title, and can be accessed visually, directly, or by means of the support devices 11.2. The semantic structure and information relations within a learning scenario are made explicit in the presentation and can be accessed directly or through an access compatible with the support devices (e.g. screen reader, alternative keyboard, dynamic Braille display, or screen zoom) 11.3. Learning scenarios enable “always forward” use, maintaining previous scenarios simultaneously if necessary, and “going back to previous scenarios” if they do not have to be simultaneously maintained 11.4. If overlapping learning scenarios are allowed, they can be minimised, maximised, restored, and closed, and their size can be changed TOTAL CRITERION 11 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 4 points 12. Navigation 12.1. The name of each link is descriptive, clear, and different from the other links. Links leading to the same location use the same descriptive text. 12.2. Links work properly 12.3. At least two mechanisms are given to locate each learning scenario in the interface (e.g. in the case of a web scenario, learning scenarios are web pages and the mechanisms for page location can be links to other related pages, a list of contents (table, sidebar, etc.), a map of the digital educational materials, a search engine, a list of links to all the educational material pages on the home page, etc.) 12.4. The logical navigation order and the location of the navigation mechanisms are maintained, unless the user changes them 12.5. Users are given information about where they are in the DEM (e.g. they are told about the path of learning scenarios visited and the current scenario by means of “breadcrumbs”, progress bar, etc., or else by means of a menu/contents or map of the materials, specifying the current location) 12.6. Students are aware of their progress through the contents 12.7. The interface provides unlimited or sufficient time to read and use the content. In any case, the time for content reading and use can be adjusted. 12.8. It is not mandatory to go through repeated contents. (e.g., in the case of a website, links should be provided at the start of each webpage to “jump” directly to the main contents, avoiding going through other elements that are repeated in all webpages, such as email access, navigation menus, links to forums, etc.) 12.9. The contents return to the initial settings at the start of each session. 12.10. The DEM informs users about its status (active/non-active) 12.11 It is possible to exit the material at any point TOTAL CRITERION 12 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 11 points 13. Operability 13.1. It is possible to operate the DEM through a compatible or direct access. 13.2. Full operability through keyboard, mouse, or any other input device offered, including emulators, voice activation, tactile interaction, etc. 13.3. The focus of the keyboard (or another alternative device such as the mouse) can be visually distinguished when operating the keyboard (or an alternative device). (help: this item can be verified by navigating through the DEM with the keyboard or alternative device and checking that the location of the cursor can be clearly seen. If browsing using tabs, the position (or element) where the tab is located is marked in some way, as well as how it changes to the following position as the tab is clicked). 13.4. There is no time limitation to perform the tasks, or else there is a limited period of time which students must be able to adapt to their speed and needs. 13.5. Keyboard shortcuts or hot keys are provided for the main links and main form controls (help: hot keys are defined by the DEM author and are specified in an accessible way in the same resource) 13.6. All the learning scenarios in the DEM appear and operate in a predictable way. Students are previously warned if there are any changes in context. (e.g. an icon in a DEM screen receiving the focus is highlighted and a text box giving an explanation is displayed. This way of operating must be maintained for all icons, or users must be otherwise warned) TOTAL CRITERION 13 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 6 points 14. Accessibility of the audiovisual content 14.1 There is sufficient contrast between the colour of images and the background colour so that they can be seen well. 14.2 All audiovisual contents (images, graphics, figures, etc.) must have an alternative text description which can be accessed directly or through support devices 14.3 Audiovisual contents (video, audio, animations, etc.) have synchronised alternatives, such as subtitles (for people with hearing disabilities or aural comprehension difficulties), audio description, full transcription, or Sign Language 14.4 Students can play audiovisual contents and manage their text alternatives (e.g. activate/de-activate subtitles, etc.) 14.5 If the DEM includes unexpected sounds, students should be able to control them (turn them off, lower the volume, etc.) 14.6. The content does not include flashing effects with a threshold that can cause seizures or convulsions. 14.7. If there are any visual or sound alerts, they must have alternatives: sound alternatives for visual alerts, visual alternative for sound alerts. TOTAL CRITERION 14 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 7 points 15. Accessibility of the text content 15.1. The text is legible and/or its size can be adjusted 15.2. There is a contrast between the text and the background colour to read the text clearly and effortlessly 15.3. No information is given exclusively by sensory means (e.g. if colours are used to convey information, text marks must also be used so that people with visual disabilities do not miss this information) 15.4. If there are any forms, they have a clear structure that is consistent with the information presented and requested. They are easy to complete and the necessary assistance is offered to complete them as well as to prevent and/or correct any mistakes when completing them. They should also be coded in such a way as to enable compatible access through support devices 15.5. If there are any tables, they must be correctly used, be well structured and described, and be coded in such a way as to enable compatible access through support devices 15.6. If there are any tables, they are simple, avoiding combined, divided, and nested cells as far as possible. They are only used to display/arrange data, not for layout purposes. They must be correctly coded so as to enable compatible access through support devices (e.g. HTML tables on web pages should use the correct HTML elements to specify the summary, title, header, rows, and data) 15.7. If there are any lists, they are only used to display/arrange sequences of elements, not for layout purposes. They should also be coded in such a way as to enable compatible access through support devices (help: if lists are created using the list function provided by content editors in each of the potential formats: text, webpages, pdf, etc. support devices have no problems) TOTAL CRITERION 15 TOTAL NUMBER OF NON-APPLICABLE ITEMS Maximum 7 points Open questions: Is there anything else you would like to tell us about regarding your experience using these materials? Is there any way in which these materials can be improved? If so, please specify the number of the item you would improve, how, and why. ANNEX F (informational) Identification of the Digital Educational Material (DEM) Identification of the Digital Educational Material (DEM)