A threshold for a q-sorting methodology for computer-adaptive surveys

© 2017 Proceedings of the 25th European Conference on Information Systems, ECIS 2017. All rights reserved. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Due to the complexity of CAS, little work has been done on developing methods for validating their content and construct validity. We have created a new q-sorting technique where the hierarchies that independent raters develop are transformed into a quantitative form, and that quantitative form is tested to determine the inter-rater reliability of the individual branches in the hierarchy. The hierarchies are then successively transformed to test if they branch in the same way. The objective of this paper is to identify suitable measures and a “good enough” threshold for demonstrating the similarity of two CAS trees. To find suitable measures, we perform a set of bootstrap simulations to measure how various statistics change as a hypothetical CAS deviates from a “true” version. We find that the 3 measures of association, Goodman and Kruskal's Lambda, Cohen's Kappa, and Goodman and Kruskal's Gamma together provide information useful for assessing construct validity in CAS. In future work we are interested in both finding a “good enough” threshold(s) for assessing the overall similarity between tree hierarchies and diagnosing causes of disagreements between the tree hierarchies

A test of a computer-adaptive survey using online reviews

© 26th European Conference on Information Systems: Beyond Digitization - Facets of Socio-Technical Change, ECIS 2018. All Rights Reserved. Traditional surveys are excellent instruments for establishing the correlational relationship between two constructs. However, they are unable to identify reasons why such correlations exist. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Assessing the validity of CAS is an underexplored research area as CAS differs from traditional surveys. Therefore, validating a CAS requires different techniques. This study attempts to validate the conclusion validity of a CAS about café customer satisfaction using online customer reviews. For our CAS to have conclusion validity, there should be a high correspondence where most respondents in CAS and online reviewers both agree that certain constructs are the cause of their dissatisfaction. We created a Computer-Adaptive Survey (CAS) of café satisfaction and used online customer reviews to assess its conclusion validity. Our research thus contributes to the measurement literature in two ways, one, we demonstrate that CAS captures the same criticisms of cafes as that in online reviews, and two, CAS captures problems about customer satisfaction at a deeper level than that found in online reviews

A Q-sorting methodology for Computer-Adaptive Surveys - Style "Research"

Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Due to the complexity of CAS, little work has been done on developing methods for validating their construct validity. This paper describes the process of using a variant of Q-sorting to validate a CAS item bank. The method and preliminary results are presented. In addition, lessons learned from this study are discussed

Beyond conventional teaching towards networked learning: The role of generative AI chatbots in enhancing program evaluation skills

With Generative Artificial Intelligence (GenAI) adoption growing, education has seen the emergence of innovative technologies like chatbots. However, little research has examined the impacts of GenAI integration in specialized higher education contexts. This study explored graduate students’ experiences using a GenAI chatbot, PEARL, within a graduate-level teacher education course focused on teaching students how to collaboratively conduct program evaluations using practice cases. Four students participated in our study and shared perceptions of interviewing personas with PEARL when evaluating the practice cases. Thematic analysis identified advantages like enhanced efficiency and accessibility, plus limitations regarding authenticity of artificial interactions. Findings emphasized the continued importance of human guidance and peer learning to enrich GenAI-enabled education aligning with principles of networked learning. Students highlighted the need for ethical considerations despite interacting with artificial entities, underscoring nuanced understanding. The significance of collaborative analysis and ongoing iterative improvements also emerged as themes integral to meaningful learning. Although GenAI presents transformational potential in instructional designs, findings support the use of blended approaches that strategically integrate its advantages with human activity and collaborative inquiry. The study makes contributions by elucidating domain-specific nuances of integrating GenAI into teaching in higher education. Practical implications encourage scaffolding GenAI curricula to promote authenticity and collaborative knowledge construction. Further research could examine variations across disciplines, technologies, and demographics. Overall, as GenAI shapes academia’s evolution, reflective pedagogical examination will be key to evidence-guided integration. This exploratory study presents a preliminary yet important step, unveiling opportunities for networked learning and complexities of GenAI adoption in teaching program evaluation skills in education contexts

The Multimodal Effects of Voice-Based Asynchronous Technology-Mediated Communication on EAP Speaking Performance

This study investigates how using a voice-based Asynchronous Technology-Mediated Communication framework (ATMC) can affect English for Academic Purposes (EAP) students’ fluency, accuracy, and intelligibility. Research has highlighted the benefits of ATMC in improving various elements of spoken communication, but to date, its application to EAP contexts has been little studied. In EAP speaking/listening courses at a large Canadian university, a team of instructor-researchers developed a system to provide learners more speaking and listening practice opportunities and to give individualized video feedback (veedback) on speaking performance. The instructor-researchers used a practitioner enquiry research methodology to investigate the effectiveness of the voice-based ATMC for 14 participants, drawing upon qualitative data from student interviews and the instructor’s qualitative and quantitative assessment of the students’ responses to tasks. Findings indicate that the system resulted in overall gains in fluency, accuracy, and intelligibility due to a combination of the voice-based ATMC design, repetition and practice, and the veedback. Qualitative comments from participants indicate personalized feedback led to increased motivation. The framework described in this article, therefore, represents several important benefits for the application of voice-based ATMCs in EAP classrooms. La présente étude examine les façons dont le recours à un cadre de communication asynchrone au moyen de la technologie (ATMC) axé sur la voix peut affecter la maîtrise de la langue, la précision et l’intelligibilité chez les apprenantes et apprenants d’anglais académique. Les recherches ont fait ressortir les avantages de la communication asynchrone au moyen de la technologie pour l’amélioration de divers éléments de la communication orale, mais cette méthode d’enseignement a été peu étudiée jusqu’ici dans des contextes d’anglais académique. Les membres d’une équipe de professeurs-chercheurs actifs dans l’enseignement de l’utilisation et de l’écoute de l’anglais académique dans une grande université canadienne ont élaboré un système permettant de fournir aux apprenantes et apprenants davantage d’occasions d’apprendre à utiliser et à comprendre l’anglais académique en mettant à leur disposition un système de rétroactions personnalisées par vidéo (veedback) qui leur permet de constater la justesse de leur expression orale. L’équipe de professeurs-chercheurs a utilisé une méthodologie de recherche basée sur l’interrogation des praticiens pour examiner l’efficacité de la communication asynchrone au moyen de la technologie axée sur la voix chez 8 participantes et 6 participants à l’aide de données qualitatives obtenues lors d’entrevues avec ces derniers et d’évaluations qualitatives et quantitatives fournies par les enseignantes et enseignants relativement à la réponse des étudiantes et étudiants aux tâches qui leur étaient assignées. L’étude a permis de constater que le système produisait des avantages globaux dans les domaines de la maîtrise, de la précision et de l’intelligibilité de l’anglais académique grâce aux effets combinés de la conception du système de communication asynchrone au moyen de la technologie axé sur la voix, à la répétition, aux exercices et au veedback. Les commentaires qualitatifs fournis par les participantes et les participants indiquent que la rétroaction personnalisée faite par le système a eu un effet positif sur leur motivation. C’est donc dire que le cadre décrit dans le présent article représente plusieurs avantages importants relativement à l’utilisation d’un système de communication asynchrone au moyen de la technologie axé sur la voix dans les classes d’anglais académique

Teaching as Coevolving: An Approach to Online Course Design

The growth of online learning in higher education, over the last decade and its exponential development due to COVID-19, has opened up exciting possibilities for instructors by providing access to new modes of course design not possible within the constraints of a traditional classroom. One approach to enhancing the student online learning experience is Universal Design for Learning (UDL), in which students are able to engage with the material in a manner appropriate to their current situation. By using this approach, courses can be designed in ways that allow students to take personalized paths to achieve the course outcomes. The purpose of this presentation is to outline the lessons learned for designing online courses using UDL. The courses used courses provided multiple entry points for learning, so that students, with all their diversities, can adapt activities to fit their needs, emergent abilities, and interests. UDL can be expressed in four sub-principles: 1) providing multiple means of representation, with spaces for unanticipated possibilities to emerge; 2) providing multiple means for students to express what they know and what they have learned; 3) offering ways into, and explorations beyond, planned experiences; 4) permitting and nurturing specialized interests of individuals, while enhancing possibilities for the collective

Entropy Estimate for Maps on Forests

A 1993 result of J. Llibre, and M. Misiurewicz, (Theorem A [5]), states that if a continuous map f of a graph into itself has an s-horseshoe, then the topological entropy of f is greater than or equal to logs, that is h( f ) ? logs. Also a 1980 result of L.S. Block, J. Guckenheimer, M. Misiurewicz and L.S. Young (Lemma 1.5 [3]) states that if G is an A-graph of f then h(G) ? h( f ). In this paper we generalize Theorem A and Lemma 1.5 for continuous functions on forests. Let F be a forest and f : F?F be a continuous function. By using the adjacency matrix of a graph, we give a lower bound for the topological entropy of f

Computer-Adaptive Surveys (CAS) as a Means of Answering Questions of Why

Traditional surveys are excellent instruments for establishing the correlational relationship between two constructs. However, they are unable to identify reasons why such correlations exist. Computer- Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Their principal advantage is they allow the survey developer to input a large number of potential causes. Respondents then roll down through the causes to identify the one or few significant causes impacting a correlation. This study compared a café satisfaction CAS to a traditional survey of the same item bank to test whether CAS performs its intended task better than a traditional survey. Our study demonstrates that when one is trying to find root cause, CAS achieves a higher response rate, requires fewer items for respondents to answer, has better item discrimination, and has a higher agreement among respondents for each item

A TEST OF A COMPUTER-ADAPTIVE SURVEY USING ONLINE REVIEWS

Traditional surveys are excellent instruments for establishing the correlational relationship between two constructs. However, they are unable to identify reasons why such correlations exist. Computer-Adaptive Surveys (CAS) are multi-dimensional instruments where questions asked of respondents depend on the previous questions asked. Assessing the validity of CAS is an underexplored research area as CAS differs from traditional surveys. Therefore, validating a CAS requires different techniques. This study attempts to validate the conclusion validity of a CAS about café customer satisfaction using online customer reviews. For our CAS to have conclusion validity, there should be a high correspondence where most respondents in CAS and online reviewers both agree that certain constructs are the cause of their dissatisfaction. We created a Computer-Adaptive Survey (CAS) of café satisfaction and used online customer reviews to assess its conclusion validity. Our research thus contributes to the measurement literature in two ways, one, we demonstrate that CAS captures the same criticisms of cafes as that in online reviews, and two, CAS captures problems about customer satisfaction at a deeper level than that found in online reviews