a) Consent
Receive a link to a dedicated, secure Learning Management System (LMS) site created for this research. The site provides a clear description of the study (objectives, requirements, and procedures). My consent will be indicated by ticking a consent checkbox.
b) Be Given Anonymous Access
Be provided with randomized login credentials (username and password). On login, the system assigns a random code as my username (e.g., LLM2026_XXXXXXXX). No names, student IDs, or other personally identifiable details are used.
c) Complete Computer Programming Tasks (20-30 minutes)
Complete three tasks in one of the following languages—Python, C, or C++—at ascending difficulty levels:
a) Beginner (~10 minutes)
b) Intermediate (~10 minutes)
c) Advanced (~10 minutes)
d) Use of the LLM Chatbot (throughout tasks)
A built-in LLM chatbot is available within the LMS for guidance and explanations at any time during the tasks.
e) Answer Exit Questionnaire (20–30 minutes)
(i) Demographic Questions (15 Questions): age range, gender, study level, programming experience, and prior use of AI/LLM tools. No identifying personal details are collected.
(ii) Likert-Scale Items (50 Questions): e.g. assessing perceived usefulness, learning support, and views on LLMs in computer programming education.
1 – Strongly Disagree
2 – Disagree
3 – Neutral
4 – Agree
5 – Strongly Agree
(iii) Open-Ended Questions (5 Questions): optional free-text reflections (sentences, short paragraphs, or bullet points).