Getting Ready for Online Exams

so gang ray for exams at brock university i want to start by sharing some of the things we can consider across all of our options and that are remote conditions so we are all uh spread across the globe as it were right now we have very little assurances of the actual conditions that students are writing in but the interesting thing with that is the conditions our students find themselves in is actually uh in by some measures more like the professional experience for anyone these days too we are all in our bedrooms and remote locations most of us are um so it’s interesting that that same conditions are different for us are actually from a perspective more like the professional conditions our students might ultimately find themselves in or currently find themselves in we can ask students to engage the whole internet as they respond or seek assurances from that they will not that’s kind of our options as far as how students engage with our examinations we can acknowledge that they have an open internet open book or we can on their honor ask them to restrict themselves to access of course we have no direct assurances of the latter but those are the conditions which we find ourselves in so important things to keep in mind that the instructor controls in all this mention the things we don’t control the main things we do control we the instructor uh or the department set the learning outcomes identified the grade weighting and was in control of almost everything that was added to this course or part of this course up until the moment of examination the instructor also controls the assessment content the format the duration and grading policies so even though we no longer have a shared gymnasium for these examinations and there are lots that uh we do share and there’s lots that are remain within the instructor’s control even these new conditions so as i go through the options here on our toolbox assessment tools at brock i want to highlight these three things to remember because they apply to all of our options it’s really important to communicate expectations to students so one of them i like to highlight is there are other issues for the students on their end um important to communicate in advance if students should number one be alerting you to them and number two how especially when we think about scaled uh assessments if there is an issue um how how would students reach out to you um what kind of uh information would you want to collect important to be clear about that before an instant incident and if uh in the unfortunate situation where there is an incident students know what their options are and that is likely the same way they contact you for anything else in the course but again clear communications is important here also important to be clear about how you respond to unexpected events um it’d be important to let students know what kinds of affordances there might be or might not be for student related technical impediments on their end the situation where that might arise is hard to identify but we can make plans for how we might address those or not in their event and i always like to think about suggesting that students be able to visualize the examination process now not all students need to visualize the examination process but i’m asking instructors if you give students the information that they could use to visualize the examination process you’ll be at your desk it’ll be in sakai or it won’t be it’ll be multiple choice or it won’t be if you’ve communicated that information to students then they have a good expectation set for how they’re going to undertake this assessment they’ll need to actually sit down and visualize it but if you’ve communicated everything all the information that you have to a student to visualize what their new examination context is then they will be able to anticipate potential issues ideally and address them and hopefully begin this assessment process under less stress or concern than the otherwise might the other thing i want to highlight of the three things is the importance of practice that goes right along with the idea that students should visualize things it’s important to set up low stakes or practice conditions for students to be assessed in a similar format to how you ultimately want to examine them so that they can be ready for that process they can anticipate any issues and if you are going to expect an outcome from students students can be assured that they can meet that outcome or take the steps they need to take in order to meet that outcome so if you are assessing students in a way that requires a camera scanning or other

kinds of media tasks it’s really important to make sure students can achieve those tasks before in a practice condition and then when the examination starts any potential barriers have been removed and students are familiar with their options a lot of what we ask students to do in everyday life they are able to do them simply because they’re familiar with them there are no issues it’s when we encounter new experiences that we can often encounter issues especially with technology so practice and low stakes opportunities are important to address those early and then the last of the three things i want you to consider across all of our options is constructive alignment our what was taught in the course uh and identified learning outcomes and will a student that has demonstrated this knowledge understands the con the course be successful and rely on that knowledge when they’re assessed or are they going to be relying mostly on their test taking abilities and certainly that’s not our intent but sometimes the tools manifest this way if a student knows that the answer is always c or the longest option is often the correct option that’s a skill but we would prefer that students are being assessed on their knowledge of the topic and achieving success in their responses based on that knowledge so as you construct your assessment it’s important to make sure that that’s what’s being tested not a student’s neutral test smarts so those are the three things communications practice and constructive alignment that i want you to consider across all of our options so what are our options i shared in the chat a link to the decision tree that i’m showing you right now and uh the link to the web page that should have it on it will make that sure it definitely is in the end but here are the assessment options at brock university based on our current toolbox and their most likely use so starting up the top left as an instructor i recommend asking yourself should the exam script be unique to each student so if the exam script needs to be unique to each student then that probably means the sakai test and quizzes tool is the best option for a course with unique uh scripts for each student it gives you the option to randomize the position of distractors in questions so if you only have 50 questions and they’re all multiple choice you can make sure the positions of a b c and d change for your students additionally you can construct more than 50 questions and ensure that the sequence of those 50 questions are randomized for each student and that further if you construct a 75 questions then each student gets a subset of 50 making sure that students have a unique script and should students be tempted collaborate in a way that they aren’t intended to in this examination then the effort they have to put into collaboration is much higher and hopefully it returns if the uniqueness of the exam script is not a priority um then the next question i would ask is should grading be automated if grading needs to be automated similar to a scantron or simply for scale then next question would be if i have to automate my assessment can i assess students through multiple choice true and false short answer other questions like matching and similar uh tools that we have in our sky tests and quizzes or other options if that’s the case then we’re back up at using sakai’s test and quizzes tool and i’ll definitely discuss a lot of options with tests and quizzes if multiple choice and true false cannot assess what the learning outcomes this course are or the modalities in which students are asked to respond then it’s worth considering other tools if it’s a long written response is what we’re seeking from students then the assignments tool a pdf or word document submitted to the assignments tool might be the best way to collect that mark that similarly if students are generating files from a piece of software if you’re working in spss sas or other computer programming software where an outcome of files is what’s being turned in gis course for example then again the assignments tool might make the most sense if the response is more visual if the student needs to label a diagram or mike we lost i can’t hear you

i i felt myself press the mute button thank you norbeck if the student needs to respond through something uh more visual mark up a graph or use scientific or math notation the new tool brock called crowdmark might be the best option i’ll cover that as well that allows us to give students something like a diagram to label and then turn that in as an image and we can assess it there so that’s ending up in all three of our options here there’s a couple other scenarios where if i need grading to be automated if the time limit is important then often tests and quizzes tool is the best tool to be administering varied or strict time limits but this is generally a good way to begin to start with what questions what kind of exam script do i need to generate and then what tools i would perhaps use that said i think there’s nothing wrong with having a tool you’re familiar with here and working backwards from that and understanding what’s capable of so we’ve shared that uh decision tree in the chat it’ll be in recording and in the link afterwards and of course if you can’t reach out to the cpi this is as you describe your course the kinds of things that members of cpi will be considering as we respond so i want to talk about crowdmark as it might be the simplest one to explain and the one that you may not be familiar with at the moment thanks to a special allocation from the provost for this year when we don’t have access to our physical dropboxes we have access to a digital dropbox and most importantly a digital grading path in the form of crowdmark so the way crowdmark works is the instructor prepares homework or tests they construct questions in the crowdmark interface either by attaching a pdf or making the questions one by one inside the crowdmark interface they can sync up the crowdmark site with the sakai site to get student enrollment students get a message telling them they need to respond to an assessment they go in the crowd mark they see the questions and they’ll get prompted to attach a file or take a picture and then ultimately submit the assessment and on the instructor side they can grade that assessment all digitally and you can even see in this particular picture that a scan or a smartphone picture has been used to take uh this mathematical notation digitize it and the instructor is giving feedback based on what was submitted so from a student experience i’ll just uh begin this playing as i talk more about once i dismiss my markup tool press escape so my student experience what they would do in uh in the process experience or the ultimate examination go a crowd mark see their assessment in this case we have someone uh me who’s printed the assessment in advance marked it up and is ready to submit on in this case in their smartphone take a picture attach it and press submit students can do this with their smartphone students can do this all digitally if they happen to have a tablet or other digital workflow that works for them primark expects an outcome and and thus allows students to find the path that works best for them which is also why practice is very important on the instructor experience side i’ll advance through this instructor creates their assessments in the crowdmaker interface and deliver that to our students actually i’m gonna move on one more slide and then when the student submissions are actually uh submitted the instructor and their marking team can mark up those submissions give the grade return that mark back to the student anything the student can take a picture of scan turn into a pdf they can submit and thus if we need them to do scientific mathematical notation mark up a diagram or a graph or create a graph we can do that i know from personal experience and experience of other instructors we’ve worked with it’s frustrating when the assessment you want to have constructed involves more instructions but using things like powerpoint or canva or anything else than the actual subject matter assessment instructions something like crowdmark lets you

uh just tell students take a picture and submit it and on the marketing side there’s no incompatibilities etc with the images or loose files you get this nice interface of feedback so that’s the first of three tools we talked about the kinds of decisions that might lead you to one of these three tools the next tool i want to talk about is sakai’s test and quizzes tool i wanted to pause that prompted any questions that people might want to share in the chat or aloud about the kinds of decisions you might make about what assessment to use something that’s seeing that paper-based crowd mark experience might have prompted for you or other questions you might have thus far so you want to add a question in the chat raise the hand unmute see mark shared our notes for today which is great and the decision tree testing quizzes is a very powerful very and also very complicated so i want to give everyone a chance to ask questions they might have about um how one ends up using these platforms for exams or even just crowd mark before we get into the complexity that is the test and quizzes tool generally that said any one person’s use of tests and quizzes can be quite simple the challenge is finding your use for tests and quizzes and your familiar space within that complexity that’s i think the challenge of tests and quizzes overall okay tests and quizzes allows us to deliver assessments to students that um can be drawn from a question pool can have a time limit we can make sure that we have unique assessments for each student or common one fully integrated in sakai and the gradebook it can be linked to from other tools brock university regular deliver delivers large assessments through the tests and quizzes tool and is not on by the fault in the sakai course but siteinfo managed tools allows you to add the tests and quizzes tool so the main testing quizzes tool interface and if you have a a sample course dummy course or other course you want to look at you could welcome the follow along you won’t be moving slow enough for you to construct with me but you’re welcome to look at these various parts with us in the test and quizzes tool we’ve got a couple tabs across the top updated for sakai 19. the add button which we’ll look at in a moment as we construct a assessment through its settings the actual assessments page where we have concepts of draft and publish the assessment types to set question pools an instructor can have a set of question pools which i mentioned can be drawn from in any one assessment and also be used across their courses in sakai you can deliver a good orientation uh quiz turn that into a pool and then bring that’s your other course you can refine a pool of questions from year to year and you can share your pools per site with other instructors in that site we have an event log which is important for understanding what happens with our assessment uh ongoing and use user activity report plus if you uh remove a assessment and you need to recover it there’s a trash maybe recycling can be more important but i think the use is clear so i want to give you some tips about what can help understanding how tests and quizzes tool works because within all those options again complexity but there could be a simple familiar path for anyone using it i always get the advice to consider duration before dates and times so what i mean by that is there’s an option to set a time limit typically two hours or three hours in brock university uh some of the and you’ll get a time assigned from the registrar for exams i recommend setting time outside of the fixed duration for a number of reasons so if i have a two hour exam then i’ll set my time limit for two hours i may even set my availability for if it was scheduled for seven pm i may set for 6 55 pm and if it’s a two hour duration i may set my

end availability time as late as 9 30 or maybe even 10 because the important thing for us in this assessment is is a two hour duration we want to make sure that two hours is all the time students get unless they have an exemption but the way the web works when they click that link is when they actually start so you want to make sure that the students have a moment to navigate to that page to refresh the page to see the link and other potential issues they may have along the way we don’t want students to be a denied amount of time simply because they misnavigate or took a little extra time with their password especially when we consider multi-factor authentication becoming a new thing at brock university the other side at the end of that is if a student were to start at in our scenario uh 705 we’re only going to give them two hours but if we close had the closing date at 9 00 pm sakai is going to only accept what was saved up until 9 00 pm and unless we get an extra time so that student would in effect be given an hour and 55 minutes not the two hours we intended and i would suggest that the time that may have taken them to navigate towards that start point really isn’t part of their assessment we wanted them to be assessed for two hours so by setting that duration to give some grace time at the end for students that may not have clicked and likely haven’t clicked that the exact second the assessment became available is both important for making sure students get a fair chance and i will save from experience um minimizes the number of issues you could have with students ongoing in the exam so think duration and then think about time limits that will nicely and safely fit that duration students can start an examination at any point during that availability so obviously you wouldn’t we all think there should be consequences for a student that starts two hours late um but at the same time it’s much easier to uh allow for that five minute late start that only got two hours to submit versus um having to deal with consequences of a submission that was technically blocked when otherwise we’d like to accept it so think duration over start dates think duration first then pick your dates um the other thing to consider is once an assessment is published in sakai there is very little to be changed so if the scoring of a question was uh incorrect there was a typo there was a slip of the mouse when the questions were authored there isn’t much we can do once that quiz is published there are some things we can do retroactively but be prepared that once a quiz is published is in that state so if midway through the examination students identify that it says respond to all the above but you chose the random position so all of the above is the top option unfortunately you’ll have to live with that and address it after the assessment there isn’t anything we can do to modify those kinds of things assessment once published so on taco building your assessment and tests and quizzes but again if this prompts any questions or comments you’re welcome to raise your hand play something in the chat or grab the mic as you’re building a test and quizzes tool there’s two primary routes to construct your questions so on the left here is the familiar assessment builder which is ultimately the tool that is used for creating things in tests and quizzes allows you to build calculated questions have file uploads fill in the blank hotspot selection matching options multiple choice numeric responses short answer or essays an audio recording survey like style a survey matrix style true false and then options from the pool that’s where you’ll get the most question type options on the right here is an alternative affordance i really like in sakai you can create using markup text which only supports multiple choice fill in the blank short essay true false and numeric fill in the blank but you can construct the questions in a much more natural language method in microsoft word for example or copy an old exam script follow these markup guidelines you can see in this little example here if you follow its guidelines it can

identify that text spot that those are indeed multiple choice or fill in the blank questions and then build an assessment in the regular test and quizzes tool with the regular uh interface that you also have to use for creating assessments but that can be a nice uh quick start for building assessments in sakai that markup text type it all out lay out your questions in the format that it will prompt you to here on the right press create and you’ve got a bunch of questions created in this format already so as i mentioned there are 12 question types supported by sakai’s test and quizzes tool um each question should have a point value remember to set it it is going to be blank by default and one of those challenges you can have in the delivered assessment if you deliver an assessment with a question is worth zero you’ll have to do something corrective after it’s delivered be sure you select the correct response if you’re expecting one and a good tip is there’s the option to randomize answers in multiple choice so that the correct answer and the distractors are all in different positions for each different student it’s easy to check that box the only real complication is you can’t use all the above none of the above you can have different languages such as all options are correct no options are correct because positioning moves otherwise randomizing that answer position ensures that students are looking for the correct answer not looking for the correct position speaking of ways to make things more dynamic for students we have some dynamic functionality in sakai’s questions the ability to do those question pools and draw from them is a clear example of this um if you’re capable of creating a large you have multiple pools you’re drawing from from different sections that’s helpful you can also use numeric responses where the response is a number between a range taking that even further you can have calculated questions which introduces a number of variables where not only is the response going to be dynamic but the construction of the question is as well uh right down to a formula so in this example kevin has x apples he buys y more now kevin has x plus y j zed apples kevin now has uh w apples we can because the way you set this up with those which the sakai interface will explain as you construct your calculated questions we’re asking our respondent to fill in how many apples kevin now has and the construction up to that point make sure that each student will get different values between for x 1 and 6 y 2 and 6 and z 1 6 again with no decimal places so asking students to respond to a calculated question with dynamic variables for each assessment inside sakai’s tests and quizzes they’ll have the following options to edit the questions preview it which is very helpful step that i recommend before you publish anything print so you can make a pdf the exam script likely for archival reasons today but i know that that has been used for a number of purposes including accommodations in the past the settings go through in a moment the publish option to take a draft and move it to the publish state where dates permitting students can actually see it and they’ll have an entry in the gradebook if you choose that duplicate for construction and export for moving between uh different contexts maybe inside sakai you can even export in formats that are compatible with other lms’s but the tools you’ll be using the most those actions are edit settings and then likely publish at some point i’m going to go through the settings these are the longest list of complex options in sakai’s tests and quizzes good to think about how these may particularly impact the assessment you might be considering um because there are again a lot of options but a familiar path should be there for everyone so i’m going through each item here uh the in what the ui people would call an accordion the first accordion area is uh about available in submissions expectations time limits and delivery date exceptions for me timeless and the rates grading feedback and the layout and appearance

so about just includes the title and an optional description this is also where you can require an honor pledge which will have students check off that they have neither given or received any unacknowledged assistance with the assessment doing everything a checkbox can do to promote integrity in the availability and submissions this is the area i was referring to earlier so it can be released to a subgroup or the entire site typically the entire site how many attempts to get in the context of exams it’s typically only one attempt but there are other options of course our first option there is its availability date when is it available right here and then when is it due but i always recommend consider the time limit first so this is intended to be a three hour exam set three hours and then i’d recommend availability and due dates that can accommodate perhaps students showing up early and wanting to get going is a minute or two at your discretion but most importantly uh do they if a student doesn’t make it there at the exact second that comes available they can still get their two or three hour window to submit so minutes if not tens or even a full hour’s worth of time after the time limit would expire so if a student starts minutes ten submits whatever after the actual availability date they can still get the full time limit we intended for them or less or so as appropriate this also applies to late submissions that’s our ultimate final date um so they can the students can press start up until the due date the last moment they submit is the due date but they can press start until the due dates and related by default the auto submit student work after the last acceptance date is checked you can uncheck that but i would strongly recommend you keep it this is going to ensure that if students never press submit for grading at the end of the assessment sakai will collect it at that last date similarly if students get interrupted or otherwise navigate away from the assessment the in-progress work will be submitted at the end it often adds nothing to our examination process to have students actually press the submit for grading option at the very end if they’ve responded to every single question it’s almost always better to assess what they’ve responded to versus reject everything so the auto submit there is just in case students don’t get to that very last spot of answering their final response and then pressing submit for grading that i’ll submit will make sure it’s collected and given to the instructors to create there’s also some email options there but they aren’t as important as other items so there’s a those dates put in sharper focus uh beneath that is the exemptions of time limits and delivery dates so in accordance with the instructions you may get from sas student accommodation services this is where you can add in students and their exceptions to those dates you already set this is often a duration exemption so if a student has 1.5 or 2 times the amount of time let’s say they have twice the amount of time on a 2-hour exam then you would select that individual student from the list here set a time limit in our two-hour exam to give them four hours and then further set dates that can accommodate four hours so it’s available around the same time as the rest of the students but we have to really consider that due date and late submission time late submission being the most important here of if they take uh if they start their four hours at five minutes pass let’s make sure in that late submission or do they we can accommodate that um because otherwise they’ll get truncated otherwise you’ll inherit the default date for other students and if if you don’t set a late submission that can accommodate the students extra time their extra time will not be afforded to them and they’ll be cut off at the otherwise due date so when you add exemption for a student think time limit first then think about the availability dates to complement that you can add them in sequence you can also construct group based exemptions though a lot of caution goes there there are ways to disclose those groups to the roster tool if you’re not careful and secondly you have to remember that

if a student is a member of multiple groups the last exemption in the list is the one that applies to them but otherwise you guess a combination knows for a student in your course you have an exam in sakai testing quizzes tool exemptions and time limits pick their name give them their likely duration extension and pick that dates to match add exemption so here’s what adding an actual exemption would work for one student we already have one set here we’ve picked another student setting their time limit first adding their dates at exception beneath that in our accordion here is grading and feedback if there’s multiple submissions which scores kept is important we can optionally hide the identity from the greater of student submissions we can optionally send the mark to the gradebook this does send the mark immediately so if you’re using an assessment that’s automatically scored which you likely are the moments didn’t hit submit that will be in the gradebook so some instructors like to not check this option uh initially review submissions and then change the settings to send things off the gradebook this certainly isn’t requirements but if you do want to review all submissions before students know what their numeric grade is before they get feedback you might want to check that button later but this is what controls sending the numeric grade off to the gradebook beneath that is the feedback question level reception level uh usually question levels that everyone wants why don’t the feedback be shown by default students will get no feedback for their examination not even being able to see what they submitted and certainly not was correct to show student feedback you can have it set to be immediate soon as they press submit as soon as they answer the question actually so as they’re navigating through the assessment they can actually go backwards and say oh here’s my answer probably doesn’t work in examination context what might work in an examination contact is feedback on submission but don’t forget that students are going to take varying amounts of times to submit this what is likely most appropriate in examination condition and feedback will be displayed to the students on specific dates give yourself some time to work through the submissions you can of course suggest this if you need more time but i would not set this date to be immediately after the last availability i would set that date to be something that complements your regular marking workflow and at that point students can see with every submission you intend for them so the options here that typically are chosen are things like show students how they responded and likely what the correct response would be and you can also choose to show students their scores on each question so again no feedback by default you can choose to give feedback and when you give it and even then once you choose to give feedback you guys actually choose which items students are going to see it does not show students what they submitted by default you need to check off student responses and then you likely want to add correct responses in student assessment course almost through the settings here uh couple more options here uh the layout and appearance by default uh it will be set to random access to questions from a table of contents i always like to highlight that that is the kind of engineering definition of random so students won’t necessarily randomly uh dropped a different question in your assessment means that they have random access to go back to a table of contents or navigate left and right to the questions in the assessment it’s not that they go from question 1 to 17 to 12 it’s that they have the ability if they choose to go from 1 to 17 to 12 otherwise it goes one two three four five um it also defaults to each question is on a separate web page so one question at a time this has the advantage that students will be pressing the buttons for previous next and save ultimately submit for grading more often sakai will store students responses every time they press one of those buttons previous next and save so if a student would be interrupted and know their assessment if their wi-fi goes down their battery dies um other things unpredicted occur they can within their time limits time time limit and availability log back in and pick up their assessment or if we never hear from them again it will auto submit but if you give students all 50 questions on one page

there’s a good chance that if it gets interrupted at question 49 we’ve lost everything if you give students one or a separate part on those pages every time a student hits previous or next sakai is saving it they can pick that up back up but they get interrupted we’ll submit that on their behalf if time runs out so they can press save on that same 50 question layout but to maximize natural saving of that having one question per page make sure that sakai has their responses saved and that is it for the actual settings is that prompted any questions for i don’t know how that would work i don’t understand the practicalities of that setting i’m welcome to make some questions in the chat i’ll move on the grading because that’s of course a natural next step and i’m cognizant of our time but lots of sayings again a familiar path is waiting for everyone the grading interface by default is only accessible by instructors not tas the assign ta option will allow you to get a ta access to testing quizzes which i acknowledge is not intuitive often instructors will promote promote tas to the instructor role just to ensure they have access but it’s worth knowing that there is no access for tas without the assigned ta or a role promotion in the course it’s actually so you can grade the student submissions by question or by the whole submission grading by question uh can help um instructors focus on one response if they need to do actual manual grading or address any issues or verify that the automated grading was done correctly you can go into your submissions questions question one twos before etc and then see all student responses for that particular question so here’s another example here if that could that was a short answer that might be a part where the uh system sakai will mark you all your initial questions and then the marking team might go to the final question in that assessment and then mark a written response for example alternatively you can go through a student by student submissions and see individual students response to all questions so if there’s a visual uh perspective you can click one student work vertically on there or work horizontally on one question all students both are options both grades will ultimately go against the final uh the score for that response that student i see karen constructing a comments that i’m looking forward to other options inside the scores or the grades area is things like item analysis i won’t go too much into this but sakai does offer item analysis for your questions you’re looking for a high discrimination value so what we want to know is does any one question do a good job of discriminating between high performing students across the rest of the assessment and low performing students across the rest of the assessment questions should demonstrate that students who do well in the assessment overall do well on that question students that do poorly on the assessment overall do poorly on that question we want a number like one for their discrimination value here’s a much larger set here with a 19015 student’s responses you can see the discrimination value is a much less clear cut but again we want to know if the question is performing well it’s going to uh straight between those overall responses if the question has a very low experimentation value it could require your attention in that assessment so uh and i’m speaking from personal experience did i actually identify the correct response this multiple choice or did i misidentify it a good signal there and also you may affect if you want to reuse a question type if it is a poor discriminator so that is the test and quizzes tool very complicated but also a familiar path for everyone i recommend uh reaching out to the cpi once you have a thoughts on how you want to construct your assessment and um there is a to make tests and quizzes like tool do what you are thinking about doing it just takes some orientation but once you know what that path is hopefully it’s familiar i’m going to take a moment to read

karen’s comments in the chat so karen’s mentioning that she’s concerned about students um that she wants her students to get the feedback on the responses they gave but uh if she were to reuse those questions in other contexts now the students that answer the first time have some easy access to the question construction and the correct answer i think it’s a good point the in some cases instructors only release a score for the assessment and in those contexts it’s much easier to infer from a student what they got right or wrong um but um the challenges with the withholding information is um that students will be able to make meaning of what they responded to and it could be left with the misconception um others have any strategies for how you can both give students immediate feedback and protect what might be a question you want to use in the future one thing that occurs to me is of course we have a lot of questions a lot of examination scripts on file with the library um and students have access to those it’s a similar kind of situation um one might consider this in the construction of their questions and it might be simple enough that simple variations in a construction of a question if the house was red it can turn into being blue might be enough so that students are trying to remember the meaning the correct response the knowledge behind the question versus the literal response um margaret’s asking about the idea of turn in uh last thing we’ll talk about is sakai’s assignments tool which does have turn-in integration that’s certainly one of its distinguishers between the tests and quizzes tool but the testing quizzes tool does not have turnitin integration so the interest of time uh i’m going to move on to the assignments tool but certainly we can uh revisit these discussions both in the chat ongoing and at the end as well last tool our assignments tool if you haven’t created an assignment in sakai uh this is what the grading interface looks like you get a time stamp for everything students submitted a status of submitted or not of course their names and the artifacts they submit so sakai’s assignments tool has a benefit of being well controlled instructor and student share a submission status students get a submission receipt we get reports not just the originality report that comes from turn in if you choose to use it and you inform your students that you’re using it we also can get submission status is for a whosomeone who hasn’t very immediately um on those nice lists that can create permit unlike tests and quizzes easy to give tas permissions you can even give the organizer rule extra permissions if you have something like a course coordinator who does or market versus market graders you can kind of delete between position permissions there and at a glance testing quizzes to early assignments tool gives us a great uh up-to-the-moment status of submitting submitted graded ungraded and this feedback’s released or not this is what the grading experience can look like uh for an instructor so we’ve got a turn-in report which you can go back to the main page to launch in the greater feedback area turn in we’ve got a file here that we can download to start reviewing and as a marker grader or ta i could add my comments on that file in this box to give them back their students there’s a spot for a grade here of a hundred i can also again download this particular word file uh mark that up microsoft word reattach it send it back here and give students back some feedback for that submission the assignments tool also has concepts of dates that are much more simplified to the test and quizzes tool open date the first day students can click on it and see it due date when when it is due and all submissions after that will be marked as late

and how late will we accept them on the accept until date there are no direct accommodations options in assignments so if students do get additional time you’ll have to accommodate that in the accept until date and then use those late flags to be able to sort by date to identify the student is entitled to this extra time or maybe following a foul of the late penalties associated with your course so that’s the assignments tool again a simpler tool takes artifacts like documents can give you things like turning reports typically used with longer of amounts of time options for grade status which is really useful for working and grading teams so if your tas market graders are doing uh grading you can see the status being updated and graded and are graded appear and they’ll be released typically instructors get control over release and market graders can only submit the actual grade feedback but the instructor can choose to release all or manual release and there’s options like after the uh assessment is done it’s easy to zero out students and submit so that is our three main tools uh crowd mark sakai tests and quizzes and assignments lots to consider in constructing your examination within these tools i can see some questions coming in now in the chat and mark and norbert can be in response to them i’m going to cast my eyes to the chat and see if we have any raised hands but most importantly i think uh keep in mind the cpi’s here help you with this uh exam construction it often is helpful after a very successful uh exam construction in sakai to email edtech at brock you and just simply summarize this is what i think i created would you mind taking a look and confirming this is what i created and of course any step before we’re happy to help with as well that’s asked about uh thailand for each question unfortunately there isn’t an option for a timeline for each question there’s only those limits across the whole assessment it’s a important consideration um there are ways to chain multiple assessments together they come with risks of the fact that you’re training multiple assessments together i have actually delivered such an assessment in a lab which had the benefit of being in the lab but there are techniques to have students have a certain amount of time a whole separate assessment and then move on to the next um if that something interests you edtech proctor.c we can tell you some strategies for that but otherwise you are constrained or students aren’t constrained by the assessment time limit across the entire assessment i see norman answered that already i’m i think i already spied and responded about worth uh summarizing and commenting again that margaret asked about uh most of those questions in the final exam being short answer and can they be around to turn in that might be a moment where you consider is it easier to do this assessment through the assignments tool where i do get a turning report if you have multiple let’s say three major questions with written responses maybe an assignment that gives a prompt of the three questions and asking for a file to be uploaded that can get a report on is the best format for that exam lots more to consider in that but that’s something i would suggest is something to evaluate uh argue also answered but questions randomized in the part can the part also be randomized good question they cannot um you can isolate the randomization to those parts so if i built a question pool uh of 10 questions and i want to randomize five for part one i can do that for part two i can randomize that as well some instructors will even have many parts across their assessment even situations where question one drawn from five question two drawn from five and those variations can actually be quite trivial in that model because um you don’t worry about the pool having more than one drawn from it so my trivial example of a color of a building changing isn’t that bad if you’re only a drawing one but of course be a big deal if you’re drawing two but it’s a good question if you can randomize the sequence of parts

unfortunately you cannot you can certainly do as karen’s asking a longer question at the end of multiple choice if there’s two separate time limits i mentioned that that might be addressed through two separate tests and quizzes though with the complications that might bring with it but otherwise it’s certainly possible to have a dozen multiple choice at the start and a big well-considered written response at the end and that question marking could help with that um the the general idea that the test and quizzes tool is tuned for these more discrete short responses than these long responses no turning report being example may be a factor in how you construct that but you’re certainly asking an interesting question and hopefully through this session and follow we might do with the cpi you can consider what your options might be there and the best path so knowing that’s 11 and we all have other things to do today i’m going to give you all a polite opportunity to move on to what else you might have to do myself from cpi will remain in the chat we are always here at edtech rocky ca to respond the questions about constructing exams and we’ll be offering this workshop again but more importantly the resources are available in the link shared earlier and i’ll drop it in the chat again and uh good luck with your examinations and your continued success and these unique conditions we all find ourselves in