DESIGN LAB

Human Computer Interaction research on improving peer feedback in online education

This experiment was a result of when I previously conducted Human-Computer Interaction research with Rachel Chen as a Research Assistant at the Design Lab with Catherine Hicks and Vineet Pandey, with the lab headed by Don Norman, Jim Hollan, and Scott Klemmer. We explored how to improve peer feedback, collaborating with PeerStudio at Stanford. I created artifacts, helped with experimental design, data code (qualitative), read and found papers, helped with creating interfaces for their experiments, and presented at the UCSD Computer Science & Engineering 2015 Undergraduate Poster Session.

challenge

Previous research has shown that peer feedback is often unreliable-- how can we make it better so it can be a viable solution to problems with online classes?

Peer feedback is needed of MOOCs (massive online open classes), since there are too many students to a single teacher. However, peer reviews without much instruction lead to grader inaccuracy, inconsistency, and even the perception is that it is unreliable, and the quality of feedback is low compared to a teacher’s or expert's reviews.

hypothesis

Previous Research

Communication and expectations lost online

Communication, norms, and expectations that occur through language, assessments, and social presence in classrooms often provide cues to the classroom's beliefs that lead to adoption of these beliefs for learners.

Proposed Method

Context and training provide more support

Experiments on the “framing effect” (changing behavior based on how information is presented and adjusting the environment) has hinted that for peer reviews, possibly communicating and framing the experience on how an expert would approach the review and providing more context to the situation, would train novices to think more like experts, also known as "context scaffolding".

Hoped Outcome

Higher quality peer reviews and feedback

The hypothesis is that context scaffolding can train novices to pay more attention to fundamental principles integral to success, and to reduce biases, rather than focus on the nonessential parts, therefore increasing the quality of the review by making it more reliable and trustworthy.

measuring success

We quantified the "quality" of the review by focusing on the types of feedback participants gave-- deep or surface.

Other studies show that the difference between experts and novices are the types of feedback they give, and experts usually focused on deep feedback as opposed to surface feedback.

deep features

parts of the assignment that rest on the fundamental principles that are integral to the work's success

"You could focus on the successes of your previous experience to show the value you brought at your previous job."

surface features

cosmetic, non-essential, and individual choices, such as grammar and word choices

"Make sure to use a more legible font."

method

We conducted this experiment with resumes with an experimental group and control group.

With the experimental group, we gave them training on how an expert would review a resume and provided more context on the job description, helping to frame the participant to think like an expert.

With the control group, we simply gave them basic training on the experiment. At the end, we asked both to give feedback on a resume.

outcome

experimental group

  • gave more diverse feedback (deep and surface)
  • gave more positive feedback

control group

  • focused on surface features

presentation

personal takeaways

Hearing and discussing with brilliant minds about HCI Research

This experience was extremely valuable to me, to see how psychological research can be applied in the HCI realm, and also being around so many intelligent and amazing people. Listening to people consider different ideas, share their thoughts, and share research on current topics is always great, especially when hearing about the complexities and different dimensions in all different topics in HCI. I also learned while at the Design Lab to take control and intiative of my own growth and learning and goals, where I can ask for a change in direction if I don't see what I am doing currently fitting my future overall goal, or if I wish to do something different. The brilliant minds there have also taught me to continually be curious, and to continue to ask questions (and the right ones!) And ultimately, never being afraid to fail and to get feedback on your work, because although your ego may hurt a bit at first, it will ultimately be beneficial to hear from different people about your work.