Observer meta-research | Terms of use screen reader | Taxonomy card sort | Experience research | Eye tracking research
VA.gov Platform | Observer guidelines meta-research
Objective: Offer future participants more empowerment around observers in order to enhance VA.gov’s research practice, specifically concerning our trauma informed approach
My Role: I led this research as part of my duties on a federal contract position on the Platform Governance Team for the VA digital experience. I didn’t work alone, however, as this research required collaboration with several other researchers and stakeholders to accomplish. I closely consulted with VA stakeholders about guidance and improvements to our research practice, and other researchers were needed to test out the proposed protocol with participants in their studies
Starting Point - current guidelines
At the time of this study, the Platform guidance for VA.gov research observers had been to join a remote session without audio or video and with “observer” in place of their name. In talking with researchers and other stakeholders at VA, I discovered a shared interest in considering a more trauma-informed research design. For this work I focused on how observers join our remote sessions, since nameless, faceless people could make it feel less inviting and increase feelings of unease with the process.
Plans take shape
Researchers for the VA digital experience have a degree of autonomy. Thankfully several were eager to try a new method of observer protocols during their next round of sessions. I also asked them to pose three questions about observers to their participants at the end of the sessions, after observers left. Asking questions at the end allowed researchers to focus on their study, and it meant less chance for a bias since observers, who were the subject of the questions, were off the call.
Updated guidelines
It took several weeks for the four participating researchers to collect data across their studies. I then collected the data from each researcher. I compiled and analyzed the data, then drafted findings, and offered recommendations for how to proceed. VA stakeholders agreed with the recommended changes, which included asking the participant’s consent at the start of the session before admitting observers and having observers display their name and offering a short greeting before going off camera.
I took on implementing these changes by updating the stated guidance, updating the conversation guide template, and presenting the research during a UX community of practice meeting. I also continued to meet with individual or groups of researchers to make sure they knew about and understood the change.
Participant experience enhanced
This important work uncovered a modified protocol for including observers in research sessions that allowed participants to feel more comfortable. The new protocol proved itself during a pilot run, so I worked with four other researchers with upcoming sessions to expand the trial and also ask their participants about observers being present. During my time on the VA Platform contract, I participated in other work updating guidance for researchers who work on the VA digital experience.