Earlier this week I opened an email from someone I didn’t know asking me to take a survey about equity in education. I closed the email without taking the survey. I’m tired of taking surveys, especially from people I don’t know. I’m saying no more taking surveys unless I know the person sending it, or if there is a prize or compensation attached – I can be bought, often for the price of a taco. I realize I’m being a survey hypocrite since my organization just ran a large successful survey project. I know the value of a good survey and the data generated can be powerful. But I’m tired of being over-surveyed by askholes (people who ask questions and don’t listen to the answers); so many of the surveys are well intention but poorly designed or executed the data becomes meaningless and waste time, resources, and burns goodwill.

Surveys do not Substitute for Community Engagement
Often times organizations produce a survey “to hear” from the community. Just this week, I’ve gotten four request to complete surveys. All of these well-intention surveys are asking questions about topics important to them, but I don’t know what I’m gaining or what communities of color are gaining by filling out the survey. Is launching a survey a way of ‘checking the box’ and saying the community had a say in the project? Will things change because of the survey results? Is the survey collection method right for the community being asked? Asking without first listening is fakequity; good surveys are driven by the community and the community has ownership over the process and data.

How to do better surveys
A good survey design produces good survey results; vice versa crappy design produces crappy results, and fakequity in the design produces fakequity in results. Designing a survey rooted in the principles of equity will produce more equitable results. This isn’t hard, but it takes intentional thought and work.

One of the first steps is to look at who has a say in the survey design. The project should be centered in communities of color, which means you are doing the upfront work of gaining trust and buy-in, communities of color have a say on what is included and counted, and communities have control of the data. Inclusiveness is a key step in engaging the community.

My organization just completed a big survey project. Our design team was made up of a diverse group of stakeholders. We were thoughtful in putting together the team to include a mix of people, including parents of color, school partners, and community based organizations. We also invited partners to expand the table as we went along. This team guided the survey design process. They had a say and ownership around what data was captured, how questions were written and translated, and how surveys were disseminated and collected.

On the flip fakequity side, I recently reviewed survey data from a process put together by a small insular team. The return rate from people of color was so low it wasn’t a representative sample and the results were invalid for communities of color, essentially wasting the time of the people of color who took the survey.

Survey Delivery Methods – SurveyMonkey FTW (for the win), WRONG
Online surveys have become the default method of collecting surveys. Online methods definitely have their perks: cost effective, easy to put together, eliminates the need for data entry, and they are easy to disseminate. However, we need to ask is the ease of use still the right way to get results? One of the biggest limitations with online surveys is in-person engagement is lost. Online surveys are a one-way communication stream out then in, they don’t allow for dialogue to happen while taking the survey. Many people of color, especially non- or limited-English residents, benefit from having a translator and/or cultural broker who can explain the survey questions to gather the feedback.

My colleague Jondou shared a story about working on a survey translation into Somali. His Somali colleague looked at the survey and said “Somalis don’t talk like this. We are more direct and we ask open ended questions. You took an American survey and translated it into Somali, that doesn’t make it a Somali survey.” He also shared a similar story out of the Chinese community. Jondou was orally translating a survey and the survey taker was getting agitated. After a few questions he paused and said (in Chinese)  “I know this isn’t how Chinese people talk, these are American questions,” after  acknowledging the questions were biased and not culturally appropriate they were able to continue. These types of dialogues and exchanges can’t happen over a computer screen; they happen in person and where the experience is validated.

Paper surveys are much more adaptable for focus groups, interviews, and guided conversations around survey collection. Paper also wins when you have little children who want to ‘help’ you complete a survey – the picture is of a survey completed by a preschooler; can’t get that experience over a computer screen. Community engagement starts young.

Invest in People for Better Results
Racial equity work happens best when there is a relational component. When we invest in people we get better results. As you design surveys think about the people part of the project, what is the experience you want to give people? Don’t focus on the data, focus on people of color.

Quick tips:
Design team: Ensure you have diverse voices helping to write the questions.

Translation: Translate your survey to make it language accessible, and ensure the translation is appropriate and high quality (have a second person proof the translation). We also have to acknowledge having it translated is at best a minimum step, it is still an American/dominant culture survey and may need cultural brokers to help explain the nuances of the survey.

Allow for multiple ways to take a survey: Invest in multiple ways to take the survey, online, paper, focus groups, interviews, etc.

I have more to write about around survey designs and how to keep the experience positive for communities and people of color, but I’ll save the rest for future posts. Feel free to post questions or email us if you have thoughts or questions for a future post on surveys, fakequity@gmail.com.

Posted by Erin Okuno