Online Dating Magazine > Columns > Office Hours with Dr. Jim > Compatibility Tests

Office Hours With Dr. Jim
by James Houran, Ph.D

In this column, "Dr. Jim" honestly and candidly answers your questions about dating, love and sexuality. He doesn’t tell you what you want to hear – he tells you what you need to hear. Dr. Jim is committed to offering you guidance based on responsible clinical practice and hard data from the latest scientific studies. Send Dr. Jim your questions today for consideration in an upcoming issue.

Compatibility Tests by Online Daters

Quick Access:
Compatibility Tests - Online Daters or Experts?

Why do some dating sites offer compatibility tests written by online daters and not experts?

For many sites, compatibility questionnaires are popular tools for helping consumers screen romantic prospects. It is understandable that sites attempt to enhance the sense of community and site ownership among users by enabling them to help create the site content. In these cases, such quizzes or questionnaires are much like ice breakers or fun, online activities. Also, user generated content saves dating sites lots of money. But, allowing clients to make life-changing decisions on the information from poorly designed compatibility tools may be a disservice in the long-term.

In the long-term, online daters can make incorrect and costly (financially and emotionally) decisions based on information, feedback and recommendations from poorly designed questionnaires. That said, it is unfortunate that many websites encourage amateur questionnaires, tests and quizzes– without any guidance from specialists or without any obvious disclaimers that such information is likely to be unreliable and invalid. 

Readers are treated here to a rare insider conversation with one of the most prominent experts in tests, measurement and questionnaire research. Meet Dr. Rense Lange, a pioneer in applying modern test theory methods to relationship testing and assessment. Dr. Lange recently fielded some pointed questions from me about the pitfalls of user-generated questionnaires which online daters should be aware.

Is survey and questionnaire design a science?
Definitely yes! Unfortunately, I know that too many surveys are constructed based on “instincts” and whatever else seems important to untrained people. But, this really is not the way to design good questionnaires, especially when trying to predict people’s behaviors. But, unfortunately, all too often questions are selected based on how interesting or appealing they sound.

Instead of a haphazard approach, one needs a systematic, explicit (and correct) theory about people’s behaviors, including factors like a person’s beliefs, attitudes and intentions. Even when dealing with the simplest and shortest of questionnaires one has to know all these issues so that the right ones can be included or omitted.

Designing surveys and questionnaires is often regarded as something anyone - even non-specialists - can do. Why is this?
There are all kinds of bad reasons for this. The most important being that given we can all talk, we also can all write good questions. This reasoning has the same flaws as saying that since we all went to school we are educational experts, or since we have all been sick we all have medical qualifications.

This will not work for serious questionnaires. I find it amazing to see how million dollar decisions are quite often made based on shoddy questionnaires, guided mainly by questionable insights and theories that have long been discarded in the scientific psychological literature.

Maybe this is because people do not know how behaviors can be predicted quite reliably from the right indicators. Also, people tend to over-emphasize their own pet explanations and insights based on anecdotal evidence or mistaken media reports. I have even seen cases where making a questionnaire was mostly seen as a matter of typing - so, they had the secretary do everything.

Most people - even many survey vendors - analyze questionnaire and survey data with traditional approaches like raw-score sums, percentile rankings or percentages. What is wrong about these standard approaches?
Standard design, analysis and reporting are often wrong and incomplete at so many levels. For instance, take the use of rating scales such as “agree completely,” “agree somewhat,” and so on. Here it is often assumed that (a) using more answer categories is always better, (b) some “neutral” category is needed to allow people to be non-committal. Both of these “insights” are wrong. Most people cannot handle more than six pieces of information at a time, so do not give them more response categories than that. In fact, to be on the safe side, four categories are probably fine. Also, neutral categories are usually counterproductive; they rarely get you the information you want. Often, neutral categories do not reflect uncertainty or indecision, but rather they hide socially undesirable answers. That is, they mean “I do not want to say” or “does not apply.”

The preceding is a direct consequence of the use of poor methods of analysis. I believe that our method of analysis should tell us whether middle categories are used inconsistently - just like they should tell us whether someone is giving valid information in the first place. Second, middle categories are often seen as a panacea for avoiding missing data that would foil standard statistical procedures.

For these reasons, the leading edge assessment companies rely almost exclusively on the use of Rasch scaling. It is infeasible to go into great detail here, but this approach is unique in that (a) missing data are inherently acceptable, (b) we can clearly judge the quality of the data and the questionnaire from the responses, (c) one obtains linear (i.e., interval-level) measures and (d) it can be determined whether (and if so, how much) the data are biased by factors such as age, gender and other demographics. Such information allows daters to make more targeted and valid relationship decisions, whereas traditional approaches like raw-score sums, percentages and percentile rankings are severely limited and can even be misleading.

Are there any exciting developments in the field of assessment, surveys and questionnaires?
Surprisingly, professional survey companies (like those who make compatibility questionnaires) almost never use Rasch scaling. Thus, the business advantage for websites that apply these methods in their analytics for customers is huge. My personal favorite development is the application called “Action Plans.” Here we build a mathematical model of the data and this allows us to identify statistical outliers. We then feed this into our software to generate an interpretation for the observed misfit. The result is a person specific and tailor-made diagnostic profile that can be used in a variety of ways, such as customized and automated date coaching or relationship development.

The Bottom Line
Keep these points in mind to help avoid pitfalls in the way you use feedback reports from online surveys on dating sites. If there’s no scientific basis for the questionnaire, then think of it as simply a fun, diversion to start a dialogue with others. Using such quizzes and questionnaires “for entertainment purposes only” is fine, as long as you know what kind of questionnaire you’re using and then act accordingly.

Related Links:
» Interview with Dr. Rense Lange
» The Truth About Compatibility Testing
» Compatibility Tests and Attraction

All Online Dating Magazine content, including the content on this page,
is © copyright by Online Dating Magazine and may not be
republished or reused in any form. You do have
full permission to link to this article.

Do you agree or disagree with this article? Have
more to add? Submit a Letter to the Editor today.