After testing ChatGPT, Claude, Gemini, and Microsoft Copilot, I wanted to see how Dovetail and Perplexity would handle the same survey. Would these two tools give me results I could actually use?
I gave both tools the same survey data and prompt. The survey included responses from 34 UX research professionals. I compared their outputs with my verified analysis, checking their counts and how they grouped and counted ideas in written responses.
Both tools got several questions right. They correctly counted respondents' roles, years of experience, and frequency of AI use. They also correctly counted responses about human verification, comfort presenting reviewed AI findings, and experience finding AI errors.
That was encouraging. But as I checked the remaining questions, the problems became clearer.
Where the counting broke down
One question asked which AI tools respondents had used. They could select more than one. Dovetail miscounted three of the five tool entries, while Perplexity miscounted four.
The question about trust in AI covered nine research activities. Both tools correctly counted only five of nine activities (55.5%). They made errors on the same four: usability test tasks, quantitative analysis, recommendations, and stakeholder summaries or reports.
That caught my attention. Two tools agreeing on a result could make it seem more credible. In this case, agreement did not establish accuracy. Comparing their answers with my verified counts exposed errors in both.
These results also showed why checking a few correct answers was insufficient. Both tools handled several questions correctly, yet other counts still needed correction before I could use them.
Written responses required more work
The written responses revealed another difference between the tools.
For the question about which research activity respondents trusted AI with most, my analysis identified 94 mentions across distinct categories. Perplexity grouped and counted the responses, but its counts were off by one or two. It came closer than Dovetail, which provided extensive explanation without usable counts of the distinct ideas.
The question about which activity respondents trusted AI with least was more problematic. My analysis identified 57 mentions, including 44 mentions of AI reliability concerns. Dovetail did not provide usable counts of distinct ideas. Perplexity's groupings were confusing and did not match my categories. Neither produced groupings and counts I could use.
What this meant for my work
I still had to check the numbers and assess whether the categories represented what participants said. Dovetail's lengthy explanations did not remove that work. Perplexity's closer result on one written-response question did not establish that its other results were accurate.
For researchers considering these tools, I would recommend verifying every reported count against the source data and checking written-response categories against participants' actual answers. Agreement between tools should also be checked.
What this study can tell us
This evaluation used one survey with 34 respondents. I assessed counts and written-response analysis. I did not evaluate the additional statistical analyses in the AI outputs. These findings describe the outputs I tested, rather than everything either tool can do.
Both tools produced some accurate results, but neither delivered a complete analysis I could use without human verification. Perplexity came closer on one written-response question. Across the survey, I still needed to establish which findings were dependable before using them in a research report.
I also noticed a similar pattern in both studies: the AIs seemed to struggle as they had more text and more separate items to keep track of. Simpler questions were handled better, while the nine trust ratings and written responses revealed more problems with counting and grouping. I cannot say that the amount of text caused those errors, but it is something I would like to explore in future testing.
Across these two studies, all six AI tools produced results that needed human verification. Their strengths and errors varied, but none delivered a complete survey analysis I could use without checking and correcting it. For me, the practical question is how much verification and rework an AI tool requires before its output is dependable enough to use.
Email: theresaw@columbus.rr.com
LinkedIn: theresa-wilkinson