User problem: believing app has crashed when surveys completed

I’ve had a similar query from a number of different participants in the cohort study I am running: when they finish a survey and the Ethica window closes, they believe the app has crashed. And then many of them believe their data will not have been uploaded, and because some of them are doing the study for undergraduate course credit, and want to get that to do well in their course, they become anxious about it. Then they email me …

I am considering putting a short sentence explaining that the app is not crashing on the first Page of the surveys.

However, it would be better if there was a dialog box when a survey was completed. Perhaps something like “You have finished this survey. Your answers will be uploaded when you are next on a wifi network.” with a button captioned “OK” or “Dismiss”. I’m assuming that Android and iOS will actually allow such a dialog, of course.

Would you consider this? Perhaps as an option in the Study design?

I will add: this is a negative experience for research participants–even if they are mistaken about it–and I think it is a barrier to them remaining in the study. It is hard to recruit participants to cohort studies and hard to retain them across the course of the study. I think the fix I have proposed should help, and hopefully it would not be too hard to implement. And as I said before, it could be made an option for the researcher, both as to whether the completion message appears and, perhaps, the content of the message itself.

Hi @atdutoit Thanks for the note. Do you mean the app actually crashes because of a potential bug, or just because the survey screen is closed they think the app is crashed?

I assumed you mean the first one, so we checked all logs we had for potential bugs. So far no indication of crash at the end of survey response. But if you send me the ID of individuals who have this issue, we can look specifically on those cases.

If you mean when they finish a survey on the last page and the survey page is closed they think the app has crashed, … well, … that’s something we had not heard before. The button for the last page of the survey says Submit and when pressed, a dialog is shown saying Done. Thanks. The dialog does not require pressing a button, and it dismisses itself after a second.

But I see your point that people might this dialog and the Submit button. In that case my suggestion would be to just add a page to the end of your survey, saying

This is the last page. Your survey responses are completed. Please press “Submit” below to upload the responses and close this page.

This should provide the same functionality as the dialog you suggested. We will continue looking at this closely and check if there is anything to be done by the app to make it more explicit that the survey is completed.

Thanks

Hi Mohammad,

I like the idea you have for the last page; I think I will put that in.

However, I had noticed that the “Done. Thanks.” dialog does not appear on my phone any more, whereas it did previously, and I think it probably does not appear for others either. (NB. I just sent myself two separate surveys to check on this and confirm it.)

Perhaps there has been some change to Android re: popups like this? (I am using a Pixel 3a, Android 9, Build PQ3B.190705.003)

Cheers,
Anton

I have made the change you suggested to the EMAs. I’ll seek some informal feedback from users on it.

A further thought about the “Done. Thanks.” dialog – I would like to see some changes to it:

  • make the wording more specific; perhaps, following your suggestion above, something like:

Your survey responses are completed, and will be uploaded. Thank you.

  • make it persist longer on the screen - it does flash up very quickly. Or put a button with “OK” or “Dismiss” to dismiss it, if this is possible

  • make it larger - it’s quite small and not that easy to see

I think the issue is a UI / user psychology one rather than a technical one–for everyone who has emailed me about this, I have checked their data and it has always been present. However, even the perception of such a problem is likely to affect participant retention: people may think, “The app doesn’t work” and delete it.

A supplementary point: I now have over 250 people who have participated in the study for at least some time, and there is a distribution of technological literacy, one might say. This has become apparent to me in informal discussions with research participants. I think–but cannot prove–that this affects their level of patience with the software and likelihood of dropping out.

Hi @atdutoit

Thanks for the comment. I will make sure that the dialog is tested against Android 9 and if there is any issues is resolved in the next build.

Regarding your comments, I agree that having the option to customize the message that is being shown at the end of the survey is a very useful feature. This is been in our roadmap, but we have not got a chance to implement it yet. We should add it soon.

About the changes to the dialog UI, that’s the standard used in Android (called a Toast message). Basically you can show two types of dialogs: either a quick short one that is dismissed without user interaction, or a long one that user would require to tap on “OK”, or “Cancel”, or similar buttons to dismiss it. As I explained above, if a survey requires a long confirmation dialog upon submission, they can use a survey page for that purpose. It provides the same interaction flow. That’s why we chose to use the first type of dialog (short, quick, with auto-dismiss).

Note that while this Toast message might confuse some people, converting it to a long actionable dialog will annoy others who think “I know I just finished the survey. Why are you asking me to confirm that I know that?”

So I guess the best option here is to add a last page to the survey and explain the survey is completed, as we discussed above.

Bests,
Mohammad

Thanks Mohammad. I’ll look forward to the customisable message … for the next project! If there is some way to make the Toast message larger and appear for longer, I think that would be good. But it’s a good point about the actionable dialog, and as mentioned I have implemented the last page you suggested.