Conduct survey on App Menu redesign #6573
Comments
ninavizz
added
P: default
|
P.S.: Unlike the other "general" survey, this one will only be live for 14 days. FYI. If folks have feedback about the survey itself, pls share it here. We won't be blasting it on Twitter or the forums, until tomorrow, so that minor changes can be added to it that folks might flag, here. TY all for being a wonderful community to work with! |
andrewdavidwong
changed the title
|
Feedback:
|
@ninavizz provided these images and requested that they be added here for use in QubesOS/qubes-issues#6573.
|
@andrewdavidwong TY for taking the time to comb through my text, and for the helpful suggestions! I have implemented them all, as well as the updated image links. I also smoothed-out some of the other bits of language that may have sounded too conceptual (Marta cited a few things as strange to her not-English-first brain, and our few user answers so far indicated the same language could be simplified). So far Jackie and Bessie are neck-and-neck, and my favorite feature between both of the prototypes is getting the most negative response. Why research is so important, lol! I'll post to the forums tonight, and will email you with the Twitter version of the blast, too. Thx Andrew! :) |
|
Feedback: It's kind of hard to map the text explanations to the visual elements in the menu. I wish before the big table there were two pictures (one for each) where each question would be numbered and the picture would have numbered labels and arrows to the corresponding question number But making this would probably be a lot of work and LimeSurvey may not support dropping in images like that. |
|
@deeplow Lime makes it regrettably difficult. I'll certainly give that a second look tho, as you being a not-English-first human only reitterates the urgency of that. @marmarta initially flagged a concern to me that my written delineation of features felt a bit vaguesauce to her... and her grasp of language is pretty phenomenal. But, she also comes to English as a Slavic-native speaker. I'll ping here when there's an update! |
|
Hey @deeplow! Thanks again for the comment and suggestion. I revisited it... and after some thought, decided not to do pictures. Honestly, if something did not make a strong impression with folks, I'm ok with not getting their feedback on a detail. The objective with that question, is purely to get a measure on things folks had extreme feels around—one way or another. No design solution will ever please everyone, but my primary goal with this survey is to get the best measure I can from a survey, on community sentiment. We also cannot measure true usability, by asking folks to watch videos. Much more feedback can happen, later, when I can do in-person user testing from a laptop with a beta from Marta as its functioning app menu. TY, again! :) |
|
I have no doubt that surveys are very useful, and that collecting
private feedback is also useful, but I also know from direct
experience that it is very useful to be able to answer questions of
the form "what information was considered as the basis for X decision
with Y outcome?" by consulting public discussion archives. Answers of
the form "because private information you don't have access to
indicated we should" are not welcoming or encouraging to future
contributors, and makes open source projects feel less open and
inclusive.
This prompted my desire to leave feedback in public on github instead
of in the survey or via mail to ***@***.***
Perhaps future surveys could come with some statement of or perhaps
user-selectable choice about how the results may/may not be published
after?
…---
Additional feedback which was more regarding the survey itself than
the app menu designs, which I figured more appropriate to leave here
rather than in #5677:
There were a couple questions regarding aspects which I truthfully did
not notice in the videos, but that does not mean I would not
appreciate / do not have thoughts on such a feature, causing me to
rewatch the videos to try to notice. There did not appear to be a way
to indicate both simultaneously, and it wasn't indicated which should
take precedence when answering (e.g. unclear whether you more
interested in whether something was conveyed through the video, or
whether the given feature/attribute was apparent/intuitive/noticeable
from the mockup, or what users would think of such a feature/attribute
in the abstract when divorced from the particular implementation). I
suspect I might not be the only survey responder with this dilemma. It
may be ambiguous from the quantitative results of the survey as
written how many answers of "didn't notice" might be more a reflection
on the video format, as opposed to conflating it with a response of
"neutral" / don't care about the feature/attribute in general.
This was the case for me for at least the "Seeing the template for
each qube in its sub-menu", "User created (custom) groups for qubes",
and "User created (custom) groups for apps" questions.
|
|
I forgot that github treats --- as a signature separator (and hides
it) and instead of <hr> when commenting via email. Rest duplicated
below:
Additional feedback which was more regarding the survey itself than
the app menu designs, which I figured more appropriate to leave here
rather than in #5677:
There were a couple questions regarding aspects which I truthfully did
not notice in the videos, but that does not mean I would not
appreciate / do not have thoughts on such a feature, causing me to
rewatch the videos to try to notice. There did not appear to be a way
to indicate both simultaneously, and it wasn't indicated which should
take precedence when answering (e.g. unclear whether you more
interested in whether something was conveyed through the video, or
whether the given feature/attribute was apparent/intuitive/noticeable
from the mockup, or what users would think of such a feature/attribute
in the abstract when divorced from the particular implementation). I
suspect I might not be the only survey responder with this dilemma. It
may be ambiguous from the quantitative results of the survey as
written how many answers of "didn't notice" might be more a reflection
on the video format, as opposed to conflating it with a response of
"neutral" / don't care about the feature/attribute in general.
This was the case for me for at least the "Seeing the template for
each qube in its sub-menu", "User created (custom) groups for qubes",
and "User created (custom) groups for apps" questions.
|
I do not agree with your assertion. Nothing is preventing people like you from giving feedback publicly. Privacy has a big impact on usability surveys and what matters is the aggregate results, and the conclusions taken. Most are only genuine under the condition of anonymity. Anything else is adding complexity and unnecessary scrutiny to what is already an established practice. User-selectable choices for publishing would open the possibility for privacy mistakes and increase the work of @ninavizz to have to publish those. With Qubes OS you have the guaranteed scrutiny of the core developers. So it's guaranteed none of these changes will be degrade security. Quite the contrary. If your concern is that because the discussion / feedback is not public, it creates possibility for abuse in justification of features, just be aware that if all feedback were to be public, then you'd miss half it (but I don't think this was your suggestion). To conclude, I don't get your point. If you want to make your feedback public, just post it here. I don't think anyone has questions over their ability to post things publicly here. So what are you trying to address? |
|
deeplow wrote:
Nothing is preventing people like you from giving feedback publicly.
Sure, but I certainly wouldn't expect any statistically meaningful
sample to do so, given that it isn't suggested or recommended or in
any meaningful way visible. Such is the nature of the survey.
Privacy has a big impact on usability surveys and what matters is the aggregate results, and the conclusions taken. Most are only genuine under the condition of anonymity.
I'm certainly *not* advocating for the removal of a way to anonymously
provide feedback, or even for that to be the default.
I certainly would not want to exclude users with the most sensitive
needs from being understood. They are perhaps the most important to
better understand in order to appropriately serve! I'm only noting
that it would be useful to the broader community of Qubes contributors
and secure operating system development community in general if the UX
preferences of users were better understood in general, that this is
probably not the last survey which will take place, and that some
users are surely comfortable sharing more or less than others. If
effort is being expended to attempt to better understand workflows and
preferences now, it seems IMO a shame to isolate all the resulting
knowledge and insight to a select few forever, and worse yet to *not*
isolate especially the potentially more-revealing free-form responses
without the informed consent of those submitting.
All I'm advocating for is that it be clear up-front to survey
responders what will happen with responses (for example, will these
results be published in aggregate as the first survey's was? If so, I
don't see that stated anywhere), and suggesting that allowing users to
opt-in to sharing beyond the survey operator(s) has utility to other
contributors, and may be worth considering in the design of future
surveys, depending on their nature and generality. (Better
understanding general workflows, for example, being more worth sharing
more broadly (if acceptable) than "do you prefer A or B between these
specific design aspects".)
Anything else is adding complexity and unnecessary scrutiny to what is already an established practice.
The previous Qubes survey provided clear expectations to survey-takers
about what would end up being shared and how, and the results were
insightful! (Especially regarding the relative popularity of distro
choice for templates, for instance, which has been a bikeshed debate
for as long as I can remember and can finally be put to rest, at least
for now.)
So what are you trying to address?
The observation that in the past, some UX decisions have been made
which in hindsight have turned out to be terrible decisions (including
by me! not trying to throw blame here) and eventually reversed course,
but could have been avoided entirely if the contributors had known
more about how real users use things / what real users actually want /
what workflows to test.
I'm referring to times like when qubes-manager was deprecated
entirely, in favor of tray items as (rather incomplete) replacements,
and this was deemed a Good Thing, until it was realized that very many
users really did want something like qubes-manager, and it was
revived.
Or that time when I chose a most regrettable keyboard shortcut for
pausing VMs [1] which, to users who didn't know otherwise, appeared to
cause their whole machine to simply freeze up. This pathological
behavior went undiagnosed for an embarrassingly long time, in large
part because I was aggressively using DispVMs to compartmentalize my
browsing whereas other people were using Firefox's
new-window-in-private-mode shortcut, and the primary insight I had
into how other people use qubes was from the limited set of IRL
friends I had convinced to use Qubes at the time (whose own workflows
were to varying degrees biased by demonstrations of how I used my own
system, in an effort to convince them that Qubes is awesome and worth
a try -- so, definitely not representative of the whole diverse user
base).
[1]: #881
|
|
hi,
as a data point: I wanted to participate in the survey, even though I'm a
"weird" i3 user which almost exclusivly starts apps by pressing shift-enter
and then typing the vm name to launch a terminal there (and another combination
to start browsers), so I opened the survey, saw that I need to watch videos to
particite and closed the survey.
I'm the type of user who given the choice between reading 4-8kb text or
watching 1m video will prefer to read the text. or skim the text.
I wouldn't have minded if the survey had some questions which require one
to watch a video, but a whole survey which doesnt work without watching
some video?
it also sends a very unfortunate message to blind qubes users...
…--
cheers,
Holger
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ holger@(debian|reproducible-builds|layer-acht).org
⢿⡄⠘⠷⠚⠋⠀ OpenPGP: B8BF54137B09D35CF026FE9D 091AB856069AAA1C
⠈⠳⣄
|
|
On Mon, May 03, 2021 at 01:45:18AM -0700, Holger Levsen wrote:
hi,
as a data point: I wanted to participate in the survey, even though I'm a
"weird" i3 user which almost exclusivly starts apps by pressing shift-enter
and then typing the vm name to launch a terminal there (and another combination
to start browsers), so I opened the survey, saw that I need to watch videos to
particite and closed the survey.
I'm the type of user who given the choice between reading 4-8kb text or
watching 1m video will prefer to read the text. or skim the text.
I wouldn't have minded if the survey had some questions which require one
to watch a video, but a whole survey which doesnt work without watching
some video?
it also sends a very unfortunate message to blind qubes users...
Thank you Holger. I second this, although, of course, visually impaired
users will be unlikely to have opinions either way, unless one or other
menus has been designed with them in mind.
I have received many reports from Tor users that they are unable to
participate. Again, I think this sends the wrong message.
|
|
(...also: I am now thinking to do an FAQ page to publish somewhere, to publicly answer some of the above. I will work on composing something |
|
@unman What are desirable video formats, for security? I just uploaded Or, perhaps more broadly—what is the best way to get Tor users videos? Yes, for user research best-practices reasons, I need folks to answer a bunch of questions to the best of their ability, no right or wrong answers after watching the two complete videos. I could also spell that out a little better, in the page text on the survey, too. |
|
@h01ger @unman The "Meet Bessie & Jackie" page on the live survey has been updated, to speak to both our Tor and our visually impaired users. Thank you both, again, for those flags. I also, honestly assumed that Tor users could "just go use clearnet if we say it's gonna be safe, yeesh;" which in hindsight was naive and presumptuous. I also hadn't realized the first question in the first section, required JavaScript to input an answer against. That's pretty dumb. I'll update that to a radio button option, when I publish the updates with a WebM video alternative. FYI, I'll be sure to give the community a full week to respond to surveys with feedback, before I broadly publish them to the community, in the future. This survey needed to be done rather quickly because of a funder deadline, but I appreciate the wealth of feedback from everyone in this post, a lot. |
WebM is probably the best option, as it is royalty-free and so is supported by browsers that don’t include H264 decoders.
The most important part is probably testing the site in Tor Browser. I can think of three major problems that Tor users are likely to face:
Edit: Tor Browser in “Safer” mode disables the SpiderMonkey JIT compiler, as it is a large amount of attack surface. This means that JavaScript performance will be quite poor. The good news is that sites that work well on Tor Browser are likely to also work well on mobile, provided that they can handle the smaller screens. |
|
@DemiMarie The speed thing is a problem... as I can't make the videos "small," as users need to recognize legible text elements in them. Would MPEG4s be ok? I've never heard of WebM. @jpouellet Actually, I replicated the privacy disclosure from the last survey. I am a UX practitioner—a designer and a researcher—this is all I do. Prior UX decisions were made by developers. I trust developers to do their trade well, and need the community to trust me to do my trade well. I don't think that is asking too much. I am on the team page on the Qubes website, and my own portfolio of work is viewable on my website—which you can get to from my bio, here. The last survey was written-up about by Marta, whom I think did a fantastic job presenting meaningful findings in aggregate. It is also a volunteer to-do, to create a User Research page on the website, that will more broadly speak to these things eventually... but that's a volunteer to-do. Yes, I would LOVE to eventually have lots of info on Qubes users for developer contributors to look at, made available... but time/money. Research is regrettably not cheap, and we all want to see Qubes get more funding, to fund things like a robust QA lab... litterboxes that clean themselves for all of our cats, and full-time UX resources. :) |
|
@DemiMarie n/m the above, I found a good conversion tool and am converting them right now! :) |
|
it still starts with a video...that's where I stop participating, sorry. https://survey.qubes-os.org/index.php?r=survey/index&sid=434783&lang=en |
|
I'm not sure if GitHub will allow us to commit such large files, but I'm willing to try (once given video files with which to try). FWIW, I just tested, and Tor Browser on the "Standard" security level allows me to watch the Vimeo survey videos, as well as arbitrary YouTube videos (both without CAPTCHAs, though that might just be by luck of the exit node). |
|
omg! the video sizes are like... massively slashed when converted to |
|
@andrewdavidwong Sorry 'bout that! Access granted. |
I'm able to see which is which from playing them. Will correct filename before committing. |
@ninavizz provided these videos and requested that they be added here for use in QubesOS/qubes-issues#6573.
|
@andrewdavidwong Apologies for that, you are awesome—thank you!! |
|
No problem. They're now available at these URLs, @ninavizz: https://www.qubes-os.org/attachment/survey/tour-bessie.webm |
|
Ok. I've made a copy of the original survey, and made a number of changes—namely:
Please give this a spin and let me know what you all think? @marmarta @deeplow @h01ger @andrewdavidwong — EDIT — So far we have ~300 complete responses, with ~700 total attempted. The extremely high number of attempted-but-abandoned responses, is troublesome—and I suspect because of the aforementioned video issues. I am VERY pleased by the WebM format's performance—the videos look far better on their own, without the distracting UI elements of Vimeo surrounding them. Thank you @DemiMarie for that marvelous suggestion! I have updated all of the video links on the other survey. If you all are ok with the updated version of the survey (above) then I can propose some Twitter and Forum/Email text to blast to alert the community that video issues have been addressed, as well as a section added for assistive-device users. |
|
...having not heard back from anybody, and the first question's Javascript requirement being a blocker for Tor "Safest" settings users, I went ahead and stopped the survey, archived its responses to date, and implemented the changes in the "Deux" version, above. Pls to ping with further suggestions or updates. I also added additional answers per @jpouellet flagging the accidental of former users from the app menu and "how long?" questions. Thx again for all the feedback, folks! Will email @andrewdavidwong with Tweet follow-up texts to speak to users unable to watch videos, Tor/Whonix users, and users dependent upon assistive devices. |
|
FYI, we've thus far heard from two folks that use assistive devices, about their non-visual navigation needs. TY again @h01ger for the nudge on my initial oversight in offering those folks a path for solid feedback, last week. |
|
Lots was commented on in the survey, about things in Qubes unrelated to the appmenu. Those findings I've compiled in a hack of GitHub's "Project" feature, using KANBAN columns as categories: https://github.com/QubesOS/qubes-issues/projects/9 |
I really don't think that this is a good way to try to store and share this information. "Projects" are meant to be "meta" issues or "epics" -- things that contain multiple issues (in the technical sense of the objects tracked in an issue tracking system). So, this sort of survey data is not what "projects" are for. Abusing the feature in this way is likely to lead to unintended side effects, not to mention that it clutters up the projects list, getting in the way when trying to find and manage actual projects. It's also not clear what we're supposed to do with this "project." Is it just going to sit around in the project list forever? I think this sort of data should just go into a public spreadsheet that can be linked in the appropriate places. Or an actual public Kanban board on some other website, which could also be linked in the appropriate places. |
|
@andrewdavidwong All of the Tor reasons that were brought up in the AppMenu survey, had me wanting to try a tool available in GitHub. I'm currently creating a "board" in a visual tool I really like, Miro, and am crossing my fingers that one works nicely. Thx for the feedback—never wanting to throw the PM game off! |
|
This issue will be closed once I publish the findings on the Qubes website (which I will work with Andrew and Marta to achieve). |
andrewdavidwong
added
the
C: app menu
|
@marmarta, can this be closed? |
|
yes, you're right, this is generally done / turned into actionable tickets |
Hello!
As part of #5677 we are conducting user research to learn more about user needs and community desires, for an all-new QubesOS-specific app menu, that @marmarta will begin development work on very soon.
https://survey.qubes-os.org/index.php?r=survey/index&sid=434783&lang=en
I began this project with the first initial survey of QubesOS users, to more broadly learn about user needs... and now that I've generated these two different design directions (shown in videos in the survey), I'd love folks' feedback. No thoughts are "right" or "wrong," and anticipating what might be best for other Qubes users—while helpful on GitHub—is discouraged. What do you think, and how might these directions suit your needs?
The survey is the best place for the feedback I need... as I cannot do meaningful researcher analysis of GitHub comments. Much as I otherwise enjoy them. :)
We will also be cross-posting this to the forums, and the community wranglers @andrewdavidwong and @mfc to the email lists and to Twitter.
The text was updated successfully, but these errors were encountered: