But today's gripe is this: the bug reporting, triaging and fixing system appears to be fundamentally broken.
I will base my commentary of these premises:
1. All software has bugs
2. Users hate bugs
3. Users find it very difficult to produce useful (to developers) bug reports
4. Bugs cause lost productivity for both users and developers (who have to fix them)
5. Bug fixing is often very difficult
6. The sheer number of bugs is a severe psychological barrier to fixing them
Immediate conclusions that follow are:
1. Minimizing the number of bugs leads to greater productivity and happiness for both users and developers, and hence should be a major goal of GNOME.
2. In order to achieve that goal, the quality of bug reports must be increased
3. Because users have limited ability (regardless of motivation) to increase the quality of bug reports, the more the bug reporting system can be automated, the better things will be for both users and developers.
So, what is the way forward? I can think of three broad areas:
1. Forget trying to make bugzilla more friendly to non-technical users. It could probably be done, but other, easier paths will lead to the same destinations. (By the way, the recent improvements are totally awesome!)
2. Delete all bugs from bugzilla and start again. (Tell everyone that we're doing this, and why.) Devote massive resources to the triaging system. Make sure that every bug that gets assigned to a developer is assigned to the corrects developer, has sufficient information to be useful, and is not a duplicate.
3. Bug Buddy needs love. Lots and lots of love. And when it has been loved so much that it is limp, it should be loved some more. Bug Buddy is not usable for non-technical users. Not convinced? Let me show you what I mean. (At last, some bling!)
Testing the latest Evolution from CVS:
Hmm... trouble! Better report it. Press the "Inform Developers" button. First Bug Buddy tells me that my information is out of date. Well, it told me that last night too, and I updated it then, but I suppose things may have changed. OK, download updates and wait while "Bug Buddy gathers information". Inspect the stack trace and confirm that it appears to be, indeed, about evolution-exchange-storage. But wait, what's this? Bug buddy tries to help me by eliminating duplicate bug reports. Cool! But what's all this about products that have nothing to do with evolution-exchange-storage? How is that relevant? Wouldn't it be better to present information about the application that crashed?
Whatever. Let's go to the next step. What's this? I have to choose the application that I'm reporting on? Doesn't Bug Buddy know already? It's got the stack trace. And couldn't the system that launched Bug Buddy tell what has crashed? Never mind. Let's try to pick the application. But wait, it's not there!
Oh, maybe evolution-exchange-storage is not an application. But hang on, Evolution is, why isn't it there? Oh well, try "Products" instead. (Puzzling over the difference between an "Application" and a "Product".) What now? Why are there so many duplicate entries, and how can I tell which one I should choose? And where is evolution-exchange-storage?
Hmmm. Oh, wait. I've been using GNOME for years, and I recall that once upon a time it was called "Ximian Connector". Look under "X", no, ... OK, there is a "Connector" (or two), one of them is (probably) the right one. Hang on, why is "Crossover" in this list? I have never installed that, ever, on any of my computers! Whatever ...
What's next? I have to tell Bug Buddy what version of the product I am reporting on? Can't the system that launched Bug Buddy provide that information? And what about "Severity"? How do I decide what to put in there?
Whew! Finally I get to actually type in the bug report. It appears there should be exactly three steps to reproduce a bug. OK, I suppose I can adapt my experience to that framework. Now, once the information is entered, I get to choose what to do with the report. I have to choose between saving it as a text file, or emailing it. (Why can't I do both? What if mailing fails? Do I have to go through all this again?)
Yay! At last! The bug report has been filed. I feel a warm glow of satisfaction at having fulfilled my duty as a non-developer member of the GNOME community: I have filed a (damn good!) bug report. But wait, have I really? If I check bugzilla.gnome.org it does not appear to be there. Even days later. What's going on? Perhaps the mail did not get through? How do I check? What was that about sendmail in the mailing options?
Discover mailq, and the fact that the bug report was not, as claimed, file. Dismayed to find that now I have to learn how to configure sendmail. It's about enough to make a poor user want to cry...
Posts
16 comments:
I was thinking about this same thing a while back. What do you think about the idea of allowing people to track bugs from their desktop? For example, an application crashes, and the user files the bug report. Then at the end of the reporting process, they're asked if they want to be notified when the bug is fixed.
It might have more features, such as notifications for various stages in the fixing process, or if you want a panel applet. But that's the general idea. Tell me what you think.
And I don't want to email the bug report, i want to send it anonymously. Bug Buddy could probably use some sort of webform for this. Maybe it could give me some random tracking number that refers to my report.
Cheers :-)
If I file a bug off a crash I just copy/paste the debug info from bug-buddy, otherwise I find it pretty cumbersome and (specifically due to the sendmail requirement for mailing) not very useful.
I do understand that it can't exactly use Evolution to send the bug report all the time, but it would be nice if it was smart enough to figure out my email settings unless a required app is the one that crashed (and then fallback to 'save to file' or 'sendmail' with an informative dialog).
Oh comes on, I sent a bugreport from bugbuddy about a beagle crash. Then copied the info from bugbuddy I typed in and pasted it into evolution and sent it. Job done, I got a reply in 10-20mins saying it was fixed in cvs.
Whats so hard about that?, it was my first time.
John: Amen! and bravo!
You said, quite eloquently, what I've been thinking for a couple years now.
Once again, it's right on the money. The same thing happened to me and forever turned me off bug reporting.
:-(
Haha, well said. Fer, bkor, and I were talking about this recently and came to similar conclusions. It's worth noting that bkor & fer have been working hard on an XML-RPC interface (meaning no need for an email connection to be setup) with the biggest delay coming from getting the server updated with the right software.
Also, we discussed (a) not displaying the dialog about getting updated products, (b) having either bug-buddy or the server side figure out the product automatically, (c) automatically determining stuff like version as well, (d) not asking the user for any info not needed (e.g. severity), (e) having bugs automatically be stuck into a "general" product where developers don't see them until they are triaged (meaning that developers don't get worthless emails that bugsquadders have gather more information for them and marked duplicates), (f) allowing users to do anonymous submission of bug reports. Of course, all that's on the back-burner and XML-RPC is first; plus we all have lots of things to do. Anyone who wants to help, though, is welcome.
Deleting all bug reports leads to http://jwz.livejournal.com/154529.html which is obviously not optimal, particularly considering old bugs still get fixed: http://tieguy.org/blog/index.cgi/550.html
I had the exact same experience when I tried to file a bug the other day. Haven't thought about submitting bugs ever since...
Another solution to reduce the number of bug could be to automatically change the status of all bugs that do not show any sign of activity for a long period (3 or 4 months).
Of course an email should be sent a few days before to the peoples in CC list to give them a chance to keep the bug active.
That reporting bugs could be easier (or far easier for Bug-Buddy) I agree with. However, I think it is a very bad idea to ignore/delete all existing reports. Developers already have the option of marking old bugreports obsolete or bugreports without enough information as incomplete (this is always done after the bugreports sits 1 to 6 months in needinfo state).
Bugreports are currently start in the 'unconfirmed' state (basically meaning not triaged). Developers can change their queries and bugmail preferences to ignore these reports until a bugsquad member has triaged them. After triaging (enough info, not a dupe, etc) the bugreport will be marked new (bug doesn't have to be reproducable to be marked new because bugs can be system specific).
Furthermore, because duplicates are and will be filed, starting fresh will actually mean more work. A lot of bugs are reported that already have been fixed. Without the historic bugreports, developers will have to do more work to find out if a bug still occurs in the new version (instead of using simple-dup-finder and finding the dupe).
Now on to Bug-Buddy and reporting bugs in general.
Bugzilla hosts close to 300 products. Just finding the correct product in such a list is always going to be difficult. My current thought for non-crasher bugs (meaning: bugs, translation issues, enhancements, ..) is to have Bug-Buddy allow clicking the product/applet. Then Bug-Buddy starts the users webbrowser and passing the selected product (name of the executable) to the website. The website handles everything else (avoids products that are renamed, allows to better show duplicates, plus loads of other things I'll maybe explain some other time). I haven't discussed this yet with fer and other bugmasters, so I do not know if they will agree.
Now on to the crashers.
For crashers the version is very, very important. The version (and even Bugzilla product) can be specified within a .desktop file. Unfortunately this rarely works within distributions (not sure what breaks it). If it would work, bug-buddy uses that.
What we further need is the name of the executable and a good stacktrace. Using the executable the product can be found (by using xml-rpc or whatever). This can be transferred to the website (xml-rpc, no sendmail) and the bug is created.
To summarize what Bug-buddy will not do anymore (my opinion):
* Handle most of the normal bugreports (except having a way to select the product)
* Ask if the database must be updated
* Asking for product/component/duplicates
* Using sendmail (this has long been planned, hopefully by breaking lots of freezes and getting approval for them at least this part can be fixed before GNOME 2.14)
GNOME products vs bugzilla.gnome.org:
The dialog asking to file a bugreport also appears for products not on bugzilla.gnome.org (Abiword, OpenOffice and loads of others). Abiword I just want to have on bugzilla.gnome.org.. especially if we would create some Productivity suite. Just having products on different sites creates difficult problems. Some could be solved by having something as Malone (which does not exist and I do not want to recreate), however, products that are part of a GNOME release should (IMO must) be hosted on bugzilla.gnome.org (with the exception of freedesktop stuff.. although I'm not happy with having those bugs somewhere else).
Some more background about stacktraces:
Bug-Buddy needs gdb for creating stack traces. unfortunately, not all distributions actually make gdb a dependency of bug-buddy. Furthermore, distributions strip all the debugging information from the libraries and programs. This saves a lot of space, but makes most (99%) of all stracktraces useless. There is a post from alexl explaining just the amount of work a bad stacktrace causes (I'll try and find it if people want to read that).
Fer was planning to do the stack creation differently (library instead of gdb). Furthermore, having some special 'talkback' machine that add the debugging information and only then file the report would be very, very useful. Current thought is get the md5sums of all libraries used on the users machine, send that to the talkback server together with some small core file, use some magic to add the debug info (using debug packages provided by distributions) and then create the stacktrace and file the report. This is still in the 'would-be-very-cool' stage. Hasn't been fully thought ought and there are issues to work out. There is also a Mozilla bug about some open source talkback program (although Mozilla has it easier as they release official binaries.. they have the debug information on the server).
About anonymous bugreporting:
The current system is already anonymous. Providing a name is optional, and you can always create a email address somewhere. We do not care about your name/email address (well.. some products give credit to the reporter in the NEWS file). What is important is a way to ask questions to the reporter. A lot of reports are closed just because the reporter doesn't respond (this is largely caused by the bad stacktrace problem.. although we cannot usually use the stacktrace, I understand that most persons will not be able to provide a good one.. or even understand what we want from them).
Ideally Bugzilla would allow just a username/password and have an inbox or whatever to see the progress (I'm trying to do this with the My Bugs report). However, such a thing does not exist and until Bugzilla supports it (not only just username/ password, also making sure a reporter can track the progress exactly like the bugmail, but without actually using email).
I think a large part of this can be avoided by better explaining what we do with the emailaddress and name. Plus with the latest Bugzilla upgrade I tried very hard to avoid showing the email address on the website (except when logged in). This avoids the spam harvesting problem.
I've probably forgotten some things I wanted to say.. but this is enough for now.
PS: There is a hackish script in place to automatically move a crash report to the correct place (if transmitted via email). Only works for some products. For evolution-exchange-storage you could just select Evolution or gnome-panel (again.. doesn't work for all products.. and it is much better to select the right one). Before the bug is created it will move it to the right place.
Persons wanting to help with triaging or improving bug-buddy / bugzilla.gnome.org are welcome to visit #bugs on irc.gnome.org (say hi and wait until someone responds).
A Report a bug menu item (or in the about dialog) might also be nice. It should again redirect to the website and automatically select product & version.
making bugzilla easier for non-technical people to use can lead to garbage. You want to make it easier for users to give proper reports, not to reduce the amount of information you want from the end users...
(bugbuddy is a different story, there you are going for statistics mostly)
Apple has a pretty automated system, one could use their ideas and move it forward
another source of inspiration could be the talkback application that the mozilla foundation uses in their products. also far more automated than bug buddy
Post a Comment