Over the past year, the structured feedback you provided while using the IE9 Platform Previews, Beta, and Release Candidate has been invaluable to helping us build the most reliable version of IE ever. The error reports that you send us via Windows Error Reporting and other methods help us fix the most impactful issues experienced by users. Your real-world browsing enables us to identify reliability problems in ways that cannot be matched by internal testing.
In this post, we take you through our process of engineering IE9 reliability. We describe the various channels you’ve used to send us feedback. We describe how we categorize and prioritize the data to address the most impactful issues first. We conclude by sharing data of our own that show how IE9 reliability has improved since IE8.
We encourage you to keep sending feedback through our telemetry systems as you use the final IE9 release. Ongoing error reporting allows us to continue to deliver reliability improvements to IE9 through future Cumulative Updates.
Reporting the Crash through Windows Error Reporting
If you encounter crashes in IE, features such as Automatic Crash Recovery help mitigate the impact of these crashes, but we want to eliminate their occurrences in the first place. We use several different mechanisms to gather information about the crashes. One of these mechanisms is Windows Error Reporting.
For users who opt-into Windows Error Reporting, IE collects information about the state of the browser when the crash occurs and packages the information into an error report. This information helps developers debug the root cause and fix the crash. Internet Explorer can also send error reports if you encounter hangs while browsing. We’ll cover the topic of hangs in a future post.
Keep in mind that Microsoft protects your privacy when submitting error reports. IE asks for your permission before securely transmitting error reports back to Microsoft.
Windows Error Reporting dialog
Throughout IE9 Beta and RC, we used additional mechanisms such as the Send Feedback Tool and Microsoft Connect to listen to your feedback on IE’s reliability. These mechanisms may end up reporting duplicate issues back to us but we internally de-duplicate reported issues. Your reports help build a large and statistically significant volume of data that allows us to determine the reliability issues that are meaningful to address.
Understanding How IE Crashes
Once we aggregate a full sample of crashes, we categorize the data based on the source of the crashes. This helps us generally understand what areas are affecting IE’s reliability more than others. We devote resources to driving improvements in each category.
The following categories contributed to reliability problems during IE9 Beta:
IE Code Bugs
Faults in IE code contribute to a sizable portion of the overall crash volume. The ratio of IE crashes to third party crashes is higher in pre-release versions (Beta and Release Candidate) due to the amount of new code introduced.
Add-ons
While add-ons are a core part of IE’s browsing experience, they are also the primary cause of reliability problems in IE. Add-on related crashes typically occur upon opening a new tab or upon IE startup. Some add-ons are incompatible with newer versions of IE and may cause crashes.
ActiveX Controls
Many of today’s Web sites use ActiveX Controls like Flash, Silverlight, and QuickTime to display interactive content and video. Since ActiveX controls are essentially Windows applications that run in the browser, poorly-written controls can cause the browser tab to crash or hang.
Graphics Drivers
IE9’s transition to hardware accelerated graphics is highly dependent on the quality of the graphics card drivers in the ecosystem. We found that graphics card drivers caused 41% of the crash reports we received from the Send Feedback Tool during IE9 Beta. These crashes manifest in similar scenarios as add-on crashes so they have a very noticeable impact on regular browsing.
Other Third Party Software
A wide range of software programs, such as antivirus tools and custom download managers, can affect IE’s reliability.
Drawing the “Failure Curve”
The next step is to organize the data from each category in such a way that we can systematically investigate the crashes. We prioritize the issues using “failure curves,” which are bar charts where each bar represents a unique crash and the crashes are ordered from most frequently encountered to least.
After we construct the failure curve, we investigate each unique issue starting with the most frequently occurring issues at the top of the curve. The issues in the tail end of the curve are generally encountered in specific hardware and software configurations. They may also be important to fix, but we start with the top issues to ensure that we maximize the impact of our efforts.
The above failure curve example shows how we’ve successfully addressed the top 50 crashes in IE code from since IE9 Beta.
Ensuring Reliable Experiences
Through the work we do on the IE team and with third party developers, we continue to improve the reliability of browsing experiences by addressing a majority of reported crashes. This section describes how we make the final push for each category:
Internal Reliability Issues
We log bugs for the most frequent crashes in IE code from the Beta and Release Candidate. We address them as we continue to monitor the volume of data for new issues.
We also combine the various sources of telemetry to good effect. We use your Send Feedback reports to help us reproduce the issues that we investigate via Windows Error Reporting. A successful recreation of the crash helps developers zero in on the root cause and helps testers verify that the crash is fixed.
The failure curve chart from above paints an accurate picture of the progress we’ve made fixing IE issues. In total, we’ve fixed the top 85% of crashesin IE code reported since IE9 Beta, and we’ll continue to combat the long tail in cumulative updates.
The improvements in quality and reliability in IE9 are substantial. The results of Stress and Longhaul tests continuously running on the final IE9 release run 6-7 times longer than IE8.
Third Party Bugs
We work with vendors from around the world to address the top third party reliability issues from the IE9 Beta and beyond. We use the WinQual program to share the error reporting information with vendors. We ensure that they only have access to error reports relating to their own products.
Faulty add-ons or graphics drivers are known to cause IE to repeatedly crash in important usage scenarios. There are two built-in mechanisms in IE that help us work around these problems:
So far, the error reports that you’ve submitted helped third party vendors address some of the top crashes from IE9 Beta. We updated the software fallback list for IE9 RC and streamlined the Upgrade Advisor experience (details to come in an upcoming post).
- Add-ons: We use the Upgrade Advisor to disable any known incompatible add-ons so that you can launch IE successfully if you previously had one of these add-ons installed. When the Upgrade Advisor blocks an add-on upon startup, IE notifies you of the action and gives you an option to check for an update to the add-on.
- Graphics Drivers: IE9 includes a “software fallback list” of known incompatible graphics drivers, compiled from error reports and from internal testing. If you use one of these drivers, IE will use software rendering instead of GPU rendering for accelerated graphics. Running in this mode prevents the drivers from crashing the browser.
Your Feedback Matters Every Day
We followed the approach described in this post to address the top crashes across IE and various third party components from IE9 Beta and IE9 RC, thanks to the error reports and other types of feedback you contributed to us. The result is a high-quality IE9 release as indicated by the record scores in Stress and Longhaul reliability.
Reliability engineering is a closed feedback loop. After the final release we will use the feedback from an even larger group of users to continue improving reliability. As the Web and the software ecosystem evolve, we’ll work to build an increasingly reliable experience by shipping fixes to top problems through Cumulative Updates.
Thanks again for using IE9 Platform Previews, Beta, and Release Candidate and submitting feedback in all its forms. Your contributions help make IE more reliable every day.
—Herman Ng, Program Manager, Internet Explorer
More...