The most rewarding and involved projects I’ve worked on here at Axcient entailed running two Beta Programs in 2013 that served as gateways for two very big feature releases of our data protection platform. At Axcient, the entire company – from Engineering and Product Management to Sales and Marketing – focuses on the success of the Beta Programs. This allows us to gain valuable product insights on issues that happen in the real world so that they can be addressed before the product is released to the general public.

At Axcient, a Product Manager is responsible for running a Beta Program. Here are the top five lessons I’ve learned as a Product Manager running Beta Programs:

  1. A Beta Program is not a substitute for a proper Quality Assurance cycle – Treat the Program as an extension of QA to reduce potential risks once the product is released. It should be considered as a way to validate the severity of bugs found during QA. The risk of calling a product beta-ready without acceptable quality can hurt the relationship with beta participants. One of the many purposes of a Beta Program is to establish and build a long-lasting relationship with the participants so that we can reach out to them again in the future to gain valuable feedback. A beta-ready product should provide a user experience that is not too far away from a production-ready product. The last thing we want is to drive away these participants if the product is too buggy to use.
  2. A Beta Program needs to be properly planned in order for it to run smoothly and successfully
    Here are some pointers:

    • There are things outside of a Product Manager’s control; for example, finding dedicated participants is not trivial. There might be participants who don’t actively participate during the Program, or those who don’t offer meaningful feedback. Make sure to sign up extra participants to ensure you get adequate feedback.
    • The organization as a whole needs to come together and agree on the objectives of the Beta Program and its priority. If the Beta Program and what is uncovered from the Program should be treated as top priority, everyone’s involvement needs to be accounted for.
    • Recruit participants well ahead of time, taking into consideration response time from the participants. The participants also need to accurately represent the target market.
    • Decide how feedback should be collected ahead of time. Should it be through emails? A dedicated support phone line? A private online forum? How will the rest of the organization make sense of the collected data? Who should do the collecting and who should do the analysis? Logistics can take a bit of time to sort out, so plan for it.
    • Once the Program starts and participants start testing, issues will be uncovered. How much time will it take to triage and fix all the must-have issues before the official release? Adequate time needs to be allocated for this. There can also be critical bugs that are release-stoppers, so adjust as necessary and communicate changes to all stakeholders.
  3. A Beta Program should be long enough to capture enough meaningful data – If a program does not last long enough (due to pressures to release by a certain date), then the program does not fully serve its purpose. Here is where the Product Manager needs to effectively communicate the purpose of the Program to the rest of the organization, including detailed plans on how to execute the Program so that each milestone is met in line with the overall goal. An example of a milestone is to establish weekly goals to test out specific features, or to exercise certain tests to measure performance improvements, etc.

    If a Beta Program is forecasted to last longer than what the rest of the organization had hoped for, consider a phased approach. Phase 1 of the Program might only be available to a small set of participants, while Phase 2 gradually introduces more participants. Meanwhile, continue to communicate progress of the Program back to the rest of the organization.

  4. Don’t ignore issues uncovered from the Beta Program – After all of the work participants put into participating in your Beta Program (including time), they should not feel like no one is listening to them, and that their time and effort in beta testing was wasted. Keep them in the loop.
  5. The Product Manager must lead the Beta Program to a successful conclusion – In the early part of the Program, Product Management and Engineering should be heavily involved in making sure that the product works as expected. The Support team needs to be heavily involved once the product stabilizes so they can gain the knowledge needed to support the product once it is officially released. The handover from Engineering to Support needs to be transitioned well, including having Support shadow Engineering during the transition to gain familiarity with the new features. Once Engineering finishes transitioning to Support, Product Management should still stay in the Program together with the Support team until the Program concludes. Here, the Product Manager should continue to monitor the daily activities involved, actively monitor the progress, and be the voice of participants reporting any issues that are uncovered. Having routine meetings with Engineering, Support and Product Management to discuss uncovered issues has worked well for us. Continuous Product Management involvement ensures that a product is not released before it is ready for the general public.

In summary, a Beta Program is the first exposure to the public of a brand new product or a set of product features, therefore it does not replace the need for a proper Quality Assurance cycle. In order for the Beta Program to run smoothly and successfully, the Product Manager should properly plan it, keeping in mind that if it is not long enough to capture enough meaningful data, the Beta Program will not be too worthwhile. Hopefully these friendly tips will help you in organizing and executing your next Beta Program!

Nancy Chu is an Axcient product manager by day and a beauty blogger by night: http://www.msnancysfancies.com

nchu