Large Scale Customer Satisfaction Research

When third parties provide and deliver services, often operating under various business models and structures, how do you measure customer satisfaction? Research to measure customer satisfaction requires careful design. Lessons learned from this large-scale customer satisfaction research provide many insights that can be leveraged by private and public sector organizations delivering services to the public.

In this instance, EurekaFacts conducted beneficiary satisfaction measurements for participants in the Social Security Administration Ticket to Work Program. By design, networks of service providers (Employment Networks), some small, other large, and always independent of the agency, deliver the services. The SSA needed to monitor and guide the quality of service while informing new program participants on the service ratings of the service providers available to them. To this end, the SSA established measurements and dissemination of point of service satisfaction scores via a public-facing website.

The Work Incentives Improvement Act administers the Ticket to Work Program and directs SSA to conduct periodic surveys of the beneficiaries who are receiving services. These surveys ensure that Employment Networks are providing effective and quality services. EurekaFacts  conducted and adaptation of the survey instruments. We also fielded the beneficiary satisfaction among participants of the Ticket to Work program who have assigned their tickets to an Employment Network. The survey assessed the beneficiary’s satisfaction with the Employment Network’s performance.

 

Research Methodology

EurekaFacts developed two surveys. Firstly, through the overall measurements and the aggregation of data at the level of the service provider (Employment Network), a public-facing scoring was developed and presented in the form of online report cards for prospective beneficiaries to use in selecting or changing their employment network. Data overlays of survey data with administrative data and beneficiary characteristics also informed SSA of the impact of the various business models used and the development of outreach communications. Secondly, a separate instrument was used to gauge communications needs and awareness among those eligible for the Ticket to Work program in order to guide these communications. Both instruments were designed and tested using cognitive testing to ensure that the various populations consistently understood the questions. Responses were also tested for accuracy and usefulness across the public, their languages, literacy levels, and their types of disability.

This project utilized a mixed-mode data collection approach with a nationally representative sample of program participants, stratified to ensure the evaluation of each network serving beneficiaries. The methods used included notification postcards, online surveys, telephone reminders, and CATI data collection in English and Spanish. Since program participants were people with disabilities, the data collection instruments and communications were adapted for ease-of-use by the target audience. For the duration of the project, a toll-free telephone help-desk line was available to respondents in all mailings to allow them to verify the validity and authenticity of the study and to obtain help when needed. Both English- and Spanish-speaking staff were available to take respondent inquiries during business hours.

In order to increase the response rate, the entire sample of beneficiaries participating in the Ticket to Work Program received a pre-survey postcard containing a link to the online version of the survey. All those who did not complete the questionnaire online following the advance postcard mailing were sent the survey mailing in English and Spanish (where appropriate). Subsequent waves were deployed to offer program participants new chances to respond, leading to telephone calls as a last step in the process.

 

Research Impact

Upon completion of the data collection period, EurekaFacts conducted thorough data analysis and provided a full report evaluating the effectiveness of the Ticket to Work Program. The report served as a factual basis upon which improvements were pinpointed and prioritized. The Employment Networks (service providers) also were equipped with insights on their standing against each metric, their progress versus previous years, and comparisons against similar entities.

From the public perspective, the scorecards served as a tool for beneficiaries to make informed choices on the service providers available to them.

Valuable insights come from measuring satisfaction and conducting analysis to examine trends and aggregates by service unit, by customer type, and by the business models under which service delivery agencies are engaged. Insights may include specific customer-facing metrics, as well as data-driven systems by which systemic improvements are implemented.