Philip Shapira, School of Public Policy, Georgia Institute of Technology, Atlanta, Georgia 30332-0345 USA, email: philip.shapira@pubpolicy.gatech.edu ; Jan Youtie, Economic Development Institute, Georgia Institute of Technology, Atlanta, Georgia 30332-0640 USA, email: jan.youtie@edi.gatech.edu

August 1994

ABSTRACT: This paper draws on the evaluation component of the Georgia Manufacturing Extension Alliance: Overview of the Evaluation Plan. GMEA Evaluation Working Paper: E9402 Atlanta, GA: Georgia Institute of Technology, Economic Development Institute, August 1994. It describes procedures for a minimum three-month pilot of customer profile and evaluation of services components.

PROCEDURES FOR PILOT OF FIRM-LEVEL CUSTOMER PROFILE AND CUSTOMER EVALUATION OF SERVICES

I. Overview and Objectives

Georgia Tech Economic Development Institute is improving and systematizing its evaluation procedures and protocols. The broad aims of this evaluation system are to (1) provide consistent feedback about the effectiveness and impacts of services to customers; (2) provide systematic learning about what works and why, to support ongoing improvement of services; (3) meet the needs of the state sponsor; and (4) meet the reporting needs of the federal sponsor, the National Institute of Standards and Technology (NIST).

There are four major information procedures which will contribute to the overall evaluation system: customer profile, activity reporting, customer evaluation of services, and customer progress reporting. The customer profile provides basic logistical and industry information about a client, such as contact name, address, industry, employment size, and defense sales dependency. Activity Reporting tracks the specific services provided to customers by program staff. The customer evaluation of services obtain client attitudes toward service delivery and perceptions of the value of this service soon after receipt. The customer progress report follows the progress of selected customers to assess impacts one-year after services are delivered.

This document describes procedures for a minimum three-month pilot of cutomer profile and evaluation of services components.

II. Firm-Level Customer Profile

The customer profile provides basic logistical and industry information about a client, collected on a consistent basis. The customer profile protocol contains these key components:

• Company name, address, city, state, zip, country
• Contact name, title, telephone, fax, e-mail
• Type of client (e.g., manufacturer, individual)
• Single or multiple plant operations, location of head office
• Defense sales and annual sales/shipments
• Employment
• Products / SIC
• Referral Source

Customer Profile Form

A form has been prepared which contains the above elements. The form draws upon "NIST-mandated" reporting requirements. The form was circulated for comment in an earlier form and revised, resulting in the attached form. This form will be available in PROTRAC for electronic completion and recording.

Customer Profile: Implementation Procedures

The Customer Profile form and administration will be tested in a minimum three-month pilot beginning in June 1994. (This period may be extended if sufficient information has not been obtained.) Once these procedures have been tested, analyzed, and verified, management may modify them. The pilot will not only cover GMEA firm-level assistance and services (i.e., not all EDI activities).

The procedures for the pilot are:

1. Customers receiving significant assistance and services should be profiled. This includes:

• All firms receiving Formal Assessments and Technical Assistance Projects.

• Firms receiving assistance through Initial Meetings, Informal Engagements, and Referred Technical Assistance Projects (i.e., commonly recounted in activity reporting)

Please see the Activity Reporting guidelines for definitions of Initial Meetings, Informal and Formal Engagements, Technical Assistance Projects, and Referred Technical Assistance Projects.

For Training and Networking, profiling procedures and under development. Staff should consult with Associate Directors for guidance on customer profiling in these situations.

2. Agent completion. Agents should complete the customer profile at the end of the first engagement or by the second interaction (if more than one interaction is envisaged). The following guidelines should be applied:

• Most of the questions on the profile can be completed using information gained in the normal course of discussion and preliminary exploration with a customer, eg., name address, nature of production or business lines, employment size. In other cases, the field agent will know the customer and be able to determine, for exmaple, whether the customer is a single or multi-plant operation. Items which are not discusses or known can be raised towards the end of the conversation.

• At no time should a form be given to the customer to complete. Nor should the questions on the form be asked in a "formal" way at the beginning of contact to "screen" cutomers.

3. Definition of terms. Susan Vogel will provide answer questions about the definition of any item requested on the company profile. In particular, the following definitions apply:

• Customer. In a single-plant firm, the customer is the firm. In a multi-plant or multi-office firm, the customer is the particular site, branch, or location receiving assistance.
• Defense sales: shipments to federal defense agencies, prime contractors to federal defense agencies, subcontractors to federal defense agencies, U.S. Department of Energy (DoE) facilities or contractors of the U.S. DoE. NIST considers any firm with 10% or more defense sales to be defense-dependent.
• Employment: full time equivalent (full-time, part-time, and contract employees).

III. Customer Evaluation of Services

The customer evaluation protocol is implemented in the period after services have been provided and a project closed (see below). The purpose of the customer evaluation is to obtain client attitudes toward service delivery and perceptions of the value of this service soon after receipt. The customer evaluation is intended to evaluate assistance and services, not individual staff performance. The evaluation model assumes that customer attitudes toward the services received will affect implementation of results, business outcomes, and utimately the larger economy.

The customer evaluation protocol contains these key components:

• customer confirmation of services;
• measures of satisfaction with the assistance and services along several dimensions (e.g. timeliness, staff knowledge);
• an indication of whether the customer intends to take any action as a result of assistance, services, or recommendations provided;
• customer staff time commitment;
• measures of the projected dollar value of the assistance and services, along several dimensions;
• other impacts, such as introduction of new methods, technologies, processes, training, etc.
• whether customer desires additional follow-on assistance.

Customer Evaluation of Services: Survey Form

A questionnaire has been prepared which contains the above elements. The questionnarie (1) draws upon several items already used in the previous Evaluation of Benefits and Results; and (2) includes seven "NIST-mandated" questions. The survey was circulated for comment in an earlier form and revised, resulting in the attached one-page, two-sided form. A telephone script paralleling this form also has been created.

Customer Evaluation of Services: Implementaion Procedures

The survey form and administration modes will be tested in a minimum three-month period beginning June 1994. (The pilot period may be extended if sufficient information has not been obtained.) To establish and validate procedures, the pilot will be conducted from a central location, out of the GMEA Director's office. Once these procedures have been tested, analyzed, and verified, management may modify them, for example to incorporate more decentralization. The pilot will only cover GMEA firm-level assistance and services (i.e., not all EDI activities).

The procedures for the pilot are:

1. Selection of customers. All firms must meet the following conditions to receive a customer evaluation survey:

• Profiled as a manufacturing or non-manufacturing business.
• Have received the following type of assistance and services:

Firms having received formal assessments and technical assistance projects are always sent surveys.

Firms having received informal engagements (with more than two hours staff time) and referred technical assistance projects (with more than one hour of staff diagnosis time).

• Be a "closed project."

Project reporting includes a section for designating whether or not a project is closed; this should be completed.

For activity reporting (i.e., less than one day of assistance) regional offices should carefully indicate activities that warrant a survey. Susan Vogel will work with the regional offices to review activities reports to identify such significant activities. Do not include administrative activities, outreach or marketing efforts, student assistance, non-technical request from companies (e.g. about Georgia Tech admissions procedures), short question-and-answer activities, ongoing client follow-up. In general, technical assistance of more than two-hours service will be subject to customer evaluation.

If multiple projects for the same company are closed, (1) more than one survey should be sent where key contacts are significantly different (e.g., in different divisions or departments), (2) only one survey should be sent to the same contact or team of contacts where the projects are closed within the same quarter or adjacent months.

Not excluded by staff. The Assocaiate Directors in conjunction with project managers may exclude a project in the region for evaluation if they believe it to be inappropriate, but should add a note to explain the decision. Lists should be reviewed and returned to Susan Vogel within two business days of receipt.

2. Description of Service. All projects and activities to be surveyed should indicate a concise one-sentence description of the actual service provided with date and month. This description will be reproduced on the customer survey form to help the client evaluate the service and for client confirmation.

3. Administration. Three modes of administering the survey will be tested in the pilot: (1) through the mail (including a postage-paid, pre-addressed return envelope), (2) over the telephone, (3) via fax-out and fax-return. Customers will be randomly allocated to each method. Approximately 30-45 days after closure, one of these methods will be administered. A second wave will follow after three weeks to reach non-respondents. Various combinations for non-respondent follow-up (i.e., first wave mailed, second wave over the telephone) will be tested as well.

4. Cover letters. For mail and fax administration, cover letters will be sent under the signature of each Associate Director in whose region the assistance and services were provided. Under telephone administration, the introductory phrases will include similar language and references to the appropriate Associate Director.

5. Data entry and analysis. Each completed survey will be entered into PROTRAC. Summary results will be reported to NIST. The evaluation team will conduct an indepth analysis of results, broken down by type of firm, assistance and service, and other relevant attributes. This analysis will be presented to GMEA staff.

6. Access. All completed forms will be filed in the Director's office. In addition, a copy of firm-level completed surveys will be returned promptly to the Associate Director in the appropriate region, who may then forward the survey to the relevant staff. GMEA staff will have access to results in PROTRAC. No one outside of GMEA is allowed access to firm-level results without first obtaining the firm's consent.

7. For Training and Networking, customer evaluation procedures are under development. Staff should consult with Associate Directors for guidance about evaluation in these situations.

Because this is a pilot, the process is open to procedural changes or modifications to the survey form. Comments from staff are welcome. All significant changes will be communicated to GMEA staff.

IV. Summary Chart: Information and Evaluation Guidelines

For more information about GMEA's evaluation plan, please contact Philip Shapira at 404/894-7735 (email:philip.shapira@pubpolicy.gatech.edu) or Jan Youtie at 404/894-6111 (email:jan.youtie@edi.gatech.edu).

modified 10/18/96