Philip Shapira, School of Public Policy, Georgia Institute of Technology, Atlanta, Georgia 30332-0345 USA, email: philip.shapira@pubpolicy.gatech.edu ; Jan Youtie, Economic Development Institute, Georgia Institute of Technology, Atlanta, Georgia 30332-0640 USA, email: jan.youtie@edi.gatech.edu

August 1994

ABSTRACT: This paper provides an overview of the GMEA evaluation plan and approach. Included in the chapter is a discussion of the industrial and institutional context, program goals and operations, evaluation stakeholders, the approach to evaluation, and methodologies to be employed.

I. INTRODUCTION

The Georgia Manufacturing Extension Alliance (GMEA) is a partnership of several organizations in Georgia established to provide a new, integrated model for delivering management and technical assistance to small and medium-size manufacturers in the state. Led by Georgia Tech's Economic Development Institute (EDI), the partnership also includes the University of Georgia's Small Business Development Center, the state Department of Technical and Adult Education's Quick Start, and Georgia Power Company's Technology Applications Center.

Building upon EDI's industrial extension service, which has a 30-year history of providing field-based technical assistance to Georgia's manufacturers, GMEA expands the capacity of current industry, technology, management, and training services at Georgia Tech and its partners and adds several new services. GMEA submitted a successful proposal to the Technology Reinvestment Project (TRP) in 1993. In February 1994, it signed a cooperative agreement with the U.S. Department of Commerce, National Institute of Standards and Technology (NIST), the agency assigned to manage this and other TRP technology deployment projects. Over a two-year period, federal funding of $6.6 million has been committed to GMEA, matched by an equivalent amount of state, in-kind, and other funds.

GMEA has an explicit evaluation element designed around these three main aims:

1. Provide consistent feedback about the effectiveness, targeting, and impacts of GMEA's services.

2. Support systematic learning about how services are being delivered and what services and approaches work best and why, so as to assist the ongoing improvement and management of program services

3.Furnish evaluative information to GMEA's major stakeholders and sponsors, including the state of Georgia and NIST.

This paper provides an overview of the GMEA evaluation plan and approach. Included in the chapter is a discussion of the industrial and institutional context, program goals and operations, evaluation stakeholders, the approach to evaluation, and methodologies to be employed. This paper has been written during the early start-up phase of GMEA, so it is likely that some of the components discussed here will be modified and refined during subsequent implementation.

II. INDUSTRIAL AND INSTITUTIONAL CONTEXT

There are about 9,300 manufacturing plants in Georgia, of which 97 percent employ fewer than 500 employees. Some 3,900 (42 percent) of these plants are located in the 18 counties which compose the Atlanta metropolitan area. In the Atlanta area, the major manufacturing industries (by employment) include transportation equipment, food products, electrical and electronic equipment, and machinery and computers. In the remainder of the state, the major industries include textiles, apparel, food products, lumber and wood, and transportation equipment. There are several significant industrial clusters outside of Atlanta, including Macon, Columbus, Augusta, Savannah, and Dalton.

Georgia Tech, through its industrial extension service and other university technology development and transfer programs, has for many years provided industrial services to the state's manufacturers. Through 1993, the industrial extension service operated 13 regional field offices, staffed by full-time personnel with engineering, management, and industrial experience. The industrial extension service was initially established by the state government to aid rural manufacturing. There were no field offices in the Atlanta region. Georgia Tech's industrial extension service, together with incubator, quality and management assistance programs, policy research, and several other programs which had been managed by the Georgia Tech Research Institute, were re-organized in 1993 into a newly formed Economic Development Institute. Quick Start is a program of the Georgia Department of Technical and Adult Education (DTAE) which provides job-specific training for workers. The program can provide training through the 32 DTAE technical institutes around the state, although it also uses private vendors and customer facilities.

Quick Start traditionally provided customized training for new or expanding firms, but in recent years has also begun to offer workforce training for existing manufacturers. Quick Start will contribute to GMEA by providing customized training assessments and by conducting training in such areas as ISO 9000, quality management, and environmental health and safety.

The University of Georgia Small Business Development Center (SBDC) is a statewide business assistance organization sponsored by the state government and the U.S. Small Business Administration. The SBDC provides assistance to businesses and communities through a network of 18 centers in Georgia staffed by about 50 professional counselors. The SBDC will work with GMEA in providing market research, market planning, and assistance with finance and capital for GMEA customers.

The Technology Applications Center (TAC) of Georgia Power (a public utility company) demonstrates how manufacturers can cut costs, improve quality and productivity, and address manufacturing problems with advanced manufacturing processes. TAC will contribute to GMEA by expanding its demonstration capabilities in advanced information systems for smaller firms.

III. GMEA's PROGRAM GOALS AND OPERATIONS

In its proposal and operating plan, GMEA has defined three principal operational goals:¹

• To expand the customer base. This will be done by opening new regional field offices to serve firms in the Atlanta metropolitan area and in the Dalton textile cluster of northwest Georgia.
• To add new and expanded management and technical services based on surveyed needs of firms and industries. Existing services in manufacturing technology, operations planning and control, and facility planning will be augmented by expanded and new services in management and marketing systems, quality management, information technologies, energy management, and environmental and occupational safety and health.
• To deliver technology through an integrated network. GMEA aims to establish an integrated delivery system combining the services and technology of Georgia Tech, Quick Start and the state's technical institutes, the SBDC, Georgia Power's TAC, and federal labs and NASA. In addition, a Technology Linkages Office will be established to connect Georgia firms with federal resources.

Although these are the formal operational goals of GMEA, the evaluation element will also give consideration to the following substantive goals described in other parts of GMEA's proposal and/or which form the intent of the Technology Reinvestment Project. These are:

• To improve awareness, information, and understanding among firms about enhanced technology and manufacturing techniques and technology assistance sources and services.
• To improve the competitiveness and performance of assisted firms. Within this, to assist defense-dependent firms compete in civilian markets.
• To stimulate the use of new technologies and techniques to improve products and manufacturing processes.

IV. APPROACH TO EVALUATION

In developing tools and procedures to measure GMEA's progress toward its operational and substantive, we have been influenced by a particular set of approaches which might be termed our "philosophy" of evaluation. It is both useful and important to make these approaches explicit, as follows:

1. The evaluation of GMEA should particularly focus on factors which (a) GMEA emphasizes as key elements of its mission and (b) which GMEA assistance can influence. Thus, if GMEA is providing quality management assistance, we should measure whether this results in improvements in the quality of products. Or, if a company is shown a new technology, a key measure is what action it took as a result of that (which might be to adopt the technology, pursue further research, or realize the technology was not appropriate and thereby not make a costly mistake--all of which could be considered as successful outcomes). On the other hand, while indicators of broader factors such as changes in sales or jobs also should be monitored, care needs to be taken in how changes in these broader measures are attributed to GMEA because many other factors influence them as well.

2. Measures of "manufacturing effectiveness" (i.e., what end results are achieved in improved manufacturing performance by customers as a result of GMEA assistance) are probably more important in affecting improved industrial competitiveness than measures of "program efficiency" (i.e., the private or public returns compared with program costs) per se.

3. Efforts to measure the improvement in performance of assisted firms must incorporate a control element, i.e., the appropriate question becomes: how has GMEA improved the performance of its customers compared with changes in the performance of similar non-assisted firms (overall, by industry, by size group, etc).

4. When firms take positive action as a result of GMEA services, we need to learn what has happened and why, so that we can replicate those services elsewhere. We need to know how and why certain services work well (or otherwise) and under what conditions. This will require thorough and consistent documentation of what GMEA did, who did it, and when for customer firms. We will also need to develop careful case studies (which go beyond the normal "PR" pieces) to fully understand how firms effectively use GMEA services.

5. We also need to develop directions for "system-level" changes, e.g., in state policies and support (industrial rationalization and modernization). We can pick these up in questionnaires, but probably more useful will be dialogues with firms, case studies, and analysis of all our information sources.

6. GMEA's evaluation system needs to provide constructive and timely feedback to personnel at all levels in the system (from managers to field staff) and be receptive in turn to their comments on our findings. GMEA's evaluation system also needs to find ways to give feedback to other key stakeholders, including federal and state sponsors, host institutions, and the customers. Many of these stakeholders share similar interests in receiving timely information, knowledge of what works or doesn't work and why, and tangible indications of performance. In other cases, needs and interests among stakeholders potentially conflict. For example, field staff and customers desire a minimized reporting burden, whereas program managers may seek richer and more detailed information. Table 1 seeks to categorize the various interests and needs of different stakeholders.

V. EVALUATIVE MODEL AND PROCEDURES

To develop evaluative procedures, it is helpful to have a model of exactly what the program does, its inputs and targets, its work processes, and its expected outcomes. We have developed such a model for GMEA, albeit a simplified one. This model² essentially depicts the core GMEA approach of providing one-on-one services to customer firms. In future work, we will seek to develop evaluative models in terms of GMEA's initiatives at the industry cluster and institutional levels.

The essence of the customer service model is as follows. GMEA has a series of resources available to it (program inputs), which include staff time, expertise, funding, office locations, information systems, and access to other technology sources. It seeks to apply these resources to the needs of customers through the form of services and other kinds of assistance (program intervention). At or soon after the point of service delivery, customers can form a view of the effectiveness of, and satisfaction with, service delivery (customer valuation). We can further inquire as to whether the customer will pursue any implementation steps as a result of the services provided, such as making changes in equipment and facilities or initiating a new training program (customer intermediate actions). Where customers take action (and they may not do so in every case), we may seek--after an appropriate period of time--to ascertain any effects on the firm, for instance changes in sales, quality, investment or technology levels (business outcomes). We can also explore any effects on jobs, taxes, and other broader factors (development outcomes).

As we develop information on these various elements, we also need to understand the type of customer assisted, because factors such as size, ownership, industry, defense dependency, or previous experience with GMEA may influence participation with and the ability to benefit from GMEA (customer profile). The resources committed by the customer are important (customer inputs). Also, we need to be able to make comparisons of customers and non-customers in terms of participation and outcomes and consider the broader influence of changes in business factors (non-customer controls and measures of industry and business conditions).

In our evaluation protocol, we have established a series of tools and procedures to provide information and insights about GMEA's operations and impacts. These tools and procedures are consistent with the evaluative model just described. They also seek to provide information which can meet the needs of various stakeholders as discussed earlier (we are not sure we can meet all those needs, but we think we have met the most important ones). The procedures to be employed include:

• Customer Profile. The customer profile is a consistent approach to assembling basic industry information about a customer. This profile is administered (as unobtrusively as possible) by program personnel at the point of initial contact with a customer. The customer profile records information on company logistical items (name, address, phone, etc.), contact name, type of customer, type of operations and location of head office, defense dependency, annual sales/shipments, employment, products, and source of referral.³
• Activity Reporting. The aim of activity reporting is to track field agent activities and customer interactions. Items recorded include logistical information, activity or service provided, and staff time committed.
• Client Valuation. Client valuation surveys will be administered to each customer upon completion of all GMEA engagements, including formal assessments, technical assistance projects, and referred technical assistance projects. The customer valuation procedure checks logistical and service information, obtains customer satisfactions with the quality and delivery of services, gets customer inputs, asks whether the customer will take any action (if yes, what kind; if no, why not), and prompts the customer to provide initial expectations of impact (e.g., sales, employment, use of new technology). Mail will be the primary method of administration, supported by telephone follow-up.⁴ The customer valuation procedure meets NIST recommendations for this procedure, although it also explores important questions other than those NIST recommends.
• Customer Progress, Longitudinal Benchmarking, and Non-Customer Controls. NIST asks its MTC programs (such as GMEA) to maintain a client progress tracking system. An annual follow-up survey is recommended to track changes of customers in such areas as change in scrap rates, employees using computers or programmable machine controllers, inventory turns, sales per employee, manufacturing lead time, sales, and employment. We have adapted this procedure to provide longitudinal measures of technology usage in Georgia, non-customer controls, and customer progress reporting. The evaluation team will conduct two surveys of technology usage and usage of assistance services, including GMEA. The first survey, underway in summer 1994, will provide baseline benchmarks and a comprehensive overview of manufacturing in the state (all plants with 10 or more employees are being surveyed). The survey includes NIST recommended questions, plus several other items. A second statewide survey will be conducted in two years. In the second survey, some firms will have been GMEA customers, others will not, thereby providing a control. Other kinds of controls, for GMEA-assisted firms, will be provided by complementary information on the types of services provided to customers. The second survey will meet NIST's requirements for customer progress reporting. For the intervening year, we will conduct a customer progress report just for those firms assisted by GMEA.
• Case Studies. The evaluation team will conduct a series of case studies to thoroughly examine the linkages between GMEA services and impacts on firm operations and profitability. These case studies will help to understand how GMEA's services are received by firms and what factors influence how customers respond to these services.
• Institutional Assessment. The evaluation team will coordinate an institutional assessment, similar to an MTC third-year review panel, to provide feedback regarding program operations and impacts. Results will be delivered February 1996.

These tools and procedures are being implemented, but not all simultaneously. GMEA is based upon an existing and long-established industrial extension program with its own practices and traditions. The introduction of a more formalized evaluation system must be conducted with sensitivity to this and with appropriate discussion and review. GMEA brings together several programs within Georgia Tech, each with varying practices and existing evaluative systems. The GMEA partnership also involves external organizations with their own practices and evaluation systems. As a practical issue, we have chosen as our first priorities to (1) develop and implement the core customer profile, activity reporting, and customer evaluation tools, and (2) to initiate the statewide manufacturing benchmark survey, because it is important to implement this as close as possible to GMEA's start-up. Subsequently, we will implement the other evaluative procedures.

The actual task of tracking information is, in several cases, assigned to GMEA's Management Information Systems (MIS), with the oversight of the evaluation team. An on-line system (ProTrac) is being implemented to electronically track services, customers, projects, and customer valuations. In the transition to and in support of full ProTrac operations, manual recording systems and forms have been developed. Table 2 provides additional details on who collects what information and when. Table 3 considers the various program and organizational elements of GMEA. These elements include customer assessment and implementation assistance, customer assistance by associates, customer referrals, training programs, seminars and events, group networking activities, and demonstrations. The table summarizes the tools and methods used to collect information on these elements.

VI. EVALUATION DELIVERABLES

A series of interim and final deliverables will be produced as a result of the evaluation process. These include:

• Development of procedures for evaluation (evaluation plan, summer 1994)
• Design of customer profile and valuation procedures and tools (summer 1994)
• Design of additional evaluative tools and instruments (ongoing)
• Design and implementation of benchmarking surveys and customer progress reports
• Guidance on MIS systems, quarterly reporting, program operations (ongoing)
• Report on Manufacturing Technology Use and Needs in Georgia (November 1994)
• GMEA: Evaluation Progress Reports (December 1994 and December 1995))
• GMEA: Case Studies in Implementation (1995) GMEA: Institutional Review (end of 1995)
• GMEA: Program Evaluation (interim report, mid-1995; final report, 1996)

ENDNOTES

¹A Proposal to Create the Georgia Manufacturing Technology Extension Center, Atlanta, GA: Georgia Institute of Technology, Economic Development Institute, 1993 (TRP Proposal); Georgia Manufacturing Extension Alliance, First-Year Operating Plan, Atlanta, GA: Georgia Institute of Technology, Economic Institute, May 1994 (submitted to the U.S. Department of Commerce, National Institute of Standards of Technology). (back)

²In developing this model, we acknowledge and have drawn upon the work of Eric Oldsman of Nexus Associates in the evaluative materials he has developed for the New York Industrial Extension Service and for NIST. (back)

³For further details, see: Philip Shapira and Jan Youtie, Georgia Manufacturing Extension Alliance: Procedures for Pilot of Firm-Level Customer Profile and Customer Evaluation of Services, Evaluation Working Paper E9402, Atlanta, GA: Georgia Institute of Technology, Economic Development Institute, August 1994. (back)

⁴See Shapira and Youtie, ibid, for further information. (back)

For more information about GMEA's evaluation plan, please contact Philip Shapira at 404/894-7735 (email:philip.shapira@pubpolicy.gatech.edu) or Jan Youtie at 404/894-6111 (email:jan.youtie@edi.gatech.edu).

modified 10/21/96