Philip Shapira(*a), Jan Youtie(b) and J. David Roessner(a)
The expansion of public policies and programs to promote the technological modernization of small and mid-sized manufacturing enterprises in the United States has been accompanied by an increased interest in assessing the effectiveness and impact of these initiatives. This article examines current practices used in the evaluation of U.S. industrial modernization programs at state and national levels, drawing on interviews with program managers, site visits, and scrutiny of available studies. Issues related to the meaning of evaluation in the context of industrial modernization, the scale and scope of existing programs, and the definition of metrics are considered. A series of evaluation approaches, methods, and studies are identified and reviewed, including the role of program monitoring, customer valuation, external reviews, economic impact studies, control groups, and assessments of best practice. The authors then address the use of evaluation results and discuss key challenges and directions relevant to the development of more robust evaluation procedures.
Current practices in the evaluation of U.S. industrial modernization programs
Public policies to promote industrial modernization , the application of improved technologies and business practices to strengthen competitiveness and productivity, particularly among small and mid-sized manufacturing enterprises , have expanded significantly in the United States in recent years [Rosenfeld 1992; Simons 1993; Shapira, Roessner and Barke 1995]. These policies have resulted in an increased level of federal and state resources being allocated to manufacturing technology centers, industrial extension programs, industrial networking projects, and other initiatives. Industrial modernization programs typically focus on the deployment of known technologies and proven business practices, training methods, and management approaches, rather than the creation of new technology. Such programs employ a range of methods to aid firms, including information provision, assessments, demonstrations, brokering, field agent services, qualified referrals, group projects, and training. Improvements are often also sought in inter-firm and industry relationships and in public technology, training, and business assistance infrastructures. Non-profit organizations, colleges and universities, state agencies, industrial associations, and private consultants are most frequently engaged in providing industrial modernization services.
At the federal level, the U.S. Department of Commerce's National Institute of Standards and Technology (NIST) has emerged as the major sponsor of new American industrial modernization programs. By mid-1995, NIST had awarded federal funds (matched by state, local and industry contributions) to more than 45 manufacturing technology or outreach centers (MTCs). NIST coordinates these centers through its Manufacturing Extension Partnership (MEP), with the aim of combining government, industry, and academic resources to foster technological modernization [National Institute of Standards and Technology 1994b]. Numerous additional industry assistance projects are also underway, sponsored by state and local governments, industrial groups, and other federal agencies. Most modernization programs serve firms or industrial sectors in defined geographical areas (state or sub-state regions), although some offer services nationally or target specific industries without regard to boundaries.
The growth in efforts to promote manufacturing modernization has stimulated greater attention to issues of performance measurement and program evaluation. Funders are concerned to know whether their dollars are being deployed effectively [U.S. General Accounting Office 1991; Nexus Associates 1994; Mendelowitz 1995]. Policymakers are asking whether these programs are generating the promised improvements in competitiveness and whether there are worthwhile effects on jobs, wages, technology, and business and economic development [National Research Council 1993; Martin 1995]. Industry groups and companies seek evidence about how the programs might assist their business and manufacturing operations [Farrell 1994]. Program managers and staff are keen to learn about the impacts of different types of services and how they can become better performing organizations, just as they are encouraging their manufacturing customers to improve their own business practices [Heller 1995; Shapira and Youtie 1994a].
The increased interest in evaluation has also been influenced by developments at the national level. Federal funds for manufacturing modernization have been enlarged as part of the Clinton administration's technology policy commitment to establish a network of more than 100 manufacturing technology centers by 1997 [Clinton and Gore 1993]. In 1994 and 1995, much of the funding for these centers has been provided by the federal Technology Reinvestment Program (TRP) , an initiative whose aims include transitioning defense suppliers to civilian markets and strengthening the nation's industrial base [Advanced Research Projects Agency 1993]. NIST has used TRP allocations to establish new centers under the MEP program, with the intent of providing a subsequent round of civilian-side funding from the Department of Commerce's account to those centers with demonstrated effectiveness. The increase in the scale of the MEP program, coupled with the importance of defining clear standards of center performance, has obliged NIST to consider and revamp the evaluation of the individual and collective accomplishments of its manufacturing modernization centers and programs. The interest in evaluation has become even keener as changes in the composition of the U.S. Congress after the Fall 1994 election generated new questions about the effectiveness and justification of federal support for manufacturing modernization [National Coalition for Advanced Manufacturing 1994; Andrews 1995].
Yet, while concern about the evaluation of industrial modernization programs has grown, it has also become clear that developing robust evaluation approaches , as in most other fields of public policy , is not an easy task. Perspectives vary on what should be measured and how. NIST has required its MEP centers to establish program monitoring systems and report information about services provided, clients served, program linkages and referrals, staffing, and revenue and expenditures [National Institute of Standards and Technology 1995a]. While this provides a basic information base, there seems to be a consensus that evaluation needs to go beyond simple counts of the number of firms served or types of services delivered. However, there is much debate about how best to gauge program impacts on firms, technology adoption, employees, and returns to local and national economies [Shapira and Youtie 1995]. A wide range of actual or potential measures of program effect have been advocated, including profitability, value-added, sales, exports, wages, jobs, taxes, machine use, quality, training, investment, defense conversion, business stability, and customer-supplier links. Issues have arisen about the role of comparison groups of non-customers and about the dangers of imposing too many information burdens on both firms and programs. Finally, tensions are apparent between the objective of program justification, where evaluation is harnessed to present funders with broad evidence of the success of existing approaches, and that of program improvement, where methodologies are focused towards a detailed understanding of how services work (or otherwise) and how more effective approaches might be devised.
This article examines the current status of evaluation in the field of industrial modernization in the United States, exploring practices, debates and challenges, and emerging trends. First, we consider broad issues such as determining what programs and services are within the scope of industrial modernization, the context and meaning of evaluation, and the measures which can be used to gauge program performance and impact. We then examine a series of methods used in evaluating industrial modernization services, the dissemination of evaluative information to customers and other constituencies, and program perspectives on their own strengths and weaknesses in evaluation. Particular attention is given to the growing federal role in promoting evaluation of industrial modernization. The article draws on a series of interviews conducted with program managers and evaluation specialists. We also identified and reviewed a variety of program evaluation tools, such as internal assessment forms and customer surveys, as well as available program evaluation reports, published articles, and related materials.
Industrial modernization is an area of activity comprised of multiple stakeholders and varied programs seeking to influence business performance and economic development in an environment where these outcomes are also affected by a wide array of other economic, social, technological, and policy factors. The task of evaluation is thus not an easy one and there are a number of broad issues which need to be initially considered in assessing the context for evaluation in this field and the extent to which current approaches to evaluation and measurement are valid and effective. We begin by considering the meaning of evaluation within the industrial modernization field and then, to further define the focus of inquiry, the scale and scope of industrial modernization programs. This is followed by a discussion of the metrics which have been put forward to assess the progress and impact of modernization interventions.
2.1 Meaning of Evaluation
In considering how industrial modernization programs might be evaluated, one of the classical designs is to relate program operations and outcomes to goals and objectives appearing in legislative language or program mission statements. However, this can only be a starting point for evaluation, since often the goals given to industrial modernization programs are broad and hard to measure against progress without more systematic definition. For example, typical goals of the state programs we studied are to improve manufacturing competitiveness and promote industrial or economic development. Federal programs, including those of NIST, focus on improved competitiveness and technology transfer [for example, see the Omnibus Trade and Competitiveness Act 1988]. But, how programs actually interpret these goals, and translate them into program objectives and services, varies quite considerably. There is also variation in the range and delivery of services in support of these objectives, reflecting in part different institutional sponsors and regional industrial circumstances. In some instances, certain legislatively-mandated program goals have effectively become obsolete or non-workable, while programs continuously modify the services they offer in support of their goals and objectives.
The differences among service offerings present a challenge to evaluating these programs. Some services are more likely to have quantifiable effects than others. Some produce short-term gains whereas others may have deep, lasting effects on the level of manufacturing in a region or industry. For instance, information and training services may have a less tangible effect on the firm's bottom-line than equipment purchases. Equally, helping a company implement total quality management or promoting improvements in inter-firm cooperation and customer-supplier linkages should have deeper and longer-term effects than solving specific machine problems. Another variation is that the nature of services offered in part depends on the diagnosis the industrial extension professional provides to the manufacturer and the particular resources available to respond to this diagnosis. Similar problems may be diagnosed and addressed differently by different programs. Moreover, as one change is implemented, other needs may become apparent, so one outcome of service delivery may be the development of more problems (rather than total solutions). This raises the question of the time-frame over which evaluation should take place. Elected officials frequently seeks "results" not only over two-to-four year political cycles but sometimes also over annual budget periods. On the other hand, the processes of information dissemination, assessment, deployment, and subsequent improvement may often require a technology cycle of many more years before the full consequences materialize.
To address these and other complexities, we take a broad view of the meaning of "evaluation," identifying it as a process where judgement is applied about program impacts and effects using varied information sources and methods and occurring at numerous points in the service-cycle and life-cycle of industrial modernization programs. The evaluation points may be formal or informal and can occur with considerable variations in the quantity and quality of information used. Most frequently, evaluation is undertaken by programs to provide management control over resource allocations and service delivery; assess customer satisfaction and valuation; determine outcomes of program intervention for individual firms; identify impacts on industries; and identify impacts on regional and national economies. Program sponsors, review and oversight agencies, and evaluation specialists may also conduct evaluations for purposes of proposal reviews, funding, management guidance, or overall program effectiveness.
2.2 Scale and Scope
Industrial modernization is not an entirely new public endeavor in the United States: several state industrial extension and technology assistance programs, including those of Georgia, North Carolina, and Pennsylvania, have origins in the 1950s and 1960s [Shapira 1990; Combes 1993]. However, as concerns about industrial competitiveness, technology diffusion, and regional economic restructuring have grown in the United States, there has been a marked increase in recent years in the public and private resources targeted to industrial modernization [Shapira, Roessner, and Barke 1995]. New industrial modernization programs have been established in more than 30 states since 1988 by a variety of state and local organizations [Berglund and Coburn 1995; National Institute of Standards and Technology 1995c]. In addition to the Department or Commerce, multiple federal agencies, including the Department of Energy, the Economic Development Administration, the National Aeronautics and Space Administration, the National Science Foundation, the Small Business Administration, and the Department of Defense, have sponsored industrial modernization and technology deployment programs. Inevitably, there has been a degree of "mission creep" as existing entities established for other purposes have sought to access available funds by professing to offer expertise and services in the field of industrial modernization.
These trends raise the question of what should be included within the rubric of an industrial modernization program. It is an issue made more complex by the diverse services offered by these programs and the tendency, in trying to aid firms holistically, of linking technological assistance with guidance on related problems of enterprise organization, training, management, marketing and finance. Surveys undertaken in the early 1990s (and prior to the latest round of expansion in the system) found several hundred entities who asserted that they were engaged in at least some aspects of industrial modernization and technological upgrading [National Institute of Standards and Technology 1990]. The number of included programs is significantly smaller if more focused criteria are used, including whether industrial modernization is the primary aim of the program (as opposed to general small business assistance), the emphasis on technology deployment (as opposed to research or advanced technology development) and on existing small and mid-sized manufacturers (rather than new start-up firms or large enterprises), the presence of core expertise related to manufacturing technology within the program (in addition to experience in management and training), and the ability to offer in-plant field services, including diagnostics and assessments. Using such criteria, one recent study identified just over 80 separate state industrial extension and technology deployment programs in the U.S. in 1994 [Berglund and Coburn 1995]. This includes the NIST-sponsored MEP centers as of that year, although by the end of 1995 the MEP will have expanded its network to include nearly 50 centers (some of which will incorporate existing programs, while others will be entirely new).
The scale and types of services provided to firms by this group of programs is difficult to determine, although there are partial data. It is estimated that agents at 28 NIST sponsored centers operating in at least part of 1994 have visited or provided services to more than 30,000 of the 381,000 manufacturing firms in the United States. Additionally, these centers have provided training to managers and employees of over 5,000 industrial companies [National Institute of Standards and Technology 1995b]. However, the overall level of modernization services to firms is now certainly rather greater than this considering the further increase in NIST centers in 1995 as well as the activities of programs funded by state or other federal resources. Aggregate spending on industrial modernization services, including funds from federal, state, and other sources, has grown from under $80 million in 1991 [Clark and Dobson 1991] to over $200 million by 1995 [Shapira 1995].
The firms served by industrial modernization services are predominantly of small or medium size. An analysis of data from reporting MEP centers indicates that almost 70 percent of firms served have 99 or fewer employees, while a further 25 percent of assisted firms employed between 100 and 499 workers. The primary services provided by these programs included assistance with business systems and management (23 percent of all activities reported), quality (14 percent of activities), market development (12 percent), manufacturing process (11 percent), and human resources (9 percent). A secondary set of services, each representing between 5 to 7 percent of all activities, included assistance with product design, environmental management, plant layout, and computer-aided design and manufacturing. Assistance with automation or robotics is among the lowest ranked categories (just over one percent of activities), reflecting the fact that industrial modernization services are mostly aiding firms to upgrade their existing business practices, methods, and techniques rather than promoting the deployment of leading-edge technologies (which can often be too complex or untried for most small companies). Most services to firms are provided through one-on-one assistance from agents, brokers, or consultants who conduct assessments, diagnose problems, refer to qualified service providers, and assist with implementation. Generally, initial or diagnostic services are provided without charge to the client, although many programs charge fees or require a firm to cost share for follow-on assistance. There is also a growing trend to provide services to firms in groups, helping to resolve common problems and promote greater business and technological collaboration. A review of 1994 data for 27 states finds that some 140 industrial groups have been organized involving more than 2,600 firms.
Reviewing this information, it is evident that multiple organizations are now involved in offering a portfolio of industrial modernization services under varying conditions, as indeed might be expected given the diverse industrial and regional circumstances and decentralized set of institutional and decision-making arrangements found in the United States. The level of resources invested in industrial modernization programs has expanded markedly over the last decade and the scale and scope of services on offer has increased. Within this diverse field although perhaps not without some definitional challenges at the margin, it is possible to delineate a group of programs primarily dedicated to industrial modernization. These programs, which include the affiliates of NIST's MEP program as well as similar programs sponsored by other organizations, are the principal subjects of our concern about how the performance and impacts of industrial modernization interventions are currently measured.
2.3 Defining Metrics
Most efforts to evaluate industrial modernization programs draw on a sequential set of concepts about how industrial modernization operates. In this framework, program inputs (including technical staff time, equipment and facilities, and financial resources) are applied to client or industry problems and opportunities through program interventions. Through a series of intermediate steps, actions are then taken by the client which then lead to business and economic development outcomes. This framework, which was most coherently articulated in a study of the New York Industrial Technology Service [Nexus Associates 1994], is a variation of the systems analysis approach found throughout the program planning and evaluation field [Sylvia et. al.1985].
Along the stages of this framework, industrial modernization programs , whether explicitly (in a few instances) or implicitly (in most cases) , have defined a series of metrics to measure performance and impacts. Our interviews with program managers delineated a broad category of program measures which focused on characterizing the inputs to the intervention and the services rendered, including both public and private contributions. Also identified were intermediate measurement categories for client satisfaction and action, firm-level measures targeting the business and technological outcomes for assisted firms and industries, and developmental measures emphasizing economic development effects, including impacts on jobs. However, within these four categories, the particular measures used by individual programs diverged and there were differences in the amount of attention paid to specific categories. Among the programs studied, the types of program input measures used included the total number of activities conducted, the public cost, the number of companies served, the number of new companies assisted, repeat requests for assistance, the amount of time expended, the type of services delivered, the number of fee-based services, the level of revenue generated, and the extent of other private inputs such as employee time, cost shares, and consultant costs. Also measured, as indicators of service delivery, were the proximity of field offices to population centers and clusters of firms or, in other cases, the dispersion of field offices across the whole service area, the number of partnership arrangements with other service providers, the extent of client backlog, and the qualifications of staff. Most programs tracked a subset of these measures. Intermediate outcome and action measures included client satisfaction (which many but not all programs sought to assess in a variety of ways), whether clients took action, to what extent any action was attributable to program intervention, and what particular actions were taken (these latter items were tracked by relatively few programs). The firm-level metrics in use included cost-savings, increased business revenues, added investment in plant and equipment, improvements in market share, impacts on profitability, improved access to information, changes in management strategy and organization, employee training and skill development, changes in technology adoption rates (e.g., for computer-controlled machinery), and impacts on competitiveness. The developmental measures in use included jobs created or retained, impacts on taxes, and added dollars flowing into the state. Generally, most programs tried to measure at least a few of the metrics in these last two categories.
The difficulties of interpretation and estimation associated with these categories and measures are apparent. For example, while program interventions may seek to promote the use of improved technologies in client firms, other factors such as the availability of bank financing or the role of customers and vendors may influence an enterprise's decision calculus. The attribution of program interventions on subsequent "outputs" is thus complex, especially if counterfactual questions are posed about whether firms might have taken the action anyway or used other available public or private assistance sources. Similarly, it is at least equally complicated to assess the linkage to "outcomes" , the impacts of program-induced changes within firms on overall business performance and economic development , particularly when the effects of broader economic and market conditions and other public policies are considered. Definitional variations add a further twist, even among those metrics which are most amenable to quantification. In the program measures category, many programs try to capture how much time is expended with a client and what is the public cost of service. Yet, these public inputs are measured by programs in often quite different ways, for example in the way indirect or overhead costs are included or how staff time is counted. What counts as a delivered service also varies, with some programs employing more inclusive definitions than others. Moreover, besides capturing fee revenue, few programs seek to fully measure comparable private inputs such as the corresponding time commitment of the client's personnel. An especially intractable measure is that of competitiveness, which program sponsors frequently desire as a positive outcome but often do not clearly define. There is diversity as to whether program impacts on the competitiveness of clients should be measured by improved profitability, market share, value-added, business survival, use of new technology, or the ability to increase wages.
However, while many measures , especially those related to outcomes , are hard to quantify, this does not mean that programs avoid doing so, although program managers readily admit that care should be taken in the weight placed on these numbers. A case in point is the measure of jobs created and retained as a result of industrial modernization services delivered. Program managers acknowledge that employment effects are important criteria, but generally recognize that quantitative measures of job impact are usually hard for companies to estimate with much accuracy. Questions also arise about the extent to which new jobs are created versus being shifted from another company or region. Most fundamentally, the success of industrial modernization programs in aiding firms to deploy a new technology might result (at least in the first instance) in fewer jobs rather than more. Yet, despite these caveats, almost all programs report (often prominently) estimates of their effects on employment.
At the same time, the complexities of obtaining accurate quantitative data about program impacts lead industrial modernization programs to also seek qualitative information. The qualitative feedback materials collected typically include brief post-service client statements and anecdotal success stories; in other instances, more complete narrative case studies are prepared. Again, the balance between quantitative and qualitative measures differs among programs; some administrators , and their program sponsors , rely heavily on client letters stating the service is of value, while elsewhere such testimonials serve only as a supplement.
Beginning in 1991, NIST sought to develop with its seven original Manufacturing Technology Centers a common set of measures for gathering and reporting evaluation information [Luria 1993; National Institute of Standards and Technology 1993]. These measures took into account the services offered by the MTCs and tried to balance information collection with the burden of data collection. Three broad groups of metrics were identified. First, center performance measures, to be reported quarterly. These included the percent of MTC employees working with clients in the field; the number of initial site visits, informal engagements, and formal assessments initiated; the number of technical assistance projects initiated; non-governmental project fees, training fees, and membership revenues; the number of plants served by industry and size, the employment in plants served, and attendees at training sessions; and the MTC's proposal hit rate (proposed versus initiated projects) and repeat business rate (companies served more than once). The second group comprised a set of client valuation measures, to be collected after project completion. These included client satisfaction and the anticipated impact on sales, capital investment and avoidance, inventory levels, costs, and jobs created and saved. A third set of metrics sought to determine client progress, measured one year after the project engagement and requesting comparable data for earlier years so that changes could be identified. These client progress measures included changes in total sales, export sales, employment, employee income (payroll per employee), productivity (sales per employee), timeliness (manufacturing lead time), material scrap rate (value of scrap per sales $), computer use (employees with weekly use), and inventory turns (sales $ per inventory $) [National Institute of Standards and Technology 1994a].
In establishing these measures, NIST and its MTC partners aimed to define a consistent approach to assessing program inputs and determining the intermediate impacts on manufacturing performance and subsequent business and employment outcomes. There was similarity with the metrics then (and still) in use by non-MTC programs, although more attention was paid to technological and manufacturing impacts. NIST recognized that many of the metrics had a subjective element and were hard to quantify and that other "softer" measures, such as impact on business behavior, were excluded. However, although NIST tried to standardize the way in which individual MTCs tracked these measures, it found that there was still considerable variation in the completeness of reporting, the activities and firms which were subject to measurement, the specific questions and methods used, and the response rates achieved. These issues, which raised questions about the reliability and generalizability of the data collected, coupled with the massive expansion of the MEP system beginning in 1994, led NIST along with its MEP partners to review its information reporting requirements. In 1995, new guidelines were put forward containing more thorough definitions of program activities and service types, standardizing the reporting of financial data, and modifying the categories, methods, and frequencies of measurement [National Institute of Standards and Technology 1995a]. Increased attention was placed on tracking program and company costs and company benefits, while the emphasis on measuring intermediate manufacturing and technological impacts was reduced. Arguably, this is attributable to changes in the federal policy environment which have prompted NIST to be more conscious about the economic (rather than the technological) justification of the MEP program. For customer valuation measures immediately after a project has been completed, NIST has proposed using a national survey house to conduct consistent customer interviews by telephone. While many MEP centers have welcomed this (in part, because it relieves them of the cost and responsibility for follow-up), others are reluctant to participate due to concerns about client confidentiality and loss of timeliness and control. NIST's requirement for annual client progress reports has been dropped, with the agency seeking other research approaches to measure longer-term impacts.
NIST's still evolving efforts to standardize evaluation metrics for the MEP programs have brought a greater , although, as yet, by no means complete , degree of consistency to the industrial modernization field. However, even for MEP programs, other sets of measurement ledgers still have to be kept. State sponsors have their own, quite varying, requirements as to what should be counted, ranging from the assemblage of client satisfaction letters and statements of job impacts to comprehensive studies of program impact. Equally, programs maintain their own internal set of metrics as to how they determine their performance, which may include project cycle times, the ability to provide a service to any company who requests help, or the maximization of revenues ("if customers are willing to pay, they must find our services of value"). Inevitably, these issues of what criteria are used in measurement are entangled with how such measures are made, as the following sections illustrate.
Industrial modernization programs evaluate their performance and the impacts of their services at several different and complementary levels. Almost all programs maintain some method of program activity monitoring, used to track customer contacts and services delivered; many programs also try to determine how industrial customers valued and acted upon services delivered; many have also been subject to external review; and a few have attempted to measure economic impacts or assess effects on serviced firms in comparison with non-serviced firms. A variety of evaluation methods, from self-assessment, customer surveys, user and focus groups, expert review, industry surveys, and quasi-experimental designs, are employed. Mostly, programs tend to adopt relatively simple evaluative methods, with little use of systematic controls. Additionally, a wide array of evaluation metrics are employed. Our review of current practices amply demonstrates that the issue is not so much that programs do not conduct any evaluation at all, since most do something, but that there is a great deal of variation in evaluation commitment, scope, purpose, methods and metrics among individual programs. The increased role of the federal government has enhanced the resources available for evaluation and has promoted a relatively higher level of consistency and standardization in the monitoring of activities for those programs in the NIST MEP system. However, as already alluded to, there still remains much diversity in the approach to evaluation, both within these MEP centers and among non-MEP programs. The primary methods and means by which industrial modernization services are currently evaluated are discussed in detail below.
3.1 Program Activity Monitoring
Most programs have monitoring systems in place which keep track of the initial and subsequent contacts between firms and the program. The information collected by programs as part of their monitoring systems typically includes basic data on the characteristics of firms served by the program (such as firm name and address, contact person, industry, product, sales, and employment), records of interactions between program personnel and firms, problem diagnoses and project definitions, whether a referral was made to an outside source of assistance, and what actions were taken as a result of program assistance. Monitoring is generally done by program personnel, rather than firms, and is usually unobtrusive to the firm being served.
Program monitoring serves at least three evaluative functions for industrial extension and modernization programs. First, it is the basis for activity reporting. Programs usually, although not always, have to produce activity reports on a regular basis. Program managers produce such reports for funding bodies, organizational sponsors, and oversight agencies. NIST, for example, requires its MEP programs to submit detailed quarterly reports on activities, program contacts, and projects [see, Chicago Manufacturing Center 1995; Southwestern Pennsylvania Industrial Resource Center 1995]. In turn, program managers often, although again not always, require their field personnel to keep activity reports. In some cases, a regular (quarterly or annual) activity report is the only program performance information (in addition to cost accounting) that particular funding sponsors require. A second role for program monitoring is to aid internal management and resource allocation. Programs maintain or use monitoring systems to provide information for program management, such as the amount of time field staff spend with individual firms, and to assess or allocate case loads [Nexus Associates 1995]. In a few cases, programs use monitoring systems to track staff time for client billing. There is often a rich source of information here that can be used to guide program decision-making. For example, in one case where individual program reports were analyzed (for the first time), field agents tended to identify and service customer problems only in terms of their individual expertise ("hammers looking for nails"), suggesting that the program was not performing as comprehensively as it should in areas of diagnosis and service delivery. However, often this information is collected but not analyzed or presented in a way that management conclusions can easily be drawn. A third function for program monitoring is to facilitate the documentation of "success stories." Some programs regularly require field and project personnel to prepare narrative studies of program assistance. Invariably, the cases chosen are ones which are favorable and where program assistance has led to positive company results. Such cases are often presented to elected officials and the general public as examples of what the program is achieving [Carr 1994; National Institute of Standards and Technology 1995d].
In a typical example of the use of activity monitoring, one program reported that it was required to provide information on a regular basis to its program sponsor on these major items:
This program, as with most other programs, distinguished counts of ongoing work with firms who were existing customers from those of firms new to the program in a given time period. In addition, the program tracked financial information, such as the amount of money the program spent with a firm, how much money the company spent, and how much time was expended by program personnel working with the company.
Another program indicated that, to provide information for better management control, it had instituted activity accounting on a more frequent basis. Weekly reports had been initiated because monthly reporting did not give administrators information in a timely fashion about field activities. Commented the program manager: "We had problems with oversight of engineers in the field. They went out and lost track of what they should be doing , they were developing business plans, for example. They got away from providing engineering help. Or agents would put too many man hours into one company." A system of weekly reports containing a quick summary of what agents had done the previous week allowed for improved guidance of field staff resources. In a few cases, programs collate activity reports over time and then use this information in program management. After entering data from activity reports (and also client evaluations) into a computerized information system, one program manager said that they could breakdown results by type of customer, type of assistance provided, and other parameters. It was believed that this allowed them to see if one type of assistance (e.g., referrals) was more or less effective than another (e.g., direct technical assists) and then allocate resources accordingly.
Program managers emphasized the importance of combining both quantitative and qualitative information in monitoring systems. In one example, quantitative information was used to determine whether the mix of field projects and allocated staff time conformed to program goals. Qualitative information was used to determine the depth, breadth, difficulty, and expertise involved in projects. Both types of information were needed, the program administrator commented, because "just counting number of projects would be counterproductive for describing the work of one employee who mostly works on difficult, long-term projects whereas it would throw a favorable light on another employee who just does short, easy technical assists." An instance of the use of monitoring systems to document case studies was found in another program which, on a quarterly basis, required field personnel to write up a mini-case study of a particular extension activity that the agent had performed where the company was proceeding with implementation. Sometimes these quarterly cases reported program activity from a year ago, because of the time lags in implementation (for instance, where the firm had to secure financing).
Programs generally used their monitoring systems to collect information about their customer companies and to track program-company interactions. Relatively little emphasis seemed to be given to also using monitoring information to measure service performance and quality control, for example, by systematically tallying service response times to customer requests. Methods of information recording also varied between programs. Some programs used paper files or simple computer database or spreadsheet programs to maintain records, while others have developed or are developing customized and comprehensive management information systems. Detailed records may also be kept in field offices, rather than in the program head office, which may lead to considerable variability in information collection even within the same program. For example, one program manager emphasized the decentralized nature of his system: "Each office in the field operates with a lot of freedom. Field offices keep voluminous detailed project files. They send me a capsule of what they're doing , an activity report. The report includes the name of the company, projects, other pertinent information. If I need more information, I call them, but I keep my files slim." Where computerized systems had been implemented, there was no standard approach. Some programs maintained computer files directly accessible only in the central office, while others have developed networked multi-user and remote access computerized information systems to link field staff with shared program databases. In a few instances, programs have developed computerized information systems which tie together service providers in different organizations within a region.
3.2 Customer Valuation
Customer valuation of industrial modernization involves asking firms after they have received services to feed back information about satisfaction with the program, actions taken, and resulting benefits. Industrial extension and modernization programs frequently seek such information by asking their customers to complete a mailed questionnaire, respond to a telephone interview, or provide comments in a face-to-face interview [National Association of Management and Technical Assistance Centers 1989; Nexus Associates 1995]. Post-service customer valuations have the advantages of simplicity, directness, and low cost. However, there are doubts about the accuracy and value of this approach , even among programs who use it. The greatest concerns are about efforts to measure the employment and monetary impacts of services provided through client self-reporting, although questions are also raised about the reliability of measures of customer satisfaction.
The range of topics in customer valuation surveys include: confirmation of assistance received; satisfaction with program services; performance of program staff; how information or services were used by the company; project expense to the company; estimate of dollar value of services provided; resulting company investments in equipment, plant or workforce development; jobs added or saved; effects on sales, cost-control, financial management, inventory management, and access to outside capital; effects on productivity, product development, and engineering; and whether the company would use the program's services again. Open-ended requests for any further comments are usually added. Each program which uses this method adopts its own particular variation on the above questions. Not every question is asked, while most add a few other questions too. Programs also diverge in the way they administer these post-service surveys. For example, there is agreement that the timing of the questionnaire is critical: if too early, impacts will not be felt, while if too late, it is difficult to determine causal factors and secure a successful response. Yet, programs differ quite widely as to the timing of the client self-report, from almost immediately afterwards to nine months or more.
Comments from different program managers about customer valuation highlight both the variations in approaches used and some of the issues and problems inherent in the method. The manager of a program which uses a post-service "benefits and results" questionnaire said: "It is up to the person who did the project to send it out. Usually it is sent out within a month after the project. Some of the drawbacks of this distribution approach are (1) the potential for not getting feedback from the company, since there are no follow-ups; (2) this time period can be too soon to know whether the company is implementing recommendation; (3) managers in the company often have not thought about dollar and other numeric value of the job, so they typically do not put anything down for those questions." A second program administered client evaluations this way: "Client evaluations go to clients who received more than just information. We send out a formal questionnaire. The timing after project completion is left up to the discretion of the project manager, although it is usually 3-6 months afterward, depending on the type of project." In a third case, the following approach was used: "We send a survey measuring the overall quality of services provided to clients right after the conclusion of the project. We have considered going back a year later but feel that might overwhelm the company , in many cases, turnover means that our key contact is gone. We rely on the company to project the value of the service , improved sales, reduced operating expenses, increased technical capacity, increased human resource capability, capital investment, jobs saved or created. We ask them to project the economic value one year from implementation, three years from implementation, and over the life of the project based on whatever metric (gross sales, etc.) they choose. We also ask them whether they would use us again or recommend us to others. We feel that it is important to keep the survey at one page and mail it to them, rather than bringing it to the site and watching them fill it out. We want to make the process objective and pressure free."
Response rates reported by programs for customer surveys ranged from 30 to 90 percent. To improve response rates, programs use a range of techniques: "One staff member calls the client before sending them out, which ups the response rate for this staff member's client," a program manager disclosed. In another program, the return rate was increased from 30 percent to 60 percent after a more systematic follow-up system was implemented: "We've centralized it , not leaving it to the technical specialist. We send two waves, and we included a stamped, self-addressed envelope." A few programs use structured telephone interviews after customer service has been completed , an approach which often increases response rates if conducted systematically.
Several programs discussed the difficulties they had encountered with client valuations. For example, one program manager said: "Individual agents sent questionnaires to companies they worked with. These were worthless because the company, in an effort to help the program, tended to overstate benefits. Also, agents didn't send it to everyone they came in contact with, just those who they liked best." In a different program, we were told by a manager: "We have discussed how to get better information from [our questionnaire]. Two suggestions are (1) discuss questionnaire items with the client at the beginning of the job, particularly the potential impact of it; (2) follow up immediately after the job is completed and again after 6-12 months. Another suggestion is to send the questionnaire out later; however, [program] management wants to know about problems immediately so that they can be corrected."
In contrast, the managers of several other programs cited their post-service, customer satisfaction forms and protocols as particular strong points. Questionnaire brevity (usually one-page), simplicity, and aggressiveness in obtaining client responses were among the positive attributes of the client self-reporting process mentioned. Additionally, three programs had conducted statewide satisfaction surveys, which they considered to be very worthwhile and valuable. One program regularly surveys both client and non-client manufacturers in the state. Non-client responses provide information about why manufacturers do not use program services. The other two state programs who had commissioned special statewide customer satisfaction surveys believed these to be important components of their evaluation.
With the growth of the national MEP program, NIST managers have tried to develop more systematic ways to administer customer valuation surveys for centers within the MEP system. Initially, MEP centers were encouraged to administer surveys within 30 to 45 days after service, to be followed-up with a second survey one-year after service to verify subsequent impacts. However, much variation remained in the administration of these surveys and response rates were generally below the 70 percent level often used by oversight agencies, such as the U.S. General Accounting Office or the Office of Management and Budget. NIST then supported a pilot effort with several MEP centers in the midwest to use a post-service telephone approach administered to client firms by an independent survey consultant. The success of this method in achieving a high response rate led NIST, in 1995 and as noted earlier, to propose that MEP centers should use a NIST-sponsored independent survey organization which would conduct telephone interviews with client companies about six-to-eight months after services had been delivered [National Institute of Standards and Technology, 1995a]. Applied to service projects involving eight or more hours of center staff time, this method would relieve individual centers of responsibility for follow-up and allow a more consistent approach. Many MEP centers have welcomed this initiative, although other centers are reluctant to participate due to concerns about releasing client records to outside organizations and the loss of control and timeliness inherent in NIST's national approach. Differences are thus likely to continue in the way customer valuation methods are implemented.
3.3 External Reviews, Audits, and Consultant Studies
External reviews, audits, or consultant studies are examinations of program operations conducted by persons or review panels who are usually, but not always, independent from program administration. They are typically done to provide an "objective" view of the program's progress or performance. Such reviews may vary as to their purpose, their degree of independence, length of time, resources made available, and methods used. Perhaps the most common example is the financial audit. The purpose of this audit is to determine whether time allocations and expenditures were done in accordance with standard accepted accounting procedures for a program organization or sponsor. Usually such audits do not delve deeply into questions of substantive program performance or outcomes. However, where program performance is itself the issue, a range of outside reviews and assessments have been commissioned. These may include a complete review of the whole program, most often initiated by an outside funding sponsor. In other cases, reviews are made only of specific aspects of a program, such as service offerings, relationships with customers, evaluation methodology and structure, or outreach methods.
An example of one of these program reviews is the evaluation of the Southwestern Pennsylvania Industrial Resource Center directed by Pittsburgh University faculty member Roger Ahlbrandt [Ahlbrandt 1989a, 1989b, 1992]. This study used mail questionnaires, telephone surveys, and personal visits to assess technological, business, and employment impacts resulting from the program. It was found that most firms were satisfied with the services received although only a few had been able, at the time of questioning, to fully implement improvements based on the assistance provided. On a methodological note, Ahlbrandt concluded that it was important to get cooperation from the program, get access to companies, and check responses to determine whether they are valid. Further, it was suggested that mail questionnaires should be supplemented by personal interviews since responding companies are unlikely to give exact responses on surveys alone. High customer satisfaction rates were also found by evaluators at the Industrial Technology Institute used computer-assisted telephone interviews to obtain client perceptions of the Michigan Modernization Service, a now disbanded technology deployment and extension service [Morell et. al. 1989]. However, it was discovered that while clients were committed to implementing new technologies, concern about the financial risk involved was a barrier.
The "distance" of the reviewer from the program is a significant factor in the review process. Some reviewers are hired by the funding sponsor, rather than the actual program, and thus have a degree of independence. In other cases, outside evaluators or reviewers are hired from program funds, and may have a closer relationship. In such cases, the findings from the evaluation review may be held internally. In a few states, there are legislative or policy mandates to conduct external reviews on a frequent basis. For instance, New York state technology extension service is under a mandate to be reviewed every two years. In the most recent round, a consultant reviewer was selected through a request-for-proposal process and a thorough study was conducted using a variety of methodologies, including discussions with program managers and staff, document review, analysis of field agent reports, case studies, a survey of program participants, and a net impact assessment [Nexus Associates, 1994]. In another example, the Pennsylvania Department of Commerce has employed an outside consultant to verify customer satisfaction with the state's Industrial Resource Centers Program [KPMG Peat Marwick 1993].
In addition to state-level studies, a series of multi-state and national level reviews have also been undertaken. There have been several assessments by expert panels and oversight agencies of the first group of seven MTCs sponsored by NIST [Visiting Committee on Advanced Technology 1990; U.S. General Accounting Office 1991; National Research Council 1993]. These reviews, drawing on site visits, reviews of data and program materials, and interviews with clients, have generally concluded that the individual MTCs have been helpful in assisting small firms to improve their business practices and upgrade technology. Perhaps the most significant external assessments have been the Third-Year Reviews of NIST's Manufacturing Technology Centers. These reviews are mandated by legislation [Omnibus Trade and Competitiveness Act, 1988] and are carried out by independent panels of outside experts, chaired by a NIST official who is not directly associated with the MTC program. Between 1991 and 1995, Third-Year reviews were made of the first group of seven MTC centers established by NIST. The reviews required the centers to prepare written materials and reports and respond to on-site questions by panel members. Discussions were also held with other center stakeholders (such as state funders), center advisory board members, service providers, and industry representatives. The panels examined center performance in meeting program goals, leveraging funding, and linking with industry and other organizations, and made a recommendation about whether federal funding should be continued for a further three years. In several cases, Third Year reviews have led to fundamental changes in center management and organization. Observations and recommendations have also been made about NIST's overall management of the program [Manufacturing Technology Centers, Third Year Review Panel 1992]. With the great expansion of the MEP system since 1994 and the growth in the number of MTCs, NIST is building on this experience to refine the review criteria and procedures it will use for determining which of the new centers will transition from TRP to Department of Commerce funding and how this larger number of centers will be reviewed in future years.
Outside reviewers may focus on process evaluations or impact evaluations or both. Methods used by outside reviewers include comparison of legislative mandates with program goals and services, interviews with program administrators and staff, interviews or focus groups with clients and customers, discussions with program sponsors and host institutions, and examinations of annual reports, program business plans, and other program documentation. In some instances, the data collection portion of the process (e.g., interviews, document collection) often takes just a few days. In other cases, the lack of comparable data makes it hard to conduct detailed analyses [Martin 1993]. Only in relatively few studies have external reviewers had sufficient time, staffing, funds, or data to undertake more extensive evaluations, including questionnaire surveys and multiple on-site company visits.
Interestingly, the program managers we interviewed often had mixed impressions of the external review process. One reason given was poor feedback from external reviews. As one program manager remarked: "Our funding agency did an audit four years ago. They talked to 12 clients. We never saw the results." Another reason is that program administrators believe that most external reviews they have experienced are process evaluations which focus on amount of program activity. Outcome measures and detailed impacts on firms were felt by some to be given less attention by reviewers, possibly in part because data on such outcomes is usually limited, and may involve only evidence from a few case studies, plant visits, or discussions with (pre-selected) customer groups. In addition, some program managers questioned the knowledge and approach of outside reviewers. While reviewers may have expertise in business, government, or even program evaluation, they may not necessarily be familiar with specific programs. Other managers felt outside reviewers impose unwanted and unnecessary or standardized evaluation mechanisms on them. For example, one administrator contended: "External evaluators should be sensitive to local needs and differences. The evaluation should be based on an intimate knowledge of what is done." Outside evaluations inevitably impose burdens, threats, and fears on programs, depending of course on who does them, as expressed by one program manager: "Evaluations are not so much a problem if they're required by an agency within the state which is already friendly and knowledgeable. They're a greater problem if they're handled by a third party."
Nevertheless, most program managers agreed that external reviews should be included in a total evaluation system. Funding agencies respect the results of external reviews and these audits enhance the legitimacy of the program. One administrator lamented his program's lack of money for an external review. He described how a sister program in the state, which had commissioned bi-annual external audits for the past six years, was able to retain funding whereas his program's funding was virtually eliminated. "We should have had an outside source come in and review the program so we had documented proof to show legislators."
3.4 Economic and Regional Impacts
As manufacturing modernization and extension programs have increased in scale and resources in recent years, there has been increased interest in trying to assess not only outcomes for individual firms, but also economic and regional impacts. Measures of economic impacts may include the value of jobs created or retained, and changes in sales, exports, investment, payroll, and related factors. Measures of direct impacts (e.g., the number of new jobs or sales generated) may be supplemented with estimates of indirect effects (i.e., the cumulative amount of induced spending or "multiplier effect" that occurs as a result of the new direct economic activity). Ideally, such assessments should consider the variability and differences in regional economies and industrial sectors in analyzing program impacts. For example, in the numerator, one might expect program services to have a greater effect on firms located in a high growth region or industrial sector than a declining sector or outlying region. Similarly, in the denominator, there are likely to be variations in unit service costs since it is generally easier to serve clusters of firms in urban areas compared with rural locations where travel costs are higher and group activities are more difficult to organize.
Most program managers acknowledged that they had no formal mechanism for accounting for regional variability even though they operated in states with extensive differences among regional and subregional economies. Indeed, the recognition of these differences was often a fundamental element in program justification and operation. For example, in Minnesota the state initially funded its industrial modernization program as part of a strategy to aid decentralized and rural regions outside of the more prosperous Minneapolis-St.Paul metropolitan area [Minnesota Technology Inc. 1995]. Similarly, the state-sponsored industrial extension programs run by Georgia Institute of Technology had long focused the bulk of its field office assistance on industries and areas outside of metropolitan Atlanta [Georgia Manufacturing Extension Alliance 1995]. In both cases, additional federal funding through NIST allowed these programs to enhance services in their core metropolitan complexes. Yet, in terms of the relative resources allocated per industrial establishment, rural areas continue to receive favorable treatment. This reflects an overriding policy commitment (especially at the state level) to promote economic development in outlying areas whether or not this is the most effective use of resources from a narrow benefit-cost perspective.
To a great extent, however, there are many unknowns with regard to complete economic and regional impacts. As yet, industrial modernization programs have only rarely been subject to fully comprehensive economic and regional impact studies. But, while there is a relative absence of comprehensive economic evaluations, numerous programs developed partial or specific measures of the economic outcomes associated with their services. As part of customer valuations, many programs ask for quantitative measures of the dollar value of services provided , jobs created, sales increases, new investment, greater productivity. Several programs present this data prominently in reporting and marketing, while some present it cumulatively, summed for all the years of the program's existence (which usually results in presentably large numbers). Typical is one program, which reports on an annual and cumulative basis jobs added and retained, increased sales, customer savings, and capital investment. Yet, many program administrators frankly admit that such measures are not particularly reliable. Problems of tracking the full effects of program services over time and discerning them from other influences within and outside of company operations plague efforts to conduct economic impact studies. Other obstacles involve impinging on clients to obtain this information. One administrator stated, "It might take several years for advantages to pan out, which makes economics difficult to trace. Companies don't want you to get too involved in prying into their finances, especially smaller owner-operated companies. We find that a few companies will let you trace their impact, but I don't try to interpolate that their rate of success is the same as that for all companies."
In cases where economic impact studies have been undertaken, efforts have sometimes been made to estimate the "multipliers," or additional spending or income generated within a regional economy as a result of direct impacts from assistance provided to manufacturers and the implied total return on public investment [Burress and Oslund 1994]. But, inevitably, attempts to determine the total impacts of industrial modernization programs raise a series of methodological and measurement issues common to all economic impact analyses of technology and economic development programs. These include problems of attribution of cause and effect relationships, the placement of monetary values on varied and hard-to-quantify changes within firms, and the appropriate classification of costs and benefits [Feller and Anderson 1994]. Even though industrial extension and modernization programs typically serve all manufacturers in their state, the small scale of most programs makes it seem difficult to separate from other influences on the state's economy. One administrator remarked, "Impact analysis seems overwhelming to us. There are approximately 27,000 manufacturers in New York. How can 20 [industrial extension staff] people change 27,000 manufacturers? The whole economy swamps the effects of any one program's impact." Thus, while programs can observe impacts on individual firms or specific groups of firms, it often hard to attribute effects on whole industries or the broader state economy.
In some earlier studies, the inadequate treatment of the full range of costs as well as benefits and the lax use of standard multiplier ratios have led to implausibly high return ratios. For example, at times the monetary values of extra sales and new capital expenditures attributed to a modernization program are added together as part of analyses of program benefits, even though the latter is a cost to the investing firm. In other instances, only a portion of program costs are included in the analysis , an oversight most common among states and localities, who treat federal funds as money without cost, but sometimes found at the national level too in comparisons of program impacts on firms against federal (but not matching state) costs. A case in point is a published NIST estimate that there is a private return of over $7 for each federal dollar invested in MTCs between 1989 and 1992 , a calculation which both excludes state and local costs and mingles private costs and benefits [National Institute of Standards and Technology 1994a]. When a more complete accounting is made, the net economic impacts are smaller, although still significant. For instance, a model operationalized by two of the authors of this article with data from the first year of activities of the Georgia Manufacturing Extension Alliance (GMEA) finds a combined net public and private benefit-cost ratio of 1.2 to 2.7 [Shapira and Youtie, 1995]. Although these results are not entirely conclusive, since they depend, in part, on anticipated benefits and investments, they nonetheless indicate that GMEA's industrial modernization resources are leveraging relatively high levels of private investment which, in turn, appear to lead to favorable and positive public and private returns over time.
Efforts, such as those represented by the GMEA study, to evenhandedly treat the costs and benefits associated with industrial modernization must still grapple with other data deficiencies and unknowns. In many cases, firms state that their participation in program activities is beneficial, but they find it difficult, if not impossible, to estimate dollar changes in sales, investment, or other costs. The value to firms of "soft" services , information provision, seminars, or opportunities to participate in industrial networking groups , is even harder to quantify. Further problems include distinguishing net new impacts from redistributive shifts, for instance where a sales increase in one firm results in lost sales to another company, and determining the extent to which observed changes can be attributed to the program rather than other factors. Yet, despite the methodological difficulties and pitfalls of benefit-cost analysis, there remains great interest among program sponsors in using and refining the technique to assess the economic returns associated with industrial modernization programs. Indeed, many of the recent changes in NIST's evaluation reporting guidelines have been aimed at developing procedures to collect more reliable data from industrial customers to assess the benefits and costs they attribute to the assistance received including changes in sales, inventory, investment, employment, and payroll. Projects have been sponsored to improve how NIST and industrial modernization programs can use benefit-cost approaches and to use regional input-output modeling to better account for the range of direct, induced, and dynamic effects associated with program intervention [Feller 1995; Norris 1995]. Other programs are developing procedures to work with clients to estimate likely costs and benefits of projects before the project is implemented, which may then make it easier for firms to quantify the subsequent actual impacts. While these enhancements are valuable, estimates of the "bottom line" economic impacts of industrial modernization intervention are certain to remain replete with caveats.
3.5 Comparative Evaluation and Control Groups
In terms of formal evaluation design, most of the evaluation methods used by industrial modernization programs are either indirect (program monitoring or external reviews) or implemented with one data point after service completion. More systematic designs would involve a longer time series of information before and after service (before and after design) and comparison with similar firms who have not received program assistance (with and without design). Evaluation specialists are generally agreed that without this full array of time series and comparative control data, it will not be possible to conduct long-term and reliable assessments of the performance and impact of industrial modernization programs and services.
The development of common reporting guidelines between NIST and its MEP programs should allow the collection of data along several evaluation metrics for individual customers immediately prior to service delivery and post-service (at about six-to-eight months after service in most cases). Thus, for a common set of measures, two data points are being established over time, representing a basic "before and after" evaluation design. Another feature of the NIST MEP design is improved standardization of data collection and reporting procedures, which will allow comparisons between different MTCs (a "here and there" design). It is hoped that this will facilitate not only the benchmarking of center performance and best practice, but also the use of techniques such as data envelopment analysis to determine which MTCs most efficiently use resources. The data that NIST and its partner MTCs will collect is intended to provide a more accurate and consistent picture of what constituent MTCs are doing, who they are serving, and how they are deploying their resources. However, NIST also recognizes that this data will be insufficient, by itself, to reliably attribute observed effects to program intervention due to its limited time horizon (less than a year between service and follow-up) and lack of non-customer controls. Additionally, as the MEP centers will themselves insist, careful interpretation will be needed in comparisons of center performance to take into account regional, industry, technological, organizational, and other differences.
At the individual program level, there have been a few instances where longer-term, controlled studies have been conducted. These include the Nexus Associates  study of the New York Industrial Extension Service (discussed in Oldsman's article in this issue) which used state employment security records to provide non-customer controls for employment, firm size, and industry effects. Studies in West Virginia [Rephann and Shapira 1994] and in Georgia [Youtie and Shapira 1995] have used primary surveys of industrial establishments to obtain information on assisted and non-assisted firms and then used this data to statistically control for employment size and industry effects. Yet, despite these examples, the technical, resource, and data commitments necessary to implement formal controlled studies appear to be greater than many individual industrial modernization programs are able or believe are necessary to devote to evaluation. While many of the program managers interviewed in our study recognized that quasi-experimental designs based on data from assisted and non-assisted firms were particularly rich information sources, the cost and complexity of such approaches was seen as a barrier. In one case, it was noted that the cost of a full study would have consumed more than half of the program's total budget. As this program administrator explained, "One difficulty is how much money you can spend on evaluation. Therefore, we look at the resource commitment versus the kind of information coming back."
A different , and more affordable , approach to using control groups is represented by the Performance Benchmarking Service developed at the Industrial Technology Institute (ITI) in Ann Arbor, Michigan [Performance Benchmarking Service 1994]. Piloted at the NIST/Michigan Manufacturing Technology Center, housed at ITI, and now made available to MEP centers throughout the NIST system, this service allows individual MTC customer firms to assess their manufacturing performance against the typical and best practices found in comparable control groups of firms (as discussed by Luria and Wiarda in this issue). Benchmark data is collected from firms on a series of manufacturing measures, including throughput, quality, use of computers and programmable controllers, inventory turns, training, sales, employment, and material costs. A detailed report is then prepared for the MTC customer firm which, measure by measure, compares it performance with the control group. ITI conducts these benchmarking assessments for both MTC and non-MTC firms. The participating firms complete questionnaires (from 4 to 12 pages), with additional data collected on an ongoing basis. By the end of 1994, ITI had assembled a benchmarking panel of about 700 small and mid-sized manufacturing plants. The panel is strongest in the industrial specialties of stamping, metal fabrication, machining, tooling, machine building, and rubber and plastics, but also has representation from other industries. Individual firms who wish to be benchmarked are custom-matched with a sub-group of firms from the panel. For an MTC case, local MTC staff market and supervise the administration of the benchmark survey, the firm's management provides the data, ITI staff then centrally conduct panel matching, analysis, and report preparation, and the MTC staff then present the report and its findings to the firm. The benchmark report thus provides an objective basis for determining what further improvement steps the firm and the MTC might take.
In addition to its contribution to service delivery, performance benchmarking also promises to promote more systematic program evaluation. Participating MTCs are able to access, at relatively low additional cost, detailed information about their own customers and comparable non-assisted companies. This should allow performance comparisons of MTC-assisted versus non-assisted firms and assessments of the impacts of particular assistance services. In this respect, performance benchmarking has the features of a classic quasi-experimental evaluation design, combining both "before and after" and "with and without" data. It should be added that this approach depends on accurate self-assessment by participating firms to detailed information requests and requires an on-going program and enterprise commitment (to generate a sufficiently large number of benchmark reports over time and to repeat benchmark assessments for the same firms every one-to-two years to determine progress). There may also be tensions between the evaluative aspects of the approach and the service aspects. For example, the firms in the panel who are benchmarked but who are not otherwise MTC customers receive feedback, as a way of maintaining their interest in the panel. This feedback may lead these non-client firms to then go on to make changes. Thus, the "control group" has actually received service. In a sense, what may be best measured is the impact of services additional to benchmarking on MTC customers. The firms in the control group may have various kinds of selection bias as well (as do the customers, probably in different ways). Desires to market the benchmarking service (which the evidence suggests can be a valuable service to firms for which they are willing to pay), and thus generate fee income, may further skew the control group. Nonetheless, even with these caveats, performance benchmarking represents an important methodological advance and has the great advantage of "piggybacking" evaluative data collection onto a real service which is valued by firms. The role of ITI (with NIST's support) in standardizing the process and in developing the non-MTC control panels further reduces barriers for local MTC participation.
The potential efficiency, methodological, and cost-effectiveness advantages of centrally-established control groups have been recognized too by NIST, which has been promoting attempts to link reported data on firms served by its MEP affiliates with other national databases. NIST is working with the U.S. Census Bureau to examine the feasibility of using the Bureau's Longitudinal Research Datafile (LRD) and Unemployment Insurance (UI) databases to compare the performance of client with non-client firms [McGuckin and Redman 1995]. The LRD contains longitudinal data from the Census of Manufacturing and Annual Survey of Manufacturers for the 1972-92 period. The UI database contains annual information on employment and payroll. The size, coverage and consistency of these national data-sets should allow reasonable control groups to be established , providing, of course, that NIST is able to obtain systematic and matchable information from its MEP affiliates about what companies they have served. The strength of the national Census surveys is the data they collect on firm employment, wages, sales, input costs and other financial matters. In parallel research, the Census Bureau is trying to identify ways to use such metrics to assess changes in MEP client competitiveness relative to non-clients, for example, by comparing changes in sales and operating profits, the probability of business survival, and changes in productivity, employment, wages, and exports [Jensen and McGuckin 1995; Jarmin 1995].
However, while the data available through these national Census surveys can provide controls on key economic and employment measures and on general business performance, information is limited or lacking on other metrics important to the evaluation of industrial modernization services. These Census surveys do not probe direct measures of technological use and proficiency, quality, workforce skill, materials throughput, or other manufacturing parameters which industrial modernization programs frequently seek to influence. Nor is there data on whether control firms use other public or private assistance sources outside of the MEP system. For such specific information on a national basis, focused primary surveys , while more costly than using data already collected for other purposes , are necessary. Indeed, in recent years the Census Bureau has conducted two special surveys of technology use in manufacturing [U.S. Bureau of the Census 1989; 1993]. Unfortunately, these surveys did not inquire about the use of modernization assistance services. But, if such questions were asked in future surveys , or ways found to couple the data with information from MEP customers , additional valuable insights could be provided to assess the performance of MEP-assisted firms in comparison with non-assisted firms or firms assisted through other means.
3.6 Assessments of Practice and Tools
A final category of evaluation is the assessment of program practices and tools. Insights about how industrial modernization programs are organized and analyses of program "best practices" have been probed in numerous studies [for example, Clifton, et. al. 1989; Modernization Forum 1993; Shapira 1990; Shapira, Roessner and Barke 1995; Shaw 1987; Simons 1993 Technology Management Group 1989]. Findings are reported about what seem to be effective approaches for outreach and services to firms, deploying technology, organizing and administering programs, staffing and training, funding, and linking with other business assistance and economic development initiatives. Several studies have focused on the role of group services and networking as a strategic and efficacious method of strengthening small and mid-sized firms [Gaincola 1991; Rosenfeld 1992; Broun 1994, 1995]. Other reports have examined individual service tools and methods of technology transfer [Myers, et. al. 1992; Minnesota Project Innovation 1991; Morell et. al. 1992; Office of Technology Development 1991; Swanson 1991] or highlighted particularly successful cases of service delivery [Suenram 1993]. These studies have been sponsored by national funders (such as NIST), other interested federal agencies (such as the Economic Development Administration or the National Science Foundation), state agencies, and private foundations.
Varied methods are used in these assessments of industrial modernization practices, including customer surveys and focus groups. However, the predominant method is the case study. Often, the aim of the study is to highlight and document programs, tools, or services already deemed to be exemplary, based on practitioner or customer feedback. In some instances, groups of practitioners and other knowledgeable experts have been polled, surveys conducted, or program information records reviewed to identify good cases for further study. More frequently, the cases are nominated through informal discussions with program managers and field personnel. In a few cases, funding agencies have sponsored pilot programs to field test new tools or methods and have required the documentation of experience and results.
Paradoxically, while the subject matter , focusing on effectiveness in program structures, tools and services , is presumably of great interest to program managers and sponsors, these studies are often not widely circulated in written form (even fewer are published in journals). In most cases, practitioners tend to obtain insights about what works best by word-of-mouth, through conferences, professional meetings and training sessions such as those promoted by NIST or the Modernization Forum (the national trade association of MTCs), and through their own personal and organizational experience (all aspects of what Sabel describes as discursive evaluation in a following article in this issue). However, a NIST-sponsored attempt is underway to improve how case studies of effective programs, best practices, and client successes are selected, documented, and disseminated. This involves developing system-wide procedures to select key issues for study, provide training in case study techniques, and establish methods for review [Yin 1995]. The aim is to focus on evaluation issues and program practices which cannot be easily measured through quantitative methods and highlight, both to other programs and to sponsors, the steps through which best practices and good results are achieved. There is an apparent tension in this effort between targeting cases (for elected officials) which are public relations successes and (for program managers and staff) which also promote learning about more mundane but important items (like client tracking systems) or address failures. It remains to be seen how NIST, as it operates in a less friendly funding environment, will balance these competing approaches.
The interviews with program managers indicated two principal customers for evaluative information: internal management and external program sponsors, including funding entities and government officials. In the latter category, state legislators, the governor, and the agency overseeing the funding , usually the state Department of Commerce , were frequently seen as the principal targets for evaluative information, along with federal agencies. In a few cases, foundations and university deans or presidents were mentioned. Relatively little emphasis was given to industrial firms as an actual or potential customer for detailed evaluation information.
What performance standards do program sponsors look for? A clear distinction emerged between programs in which quantitative measures were considered performance standards versus those in which qualitative measures justified continued funding. On the quantitative side, the program managers noted that state or federal executive agencies more frequently favored quantitative measures. Administrators indicated that these agencies were particularly interested in process measures, such as number of projects and number of clients assisted. Program managers expressed divergent views about whether legislators favored quantitative or qualitative measures (such as case examples or success stories). While many legislators seemed to want to know about success stories, one administrator said: "A few of our legislators are more sophisticated. They want to see a return on their investment rather than anecdotal impressions."
Several administrators observed that jobs was the "quantitative measure" in which most legislators were interested, notably the number of jobs created in their district. It was generally recognized that one problem with this measure is that industrial extension and modernization programs lack good procedures for quantifying job impacts (although this did not stop programs from trying , or being required , to measure job impacts). Another problem is that programs often seek to address the needs of their client, the small and midsized manufacturer, by focusing on improving manufacturers' competitiveness. At times, improving competitiveness (via productivity increases) could result in a decrease in employment in certain instances. Thus, as one interviewee put it, "We have two real clients , the manufacturers with their own set of needs, and the sponsors with a different list of what they expect from us. Sometimes these conflict."
In at least one state (New York), a formal quantitative measurement procedure has been established. Under state law, the legislature mandated that the industrial technology extension service program submit an independent evaluation of the program every two years. The legislature also enumerated items to be included in the evaluation , a determination of whether the services provided have helped client firms to improve their competitiveness or increase their market share relative to other firms in the specific industry, and the effect of the program on the continued location and growth of industrial firms within the state. In other states, there was no requirement to report specific quantitative measures and legislative decisions about continued program funding relied heavily on qualitative factors. Particularly critical to state legislators, as narrated to us by the program administrators who interact with these elected officials, were client success stories. These were felt to be most valuable when positive accounts of the program were made to legislators by business constituents. Such accounts appeared to carry more weight with legislators than with program managers , a contention best expressed in one administrator's comment, "Our legislators have been good about taking things on faith."
Not surprisingly, the methods programs use to report their evaluative information are related to the nature of the performance standards set. Program administrators who work towards quantitative measures tend to communicate program results to sponsors and other stakeholders through annual reports and activity summaries. Some program managers felt that the reports they were asked to prepare were "superficial" and did not capture the full details of their program's activities. Others indicated that they prepared these reports, but there was little demand for them. Interestingly, those programs who work towards qualitative measures of success appear to place a greater emphasis on dissemination. One administrator indicated, "Our most successful method of communicating to legislators is via videotape of success stories or other forms presenting anecdotal information. Sometimes we will conduct legislator tours through satisfied customer plants. We also have asked satisfied customers to write their legislators letters of appreciation, an admittedly self-serving approach." Another replied, "Anecdotal success stories work best. I bring out companies to tell their stories at conferences; place stories in the press; have them testify at legislative hearings; I tell these stories in speeches; they are summarized in promotional literature." In one state, this communication has been institutionalized in the "Top Five" program. Each engineer presents the five most successful engagements. This procedure results in a bank of some 30 success stories for dissemination to interested legislators. At the federal level, NIST as an executive agency has required its state affiliates to prepare and submit detailed and frequent quantitative accounts of program activities, although more recently a somewhat greater weight has been placed on qualitative case studies.
Few programs have formal mechanisms for disseminating results to the clients themselves. Aside from private sector representatives on a program's advisory board, manufacturers typically do not play a role in setting performance standards or receiving evaluation results. But programs sometimes present brief summaries of their (positive) results and often report successful cases in magazines and newsletters distributed to the general manufacturing community in their regions. In other instances, industrial executives are asked to recount their successful stories to visiting review panels, journalists, training conferences, and legislative hearings. At least one program produces a glossy brochure which prominently indicates a high customer satisfaction rate. This is used in program marketing to potential new customers, as well as for distribution to public officials.
Programs also use evaluative information for internal program management purposes, to identify areas where they should make changes. These modifications might involve personnel, service offerings, outreach efforts, or organizational structure. When asked if their program had modified its services based on evaluation results, the examples given included: hiring a new industry specialist; avoiding low-payoff work with vendors or workshops; adding more assessments and seminars on ISO 9000; offering training only if there is follow-up technical assistance; moving into fee-based services for shop-floor, practical assists; moving away from short-term assists and into longer engagements that effect the technological sophistication of groups of firms; avoiding invention evaluation; changing an advisory board to increase private industry representation; and changing electronic services in terms of format and content , introducing new topics, bringing in new data bases, TQM, ISO 9000. This eclectic mix suggests that the changes managers make as a result of evaluation feedback are quite particular to their own programs. However, when recounting how programs had been modified as a result of evaluations, several managers said that these changes were not exclusively due to evaluation outcomes. Some interviewees credited program experience or informal client requests as more important change sources than formal evaluation results. Feedback from evaluations seemed more likely to lead to program modifications under some circumstances than others. For programs in their first years of operation, administrators appear to be more receptive to feedback. Also, when the funding agency has commissioned an external review or audit, program administrators are more likely to be compelled to make changes or lose funding.
Our review of current approaches to evaluation in the industrial modernization field presents a somewhat mixed message. There are at least as many weaknesses as strengths. Program managers were generally frank in their own assessments of their program's approach to evaluation. Indeed, they usually offered more flaws than exemplary practices, in some cases because our interview questions and probes led them through aspects of evaluation they had not previously considered or implemented. The readiness to admit to gaps can be taken as a positive sign and, bolstered by the series of new federal initiatives and observable improvements in information reporting underway at the program level, we detected for most programs a desire to upgrade the quality of evaluation practice. At the same time, real obstacles are presented by the difficulties in obtaining reliable estimates of impact, limited resources, uncertainties about methodologies and metrics, and the fragmented nature of the "demand" for evaluation by program sponsors and other stakeholders.
Interestingly, while many programs sought to pay more attention to evaluation, a few programs indicated that less emphasis would be desirable. Those who wanted more extensive evaluations did so mainly to help them better identify program impacts. In particular, the need for improved quantitative measures of such outcomes as job creation was mentioned. Also mentioned was the need to develop more systematic evaluation designs. Several managers recognized their inability to distinguish extraneous factors from program effects. Some referred to the difficulty of identifying outcomes from short-duration technical assists, whereas others described the need for follow-up mechanisms to identify long-term impacts.
On the other hand, managers who wanted less evaluation emphasized the complexity of their current methods. Several respondents said they needed to shorten their customer satisfaction and valuation reporting forms. They indicated that the forms presently in use seemed difficult for clients to complete, and that many clients left large blocks of questions unanswered.
State program managers also expressed mixed views about the federal role in evaluating industrial extension and modernization program. Some believed that NIST could play an important role in evaluating these programs. Standard-setting was most commonly recommended. Several administrators endorsed national efforts to develop baseline measures and encourage states to adopt them, particularly the establishment of common measures in customer valuation questionnaires. One administrator suggested that NIST should establish clearer performance standards for program structure and operation, for example, regarding staff capacity and expertise, public versus private sector organization, and the degree of regionalization or decentralization. Another federal role recommended by a few interviewees was that of an objective external evaluator or as a catalyst encouraging the use of professionals to independently verify program results.
At the same time, a significant number of administrators feared the establishment of a greater federal role in evaluating industrial extension and modernization programs. Some contended that federal evaluators would impose common standards on them, without knowledge or consideration of important local differences. There is also a belief that federal evaluators would over-emphasize certain activity measures and numerical counts of program interactions, based on their administrative experience with federal evaluations in other programs. And many interviewees mentioned the likelihood of substantial increases in the volume of paperwork with a federal role in evaluating their programs.
Inconsistencies in how state and local program managers viewed both the federal function in program management and the role of evaluation were evident in their comments. Program managers believed that the system should be sensitive to local needs and differences and yet they were concerned that individual programs should not reinvent what others had already done or make the same mistakes. They sensed a need for methodical standards to assess and compare program performance, but feared that evaluators would come in with the same list of questions for each program, irrespective of local circumstances. And, of course, some managers were quite willing to accept federal matching funds for their programs but simultaneously less pleased that federal sponsors should question whether these monies were being spent effectively.
However, there is more going on here than the individual idiosyncracies of program administrators. Our review of current practices has revealed a number of fundamental tensions among evaluators, evaluation sponsors, program sponsors, program staff, and businesses served. Some of these tensions may be eased through improved evaluation designs and more valid and reliable metrics, but many , perhaps most , appear to reflect inherently incompatible demands and requirements. For example, program sponsors, especially executive agencies, increasingly insist that ongoing quantitative benefit measures be obtained from client firms. Not only does this conflict with firms' reluctance to provide any outside entity with detailed cost, investment, or profitability data, but it also assumes that firms can actually provide such data reliably even if they agreed to do so. Our review indicates that firms typically do not conduct what are, in effect, internal audits of expenditures such as cost sharing with external research providers, purchase of engineering consulting services, or staff time allocated to diagnosing manufacturing problems and implementing solutions. When asked to estimate quantitatively the costs and benefits associated with industrial modernization activities provided by a local service provider, firms frequently are reluctant, resistant, or incapable of responding reliably.
A second, related issue concerns the desire for industrial modernization program staff to maintain favorable relations with past and potential clients. The more that clients are burdened with estimating the results of services provided, the more costly to the firm those services become. At some point, the intrusion is not worth the benefits. Whether client reporting is required by the program, funding sponsors, or other third party evaluators, the program bears the blame for the costs of reporting if those costs become onerous. If information collection for evaluation is conducted directly by program field staff, the potential for damaging the program-customer relationship is reduced, but this may subtract from staff time available for delivering modernization services and will raise questions about the reliability of the data.
A third and absolutely critical issue concerns the multiple and at times conflicting purposes that are evident at all points in the evaluation cycle. Federal and state goals for industrial modernization programs are not only usually vaguely defined but also at odds in some cases. Asynchronous time horizons are employed, with different stakeholders expecting results over time periods varying from a single budget year to a multi-year technological generation. The weight placed on quantitative versus qualitative measures differs by funding sponsor and governmental branch. Additionally, while evaluation is often said to be focused towards program learning and self-improvement, in reality it is frequently targeted to the requirements of program justification and the continued flow of public funds.
Fourth, it is apparent that there are wide variations in the depth and methodological quality of existing evaluation approaches in the industrial modernization field. This is not an issue of quantitative versus qualitative methods, since on both sides of this particular house we find robust as well as rather less sound evaluation designs and procedures. Rather, the variations we observe can be attributed to such factors as decentralized and diverse structures of program management, varied management and state priorities and interests in evaluation, the rapid growth of the field and the tendency to develop evaluation procedures without reference to best practice elsewhere, and the intrinsic difficulty of obtaining reliable answers at reasonable cost.
Fifth, there is the issue of the relevance of evaluation findings to program stakeholders and, in particular, how the results of evaluations can be better disseminated and used to guide, on the one hand, strategic policy, budgetary, and targeting decisions and, on the other, program service approaches, quality, and continuous improvement. As our review of practices noted, elected decisionmakers often seek results on a short timetable and are frequently most interested in selected indicators which match their own agendas, but which may not necessarily be primary aims of industrial modernization. In turn, while there are a handful of state and local industrial modernization programs which have made major internal commitments to evaluation, in other programs evaluation is accorded relatively little weight, if not feared, and is conducted largely to satisfy the requirements of external funders.
What might be done about these kinds of problems? For as long as evaluation of public programs has been a significant activity, some of these issues have existed. With the provision of public support came the requirement that public service providers be held accountable. But the more recent emphasis on the use of public resources to assist private firms, as opposed to assisting other units of government or individuals entitled by law to receive public support, has created a new set of issues and tensions. Additionally, there is now a greater desire to manage by measurable results , a feature government has borrowed from the burgeoning quality control movement in the private sector itself. Yet, the clients of industrial modernization may decline services if the burden of evaluation measurement is perceived as excessive; they may certainly refuse to supply detailed, accurate , or any , information. The low response rates often seen in surveys of manufacturers, even those served by public programs, reflects the latter point.
Various solutions have been put forward to address at least some of these concerns. One approach, often advocated most strongly by those who seek dollar cost sharing or fees-for-service from their clients, is to let the willingness (or otherwise) of firms to pay some or all of the cost of service be the primary market test of whether the program is worthwhile. Yet, unless such programs become entirely self-sufficient (in which case they would be like private consulting businesses and would have to narrow their scope of service) and as long as they accept some public subsidy, there will continue to be pressure for some level of evaluation. Others argue that public support for industrial modernization is intrinsically valuable, that it creates and saves jobs, that competing industrialized economies provide such services, and that long-term funding should be forthcoming. By this logic, not much weight should be placed on evaluation either, since it is already assumed that industrial modernization programs work well. While such "take it on faith" arguments have worked to date in some states, in the current era of tight budgets and continued national contest about the role of government, more substantial program justifications are necessary (particularly now that the U.S. industrial modernization effort is itself more substantial and thus attracts more attention).
If the need for evaluation is accepted, the challenge then becomes how to make it more effective and useful without imposing excessive burdens on private industrial clients , and also on the industrial modernization programs themselves. We do not presume there are simple or costless answers to this challenge, nor to the other matters raised in the preceding discussion of issues. However, our review suggests several promising directions to pursue. One is the further development of techniques such as benchmarking which allow evaluative measures to be collected unobtrusively while at the same time serving as a valuable diagnostic tool for firms and field staff. Additionally, the implementation of a few long-term well-designed national, regional, and sectoral special studies using controlled samples of clients and non-clients is likely to lead to more robust and policy-relevant results (with less intrusion) than multiple local programs employing disparate methods and questions in frequent universal surveys only of clients. Program managers, industrial representatives, and policymakers should be involved in the design and review of such special studies. Where possible, key questions about the deployment of technologies and techniques and the use of varied public and private assistance sources should also be integrated into ongoing and broader national manufacturing censuses and surveys to provide a better framework for decisionmaking across the gamut of policies for industrial and technology promotion.
We would also argue that there is now a need to enhance evaluation efforts which focus not so much on the programs themselves but on the effectiveness and value of particular kinds of services. This is especially important for program improvement, as managers and sponsors grapple with such questions as where best to target their resources, which services lead to what results and why, and what are the benefit-cost tradeoffs associated with particular services. Case studies, which illuminate how specific services are delivered and received (the "wiring"), as well as quantitative analyses (of service "inputs" and "outcomes") are apt to be valuable here. In parallel, considerably greater resources might be devoted to the development of better models that link the provision of specific services to measures of regional economic development. For many evaluation sponsors, this level of aggregation is all that is needed; if the intermediate step of data collection at the level of the firm can be avoided or reduced considerably, the burden of accountability would fall far less heavily on clients themselves. But where the performance of program management is itself the issue, external reviews and site visits (which can consider the individual circumstances facing a program and consult with customers as well as service providers) seem to be more useful than applying standard quantitative measures based on the number of firms assisted for a given budget. Finally, and, perhaps most important of all, is the further promotion of opportunities to engage program professionals, industrial representatives, elected officials and other stakeholders, within and across their own constituencies, in discussion and review of program practices, performance standards, evaluation findings, and related issues with the aim not so much of discussing evaluation or methodology for its own sake but of ensuring continued program relevance and ongoing program improvement.
Advanced Research Projects Agency 1993, Technology Reinvestment Project: Program Information Package (U.S. Department of Defense, Arlington, VA).
Ahlbrandt, R.S., Jr., 1989a, Assessment of the First Year's Activities of the Southwestern Pennsylvania Industrial Resource Centers (University of Pittsburgh, Pittsburgh, PA).
Ahlbrandt, R.S., Jr., 1989b, Southwestern Pennsylvania Industrial Resource Centers: Assessment of Second Year's Activities (University of Pittsburgh, Pittsburgh, PA).
Ahlbrandt, R.S., Jr., 1992, Helping small manufacturing companies become more competitive: A model and an evaluation, Economic Development Review, Winter, 67-71.
Andrews, E., 1995, Congress and White House split on high tech, New York Times, January 3, C19.
Berglund, D. and C. Coburn, 1995, Partnerships: A Compendium of State and Federal Cooperative Technology Programs (Battelle Press, Columbus, OH).
Broun, D., 1994, The FlexCell Group: Salvaging the metal-cutting and pattern-making industry, Firm Connections (Regional Technology Strategies, Inc., Chapel Hill, NC), 2,5.
Broun, D., 1995, The Philadelphia experiment: Seeing a beginning in the end, Firm Connections (Regional Technology Startegies, Inc., Chapel Hill, NC), 3,3.
Burress, D. and P. Oslund, 1994, Public Returns to Technology Transfer: An Evaluation of the Kansas Technology Enterprise Corporation Using the ROPI Methodology (Institute for Public Policy and Business Research, University of Kansas, Lawrence, KS), Report No. 212.
Carr, K., 1994, Statement about technology policy and programs at NIST, Subcommittee on Oversight and Investigations, Committee on Science, Space and Technology, United States House of Representatives, Washington, D.C., July 7.
Chicago Manufacturing Center, 1995, Fourth Quarterly Report for the Chicago Manufacturing Technology Extension Center (Chicago, IL), April.
Clark, M.K. and E. Dobson 1991, Increasing the Competitiveness of America's Manufacturers: A Review of State Industrial Extension Programs (National Governors Association, Washington, DC).
Clifton, D., L.R. Edens, H.T. Johnson, and R.W. Springfield, 1989, Elements of an effective technology assistance policy to stimulate economic development, Economic Development Quarterly, 3,1: 5257.
Clinton, W.J. and A. Gore, Jr. 1993, Technology for America's Economic Growth: A New Direction to Build Economic Strength (Executive Office of the President, Washington, DC), February 22.
Combes, R.S., 1993, Origins of industrial extension: a historical case study in Georgia (M.S. Thesis, Ivan Allen College of Management, Policy and International Affairs, Georgia Institute of Technology).
Farrell, E.J., 1994, A business perspective on evaluating industrial modernization programs, in: P. Shapira, J. Youtie, and J. D. Roessner (Editors), Evaluating Industrial Modernization Programs: Issues, Practices, & Strategies, (School of Public Policy, Georgia Institute of Technology, Atlanta, GA), 55-58.
Feller, I., and G. Anderson, 1994, "A benefit-cost approach to the evaluation of state technology development programs, Economic Development Quarterly, 8,2, 127-140.
Feller, I., 1995, The Use of Benefit-Cost Analysis in Evaluating the Impacts of Manufacturing Technology Centers (Institute for Policy Research and Evaluation, Pennsylvania State University, University Park, PA), mimeo.
Georgia Manufacturing Extension Alliance, 1995, First Year Review. Materials prepared for NIST review panel (Georgia Tech Economic Development Institute, Atlanta, GA).
Giancola, D. 1991, Heat in Ohio: The difficult birth of an American network, The Entrepreneurial Economy Review, 9, 3: 19-23.
Heller, E.S., 1995, The monitoring-evaluation-research continuum: Insights from the Massachusetts Manufacturing Extension Partnership, in: P. Shapira and J. Youtie (Editors), Evaluating Industrial Modernization: Methods and Results in the Evaluation of Industrial Modernization Programs (School of Public Policy and Economic Development Institute, Georgia Institute of Technology, Atlanta, GA), 87-98.
Jarmin, R., 1995, Using Matched Client and Census Data to Evaluate the Performance of the Manufacturing Extension Partnership (Center for Economic Studies, U.S. Bureau of the Census, Washington, DC), mimeo.
Jensen, B. and R. McGuckin, 1995, Firm Performance and Competitiveness: Evidence from Microdata Research (Center for Economic Studies, U.S. Bureau of the Census, Washington, DC), mimeo.
KPMG Peat Marwick, 1993, Customer Satisfaction Survey of the Pennsylvania Industrial Resource Centers (Pennsylvania Department of Commerce, Harrisburg, PA).
Luria, D., 1993, A Framework for Evaluating the NIST/MTCs: A Summary Based on the Draft Recommendations of the NIST/MTC Evaluation Working Group (Industrial Technology Institute and Midwest Manufacturing Technology Center, Ann Arbor, MI), January 20.
Manufacturing Technology Centers, Third Year Review Panel 1992, Manufacturing Technology Centers: Broad Programmatic Issues (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
Martin, S.A., 1994, The Effectiveness of State Technology Incentives: Evidence from the Machine Tool Industry (Center for Agricultural and Rural Development, Iowa State University, Ames, IA, 1994).
Martin, S.A., 1995, Output-based performance measures for the North Carolina Alliance for Competitive Technologies, in: P. Shapira and J. Youtie (Editors), Evaluating Industrial Modernization: Methods and Results in the Evaluation of Industrial Modernization Programs (School of Public Policy and Economic Development Institute, Georgia Institute of Technology, Atlanta, GA), 97-113.
McGuckin, R., and J. Redman, 1995, Measuring and Monitoring Client Firm Competitiveness (Center for Economic Studies, U.S. Bureau of the Census, Washington, DC and National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD), mimeo.
Mendelowitz, A., 1995, Overview of GAO's study of companies' perspectives on manufacturing extension program services, in: P. Shapira and J. Youtie (Editors), Evaluating Industrial Modernization: Methods and Results in the Evaluation of Industrial Modernization Programs (School of Public Policy and Economic Development Institute, Georgia Institute of Technology, Atlanta, GA), 156-158.
Minnesota Project Innovation, 1991, Final (Fourth Quarter) Report, Boehlert-Rockefeller Technology Extension Program (Minnesota Project Innovation, Inc., Minneapolis, MN).
Minnesota Technology Inc., 1995, Minnesota Manufacturing Technology Center: Third Year Review. Materials prepared for external review panel (Minneapolis, MN).
Modernization Forum, 1993, MTC Project Development and Management (Dearborn, MI).
Morell, J., S. Wolfe, and P. Babcock, 1989, Evaluation of the Michigan Modernization Service (Industrial Technology Institute, Ann Arbor, MI).
Morell, J. et. al., 1992, Technology Application Development Process for the State of Michigan. Final Report to the Michigan Strategic Fund (Industrial Technology Institute, Ann Arbor, MI).
Myers, C., J.T. Benham and J.W. Ahlen, 1992, Technology Extension Project of the Arkansas Science and Technology Authority. Final Report to the National Institute of Standards and Technology (Arkansas Science and Technology Authority, Little Rock, AK).
National Association of Management and Technical Assistance Centers, 1989, Members' Evaluation Forms (NAMTAC, Washington, DC).
National Coalition for Advanced Manufacturing, 1994, New direction of the House Science Committee, NACFAM Alert (Washington, DC) December 15.
National Institute of Standards and Technology, 1990, Technology-Based Economic Development: A Review of State Industrial Extension Programs, Special Publication 786 (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
National Institute of Standards and Technology, 1993, Manufacturing Technology Centers: Quarterly Reporting Guidelines (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
National Institute of Standards and Technology, 1994a, Setting Priorities and Measuring Results, electronic document available through http://www.nist.gov (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD), January.
National Institute of Standards and Technology, 1994b, Manufacturing Extension Partnership, electronic document available through http://www.nist.gov (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD), December.
National Institute of Standards and Technology, 1995a, The NIST/MEP Management Information Reporting Guidelines, Version 2.02 (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
National Institute of Standards and Technology, 1995b, Manufacturing Extension Partnership: Making a Difference for America's Manufacturers (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
National Institute of Standards and Technology, 1995c, Delivering Results: A Progress Report from the National Institute of Standards and Technology,electronic document available through http:/www.nist.gov (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD), May 17.
National Institute of Standards and Technology, 1995d, NIST Industry Impacts,electronic database of industry impact case studies available through gopher//gopherserver.nist.gov:79/1/.menus/.impacts (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
National Research Council, 1993, Learning to Change. Opportunities to Improve the Performance of Smaller Manufacturers (Commission on Engineering and Technical Systems, Manufacturing Studies Board, National Academy Press, Washington, DC).
Nexus Associates, 1994, Evaluation of the New York Industrial Technology Service. (Cambridge, MA). Report prepared for the New York State Science and Technology Foundation.
Nexus Associates, 1995, The Guide to Program Evaluation (Cambridge, MA). Report prepared for the Manufacturing Extension Partnership, National Institute of Standards and Technology.
Norris, G., 1995, Economic impact simulation to support MEP project evaluation (Office of Applied Economics, National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD), mimeo.
Omnibus Trade and Competitiveness Act, 1988, Public Law 100-418, 15 U.S.C. 278k.
Office of Technology Development, 1991, Manufacturing Technology Transfer. Final Report of Contract Linking NIST Regional Centers to Aid Pennsylvania's Foundation Firms (Department of Commerce, Commonwealth of Pennsylvania, Harrisburg, PA).
Performance Benchmarking Service, 1995, The Bench Press (Industrial Technology Institute, MI), 1,1, Fall.
Rephann, T. and P. Shapira, 1994, New Technology Adoption in West Virginia: Implications for Manufacturing Modernization Policies, WP9403 (School of Public Policy, Georgia Institute of Technology, Atlanta, GA).
Rosenfeld, S., 1992, Competitive Manufacturing: New Strategies for Regional Development (Center for Urban Policy and Research Press, New Brunswick, NJ).
Shapira, P., 1990, Modern times: Learning from new state initiatives in industrial extension and technology transfer, Economic Development Quarterly, 4, 3, 186-202.
Shapira, P., 1990, Modernizing Manufacturing: New Policies to Build Industrial Extension Services (Economic Policy Institute, Washington, DC).
Shapira, P. and J. Youtie, 1994a, Georgia Manufacturing Extension Alliance: Overview of the Evaluation Plan (Georgia Tech Economic Development Institute, Atlanta, GA).
Shapira, P. and J. Youtie, 1994b, USNet 1994 State Baseline Assessment: Preliminary Analysis, USNet Evaluation Paper 9402 (Regional Technology Strategies, Chapel Hill, NC).
Shapira, P., and J. Youtie (Editors), 1995, Evaluating Industrial Modernization: Methods and Results in the Evaluation of Industrial Modernization Programs (School of Public Policy and Economic Development Institute, Georgia Institute of Technology, Atlanta, GA).
Shapira, P. and J. Youtie, 1995 Assessing GMEA's Economic Impacts: Towards a Benefit-Cost Methodology GMEA Evaluation Working Paper E9502 (Georgia Tech Economic Development Institute, Atlanta, GA).
Shapira, P., 1995, Modernizing small manufacturers in the United States and Japan: Public technology infrastructures and strategies," in: M. Teubal, D. Foray, M. Justman and E. Zuscovitch (Editors), Technological Infrastructure Policy (TIP): An International Perspective (Kluwer Academic Publishers, Boston, MA).
Shapira, P., J. D. Roessner, and R. Barke 1995, Public infrastructures for industrial modernization in the United States, Entrepreneurship and Regional Development 7, 63-84.
Shapira, P., J. Youtie, and J. D. Roessner (Editors), 1994, Evaluating Industrial Modernization Programs: Issues, Practices, & Strategies, (School of Public Policy, Georgia Institute of Technology, Atlanta, GA).
Shaw, B.T. 1987, Strategies for Delivering Technology Assistance to Small Manufacturers: Lessons from Pennsylvania, 87-WA-TS-9 (New York, NY: American Society of Mechanical Engineers).
Simons, G., Industrial extension and innovation, in: L. M. Branscomb (Editor), Empowering Technology: Implementing a U.S. Strategy (MIT Press, Cambridge, MA).
Southwestern Pennsylvania Industrial Resource Center, 1995, Fourth Quarterly Report for the MEP of Southwestern Pennsylvania (Pittsburgh, PA), April.
Suenram, R., 1993, A Collection of Successful Interactions Between the MTCs and Client Firms. NIST SP 848 (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
Swamidass, P., 1994, Technology on the Factory Floor II: Benchmarking Manufacturing Technology Use in the United States (The Manufacturing Institute, Washington, DC).
Swanson, D., et. al., 1991, Methods to Transfer Federal Technology and Improve Industrial Competitiveness (Georgia Tech Research Institute, Atlanta, GA).
Sylvia, R., K. Meir, and E. Gunn, 1985, Program Planning and Evaluation for the Public Manager (Waveland Press, Prospect Heights, IL).
Technology Management Group, 1989, How to Start and Operate an Effective Business and Technology Assistance Program. Prepared for the U.S. Department of Commerce, Economic Development Administration, Grant Number 99-06-07263 (Philadelphia, PA: Technology Management Group).
U.S. Bureau of the Census, 1989, Manufacturing Technology 1988, Current Industrial Reports (U.S. Government Printing Office: Washington, DC).
U.S. Bureau of the Census, 1993, The Survey of Manufacturing Technology: Factors Affecting Adoption, SMT/91-2 (U.S. Government Printing Office: Washington, DC).
U.S. Congress, Office of Technology Assessment, 1990, Making Things Better: Competing in Manufacturing, OTA-ITE-433 (USGPO, Washington, DC).
U.S. General Accounting Office 1991, Technology Transfer: Federal Efforts to Enhance the Competitiveness of Small Manufacturers (GAO/RCED-92-30, Washington, DC).
Visiting Committee on Advanced Technology 1990, The Manufacturing Technology Centers Program, Report to the Secretary of Commerce (National Institute of Standards and Technology, U.S. Department of Commerce, Gaithersburg, MD).
Youtie, J. and P. Shapira, 1995, Manufacturing Needs, Practices, and Performance in Georgia: 1994 Georgia Manufacturing Technology Survey, GMEA Evaluation Working Paper E9501 (Economic Development Institute and School of Public Policy, Georgia Institute of Technology, Atlanta, GA).
This article draws upon research supported by the U.S. Department of Commerce, National Institute of Standards and Technology (under awards 70NANB3HI388 and 60NANIB4DI628). The views expressed are the authors' and do not necessarily reflect those of the research sponsor.