In spite of decades of experience, we still find that some problems in survey design recur and often invalidate or make doubtful the value of these expensive exercises.
Lack of representativeness and low response rates of postal employee surveys Postal employee surveys in German public agencies are typically conducted by distributing questionnaires to employees who are requested to fill them out, and place them in an anonymous envelope for return to a collection area.
This method offers several advantages:
The challenge in conducting a successful postal survey is, however, to achieve a sufficient response rate to ensure that the survey results are representative.
Research at the Unit for Public Services Surveys at the German Post-Graduate School of Administrative Sciences in Speyer has highlighted that the majority of the employee surveys currently being conducted in German public agencies have a low response rate. Indeed, response rates of 20 -30 per cent of distributed questionnaires are often judged as sufficient, and, in some cases, even considered as a proof of success. Often control measures to assess the level of representativeness are even dispensed with, sometimes with full knowledge that the subsequent survey results may be unreliable and may not apply to the totality of employees. Consequently, public agencies run the risk of collecting and reporting false information. This suggests a high degree of carelessness and a disregard for whether staff survey results are really representative.
Practical tip: In general, the response rate for an employee survey should never be below 50 per cent. In order to use the staff survey as an instrument for raising employee motivation and involvement in public service redesign, a minimum of 70 per cent is required. Even with a response rate above 50 per cent it is always advisable to assess how representative the responses have been, e.g. by comparing the composition of the survey participants with the totality of employees in terms of socio-demographic characteristics such as age, sex, and career profile and by monitoring the response rates between organisational units (departments, divisions, etc.).
Another quality problem in staff surveys stems from the poor quality of the answers, which are often due to poorly designed questions within the questionnaires.
Typically, the appropriate wording, structuring and lay-out of questions is a more challenging task than is assumed by inexperienced local authority staff. The most common mistakes are:
In addition to badly worded individual questions, questionnaires are often too lengthy, with too many questions packed in.
In other cases, public agencies follow the guiding principle that questionnaires should not become too comprehensive but they then mistake this rule of thumb for a need to make questionnaires "as short as possible". It is reasonable to assume that the employer as well as staff may be ready to sacrifice a significant amount of time (perhaps up to 1 - 1.5 hours per respondent) for the questionnaire to be completed during working hours. If the average completion time for a questionnaire is reduced to half an hour, one literally throws away opportunities to gather valuable information. It is also wrong to believe that short answering times help to make the survey appear less intrusive from an employee's perspective or help to gain the acceptance of staff. On the contrary, demotivation effects are more likely when the employees are given the impression that they cannot voice many issues they want to "get out of their hair". In this case, the survey may become suspected of being an inconsequential charade or a public relations ploy. As a result, employees may be less wiling to participate, or to take the results seriously, which results in reduced response rates and other negative consequences.
Practical Tip: When choosing the ideal questionnaire length, two factors must be considered and balanced against each other:
Questionnaires often give the impression that the questionnaire designers had no idea about the scope of the quantity and quality of information which they could get if they exploited the full potential of the survey approach. In particular, it has become popular for employee surveys to focus particularly on questions about employee satisfaction.
However, the "human condition" of employees cannot appropriately be assessed simply through a set of undifferentiated questions about their degree of satisfaction. Such a simplistic approach does not take into account different types of satisfaction, which, for example, can include resigned acceptance of unsatisfactory experiences and payoffs from work. The latter occurs when public service employees have come to terms with the inevitability of existing circumstances, regardless of their own aspirations, needs and motivations, etc. It is important to understand that satisfaction and motivation are two completely different issues. Frustrated employees may have lowered their expectations of their workplace so much that they are still satisfied but they will not be motivated any more. Questionnaires with an exclusive focus on employee satisfaction also miss the opportunity to get objective data on the implicit willingness and ability of employees to perform better.
Furthermore, employee surveys often fail to address employees as highly knowledgeable experts in their specific work situations - which in itself can act to demoralise and demotivate staff.
Practical tip: When designing a questionnaire, it is necessary to be clear about the purpose of the survey and the type of analysis which will be carried out on the data. It is also important to have in-depth knowledge about the formal and informal rules and processes in the local authority. In addition to questions about employee satisfaction the questionnaire should always address the following types of issues:
Short questionnaires are usually limited in their potential to help with the analysis of complex issues. For example, the evaluation of the quality of leadership in a local authority may be limited to a single question ("How satisfied are you with your boss?"). In this case, there is no possibility of obtaining a differentiated evaluation of leadership which allows the identification of weak points in the behavior of superiors on the basis of a range of behaviourial characteristics. Furthermore, this kind of generalised question produces only an average picture. As a result, wrong conclusions may be drawn from such results, e.g. that the leadership situation (or any other characteristic of the local authority) is "in general, satisfactory".
Employee surveys with badly designed questions may even become counterproductive when distorted results contribute to a false sense of security in a local authority (or within an organisational unit of the local authority), which in reality may possibly be in need of drastic and decisive quality improvement.
Practical tip: A well-designed questionnaire must address complex issues in a local authority by using consistent sets of questions which produce enough detail to provide a differentiated picture. This is a necessary precondition for providing proper guidance to on how to shape appropriate improvement initiatives.
There are a number of areas in relation to data processing which need special care if the reported results of employee surveys are to be valid.
The survey results will often tend to have a "the more - the better" structure. For example, one may find that demotivation of employees appears to increase, the more pronounced are certain problematic characteristics of managers' behaviour. However, with the right statistical analysis, it may be possible to go further - for example, to show that the tendency to feel demotivated is statistically more closely related to specific patterns of managers' behaviour in certain functions than to the general organisational culture.
The analysis of statistical relationships reveals valuable insights into the relative importance of negative and positive factors in organisational performance. Knowledge about these factors is of great importance for strategic decisions about prioritising service improvement plans. In order to use statistical analysis properly, it is essential that the questionnaire contains useable scales.
The presentation of survey results is often unsatisfactory because particular distributions of answers may be arbitrarily classified as "high", "low", "good", or "bad". When, for example, 53 per cent of the employees are "satisfied" with their superior -- is this a good or bad result? In many German surveys, arbitrary principles seem to be applied - scores which in one employee survey are presented as positive may be presented in another as negative.
Practical tip: The evaluation of results must not be conducted according to "fancy", but rather be governed by rules which are set in advance, in line with the strategy and performance targets of the organisation. For example, if a target was set at 80 per cent, a result of 53 per cent will be judged as too low. Defining expectation levels or quantitative targets ex-ante considerably helps to prevent evaluation problems.
Survey results from other public agencies may provide useful benchmarks which help the authorities to assess their performance in a realistic way. Although in German public agencies external benchmarking has already become quite widespread, there is also the possibility of internal benchmarking between the departments, divisions and other organisational units of a local authority. In practice, however, there is usually considerable resistance from the middle-management towards various forms of benchmarking - whether external or internal. It is important to make clear that information about weak points in the organisation will not be used to victimise those whom the management had wanted to get rid of anyway but rather as an opportunity for the implementation of effective improvements.
![Sigma [logo]](../gifs/sigma.jpg)