Last week I had a great opportunity to give a talk at the CES-BCY 2015 Evaluation Conference on the topic of engaging service delivery staff in evaluation, specifically in the context of non-profits. The talk was well attended and followed up with several interesting questions. More than one person also told me they wanted to attend the presentation but couldn't because of other talks scheduled at the same time.
So for people who weren't able to attend either the talk or the conference itself, I have uploaded the slides (link opens PDF file). Because the slides weren't meant to stand alone and may be a little cryptic without context, I'm also recapping my talk here on the blog.
Key Concepts
1) Frontline staff refers to the particular stakeholder group responsible for carrying out the services and activities of the organization, typically working directly with clients or other program beneficiaries. According to a recent labour market survey of non-profits in BC (link opens PDF file), service delivery staff account for a substantial 81% of employees in this sector. (Thanks to Don Ference and Brian Bauer of Ference Weicker & Company, who presented on these findings directly before I gave my talk!)
When it comes to determining an organization's agenda or deciding what services it will offer and how (i.e., organizational power), frontline staff typically have less say than do the organizational decision-makers, such as upper management, executive directors, boards of directors, etc. At the same time, service delivery staff usually have more say over how their programs run than their clients or other program beneficiaries do. But where these groups do not differ is in their organizational importance--each is equally significant to the mission and operation of an organization and therefore each is also significant to the evaluation. For instance, although frontline workers don't get the official final say over what happens in their programs, they are still ultimately responsible for implementing the services and can greatly impact the quality with which evaluators' recommendations and decision-makers' plans are carried out.
2) Engagement refers to a combination of two factors: active participation and emotional investment.
a) Active participation means taking an active role in and contributing directly to the evaluation. There are many possible roles, though different stakeholder groups tend to be associated with particular ones. To briefly and non-exhaustively sum up some common roles:
- Planner: Setting evaluative goals and priorities, identifying and selecting appropriate evaluation methodologies.
- Participant/data contributor: Participating in surveys, interviews, and focus groups; generating program data; facilitating access to other stakeholder groups for the purpose of data collection (e.g., program recipients).
- User: Being an intended user of the evaluation findings, reviewing deliverables and recommendations and making decisions about what steps to take based on this information.
- Implementer: Carrying out recommendations and any changes in operations or service delivery as a result of evaluation findings.
Frontline staff are most often associated with being participants in evaluations. They are also nearly always implementers, whether explicitly identified as such or not. But designing evaluation projects to include frontline workers in planning activities and crafting deliverables to be accessible and meaningful to staff can increase engagement. That is, assuming they also have:
b) Emotional investment or "buy-in". Buy-in means believing in the value of an initiative (in this case, an evaluation), wanting to see it succeed, and having the will and enthusiasm to contribute directly to its success if needed. Staff buy-in is frequently referenced as being vital to the success of programs. Staff who believe in their programs are more likely to put in the necessary effort and time for quality service delivery. The same logic extends to their participation in evaluation! A staff member who participates in an evaluation without emotional investment will not be fully engaged and the quality and depth of their contributions will reflect this.
Program buy-in and evaluation buy-in are not interchangeable. A frontline worker can believe in the value of their program or the strength of their organization without seeing the significance of evaluation to either of these. It's up to evaluators to bridge this gap and communicate the potential contribution of evaluation to program and organizational success.
Rationale
Why frontline staff? All stakeholder groups are important. It's not a competition or a question of whose engagement an evaluator should prioritize. But discussions of stakeholder engagement tend not to differentiate between stakeholder groups when talking about engagement strategies, even though the needs and considerations of a board of directors are different from those of a service team and different again for a client group.
The reason for focusing specifically on frontline staff here is personal. Frontline staff have often been integral to the success of my evaluations. Their insights have allowed me to generate more informed and relevant findings, which in turn has increased the likelihood that the decision-makers would act on these findings. Overall their ability and desire to get engaged was a strong sign of the organizational capacity and readiness for evaluation.
When you engage frontline staff in your evaluations, you:
- Include a unique program perspective which is both internal (unlike the typical program recipient's perspective) as well as focused on the details of day-to-day happenings in the program (unlike the typical high-level decision-maker's perspective)
- Encourage staff to be allies instead of gatekeepers or obstacles to be overcome. This is important when it comes to learning more about a program, accessing data or information which they control, or enlisting their assistance in carrying out evaluation activities (e.g., collecting data from program beneficiaries)
- Promote quality implementation of evaluation recommendations by individuals who had meaningful input into the evaluation process as well as an opportunity to understand the priorities and interests of other stakeholder groups involved
Frontline staff can also benefit from being engaged in evaluation activities, as illustrated by the quotes to the right. These quotes came from interviews with frontline staff I've worked with on prior projects.
There are challenges to pursuing frontline staff engagement too. Like any relationship-building process, engagement is time-consuming, it can be impacted by many factors outside of the evaluator's control, and engaging multiple stakeholder groups at a time always carries the risk of encountering complex power dynamics and internal organizational politics. These challenges do not mean that evaluators should shy away from pursuing frontline staff involvement, but should be aware that a thoughtful approach is needed for success.
The final section outlines specific strategies for enhancing frontline staff engagement.
"It gave me insight into the program, the data itself. I was able to reflect and it gave me insight into what we do and how we can do it better and what we are doing and if it's working."
"Being involved in the evaluation fostered a deeper thinking of some of the aspects of this work that I could have filed away in my mind."
Strategies for Engagement
1) Start with relationships
Engagement is about building a relationship between the staff and the evaluation and evaluator. Like all relationships, engagement requires time, patience, and trust.
Start the process with a face-to-face meeting. Even if you never see them in person again (for example, if they work in rural or remote locations and the travel budget is limited), having at least one in-person meeting gives you a chance to lay a strong foundation for future interactions. For internal evaluators who have pre-existing and on-going relationships with staff, starting with a face-to-face check-in is still a good way to establish engagement with that specific evaluation project, though you may skip the personal introductions.
A group meeting will allow you to introduce yourself and the evaluation project all at once. It also gives you the opportunity to observe the staff interact with each other and develop a sense of the relational context, something isolated interviews won't provide. Even if you aren't formally collecting data, you will learn much through observation about the dynamics of the group you're working with.
Clearly explain your role and the goals of the evaluation and allow time for discussion. Be prepared for challenging questions and treat these as mutual opportunities to learn about each other, the organization, and the evaluation. Staff can vary widely in their levels of interest in and enthusiasm for evaluation. Identifying their concerns and expectations early on will allow you to address these constructively through the evaluation process. Be honest about the limitations of evaluation and encourage them to think critically about how they would go about demonstrating the value of their program.
Include an open-ended question about each staff person's past experience with evaluation to assess relative knowledge and familiarity. Don't make assumptions about expertise! You may be surprised with the depth of previous experience some staff with have in evaluation. But be prepared to make your explanations accessible to evaluation novices as well.
Strongly encourage administrators/program managers to be visibly involved early on. Their presence will establish the legitimacy of the evaluation. There should also be consensus on issues like the purpose and intended use of the evaluation and the degree of frontline staff involvement expected by program administrators before staff are brought into the process. This will help avoid confusion and disorganization.
"The fact of being involved every step of the way helps you become more a part of it than if it's just, 'here, this is what you have to do'."
"The input of the worker is crucial. Have them feel as part of the process. That it's not just top-down."
"You being on the other end of the phone and not making me feel like an idiot and being able to take our questions and change it and make it work."
2) Build trust
After making a good first impression as someone who is approachable, professional, and a skilled communicator, and having laid the groundwork for why evaluation is important and useful, build on this by being responsive, reliable, and trustworthy in your actions. Staff who trust you to meet your obligations and do your job well will be more likely to persist when the evaluation gets challenging.
For example, when asking frontline staff to collect program data for me on their services, I was available for technical support in case they ran into difficulty with the tools I created for them. Inevitably problems did come up, and several times staff called or emailed me with questions or issues. Afterward, when I asked these workers what helped them stay committed to the process, all of them mentioned how important that fast and reliable support was, how otherwise they might have put the task to the side and forgotten about it entirely, and how it built their confidence and trust. The tools themselves were also designed to be user-friendly and were modified based on feedback to make them even more accessible, which was appreciated by the staff (and good learning for me!).
The other popular sentiment staff expressed was the importance of time. While part of fostering engagement is encouraging active participation by staff, it's also important to respect their workload and the constraints on their time. Do not schedule unnecessary meetings or expect them to participate in every aspect of the evaluation. If you are asking them to collect data, stick to the essentials and communicate clearly why that particular information is necessary. When scheduling interviews, be accommodating and flexible. Program administrators should support their staff in allocating a reasonable amount of time to the evaluation in conjunction with their other responsibilities. If this is not the case, work with the administrators to find a solution. Time is a valuable resource and staff should be able to trust that theirs will be respected in the evaluation process.
3) Be outside the hierarchy
This strategy is not specific to frontline staff engagement. Evaluators tend to exist adjacent to the power structures of an organization anyway, having influence but not direct control. Even internal evaluators can maintain this type of distance if their role is clearly delineated and their boundaries well-managed. As an evaluator, it's useful to be able to engage with every stakeholder group as an equal, rather than a superior or a subordinate, and not as the exclusive ally of a single stakeholder group.
As mentioned above, having the visible support of upper management will convey the legitimacy of the evaluation activities to staff and underscore the evaluator's ability to convey their concerns up the chain. But while the evaluator does not want to appear irrelevant, if the evaluation is seen as a mere extension of a program administrator's authority, then the responses of frontline staff may be more guarded. Positioning oneself as a helpful third-party who is there to facilitate communication (in more than one direction) can help as long as everyone is on board with this definition, including the evaluator and the decision-makers who brought you in.
If there is conflict between (or within) stakeholder groups, rather than let the evaluation become a conduit for power struggles, it can be more productive to frame this as an unexpected outcome of the evaluation and encourage the conflicted parties to work to address and resolve the issue together. If there is sufficient trust (and time) and the issue is directly related to the evaluation itself, the evaluator may mediate, or may step back and allow internal processes to take care of it.
"It is a team approach rather than a 'must do'. It gets buy in and participation willingly from all players."
"We get to actually see what it is that we do. […] This really goes to show, holy, we do a lot of work with these kids. It puts it on paper and helps."
4) Demonstrate impact
This does not refer to program impact, but rather the impact of the staff on the evaluation and the impact of the evaluation on the staff.
Make sure that contributions made by frontline workers are highlighted and recognized in a meaningful way. Putting an acknowledgement in a report that is never circulated to frontline staff does not constitute meaningful recognition. One example from my own experience was when we invited staff to the board meeting where we presented our evaluation findings. The staff had spent a great deal of time collecting data that year and afterward expressed how rewarding it was to see their efforts come to fruition and have their hard work acknowledged. This reassured them that the evaluation process was meaningful and that their time had been well spent.
On the flip side, it's equally useful for staff to recognize the impact of evaluation on themselves. When staff see either the processes or the outcomes of evaluation as being beneficial to them and their organization, this will increase their support and willingness to get involved. To that end, it can be helpful as an evaluator to identify the evaluative needs the frontline staff as well as those of the program administrators in order to try to meet both, if possible. Sometimes it can be as simple as providing specific feedback to frontline staff on an issue relevant to them or framing findings in terms of actions that can be taken at both the macro/administrative level and the micro/individual staff person level. When we collected data about program activities, administrators were pleased to be able to have a big-picture view to present to funders, while frontline staff appreciated seeing their site-specific results.
5) Remember their priorities
In the BC non-profit labour market survey referenced above, one of the key attractors for people working in non-profit organizations is their ability to 'make a difference' in their jobs. This commitment to the mission and purpose of their work is a distinguishing characteristic of many people working in non-profits. Whenever people care about their programs and its goals, this can translate into caring about an evaluation if it aligns with their goal of making a positive change.
Different approaches may be needed to make this link between evaluation and staff's own goals. Some staff may be of an evaluative mindset themselves and will appreciate efforts to verify that their work is effective or feel that evaluation legitimizes their program. Others may feel that evaluation isn't strictly necessary, but will respect the reality of funding requirements. Staff don't have to love every aspect of evaluation in order to appreciate its benefits. It is always more difficult when evaluative findings are seen as threatening the status of the program (and it should be acknowledged early on that evaluation findings are not always positive and often point to areas for growth and change), but in this case it is best to focus on the goals of the program rather than the program itself (e.g., is change happening, are clients' needs being met, are limited resources being deployed to greatest effect?).
This factor is the most critical when time is short or the task required is burdensome and other strategies for engagement are lacking. In situations where there has been little time to build meaningful relationships with staff prior to interviewing them, the passion of frontline staff for their work is usually the biggest determiner of the quality and depth of the information they share, even with a stranger who has just introduced themselves as an evaluator. This includes staff who are passionately frustrated with what they perceive as limitations or challenges in their service delivery and who want to see things done better, so it is not a question of emotional investment discouraging them from being critical or honest about their services.
Crucially, this is not a factor that the evaluator can control, only build on, which leads to the final point...
"The data itself that came out of it is very valuable and it does affect the way I do my job."
"It's just a pain in the ass and time-consuming and not my favourite thing, but I understand the importance."
6) Recognize structural problems
This final comment is less of a strategy and more of a warning and gentle reminder. Engagement is about relationships and not all relationships succeed. Within an organization, there may be many barriers to fostering strong engagement with frontline staff, including high turnover rates, burn-out among staff, competing priorities and insufficient resources, or a lack of support for staff engagement from program administrators and management. Some staff members may never be open to engaging with the evaluation process regardless of the evaluator's best efforts. A relationship can't be forced.
When engagement is a priority, it is critical to work with program administrators and management to ensure internal support for frontline staff participation (e.g., allocating work time, balancing priorities, communicating the significance of the evaluation process to the organization). However, while frontline staff engagement can be greatly beneficial to an evaluation, it is not always a realistic prospect, at least not on a wide scale, depending on these other factors. When staff engagement is desirable but not achieved, it may be worth highlighting the reasons why in the subsequent report as it may speak to an area for development in the organization's evaluative capacity.