This is such an important question, and it faces many of our parachurch organizations most intensely, although youth pastors and pastors in general also struggle with it.
Back in the days (days which are still around for many people) when the goal was to register as many "decisions" as possible, it was easy to give statistics to validate success. How many people walked an aisle, filled out a contact card, prayed the prayer at the end of the booklet, raised a hand, or whatever? If you could report big numbers, you could raise big money. Some organizations even used terms (unofficially) like "bang for the buck" and talked in terms of ... X dollars = Y souls saved, arguing that they were the most efficient soul-saving machine out there.
The results of that system were - evangelastic numbers (inflated with some close to if not well over the line of dishonesty), conversion without discipleship, giving false assurance to people who had no real conversion but raised their hand or whatever, punishment of groups that worked slow and deep rather than fast and shallow, etc.
But what's the alternative? What measures success?
I think a first step in the right direction was to measure church growth rather than "decisions." But even here, we can end up with big churches which have little effect outside their doors. The goal becomes attracting people to church (and thereby sucking them out of their communities) rather seeing God's will done on earth - in the community, in the city, in the public schools, in the halls of justice, among the poor, in the environment.
So, another step in the right direction would be to count the number of people we deploy into the world. However, probably the most important way people are deployed is simply to live their faith in their daily work, among their daily neighbors, etc. That's hard to count!
I was hoping I'd come up with a clever solution by the end of these few paragraphs, but I think the best I can offer is this: wise boards like yours need to have this conversation: what does success look like, and what evidence can we look for that it's actually happening?
A firm believer in boards and their role in governance and accountability, I know this is a tough question, but the labor expended in wrestling with it will make the board a better board, the CEO a better leader, and the organization a better organization.
It might result in some organizations going out of business too, which isn't always a bad thing, while others may significantly realign their work and redefine their mission. What percentage of our organizations are succeeding at missions that aren't really worth doing - at least not when compared to other things they could be accomplishing? How many dollars are funding ill-conceived mission plans?
I wrote about this at some length in the chapter on missions in Church on the Other Side. My guess is that people like yourself, who wrestle further with this important question, will help the rest of us by publishing your experiences. I wish I could offer more! - Brian