Generalized Impact Evaluators

Abstract

Existing funding systems fail to sufficiently fund public goods and common goods due to insufficient mechanisms for coordinating various agents towards valuable outcomes. Relative to traditional capital systems that scalably organize activity around maximizing financial performance, impact funding remains underdeveloped, especially in the ability to reward high-upside, high-uncertainty work. Here, we propose Impact Evaluators (IEs) as a modular system for coordinating work by measuring, evaluating, and retrospectively rewarding the impact achieved towards specified valuable objectives. We present a structure to define Impact Evaluators as well as design schematics to facilitate their implementation. We then discuss implementation considerations, practical learnings from past experiments, and integration with the broader ecosystem of public goods and commons funding systems.

Content

1. Introduction
2. Designing an Impact Evaluator
Scope (S)
Measurement function (m)
Evaluation function (e)
Reward function (r)
Summary
3. Impact Evaluators in practice
Emerging paradigms
General operational considerations
Case studies
4. Interoperability with other funding mechanisms
Prospective funding programs and cash flow
Composability with other Impact Evaluators
Hypercerts and interoperable funding systems
5. Conclusion

1. Introduction

Purpose of this document

The goal of this document is to accelerate the development of Impact Evaluator mechanisms by providing:
● Standard language to define and construct Impact Evaluators
● A framework to efficiently implement Impact Evaluators

What is an Impact Evaluator

An Impact Evaluator (IE) is a system or mechanism that coordinates groups of agents to work towards objectives by assessing impact against those objectives to retrospectively reward valuable work or outcomes (Benet, 2022a). An effective IE creates an incentive structure that guides potential contributors to cooperatively pursue the specific objectives.

Why Impact Evaluators are important

Impact Evaluators can provide incentive or market mechanisms for highly effective “impact funding” (i.e. provision of public goods) that manifests in two major needs:

Evaluating value creation: At its simplest, an Impact Evaluator is an important function to indicate what was valuable and distribute a suitable reward (noting “valuable” is subjective). In traditional market scenarios this could be used to indicate an “ROI” on deployed resources, which is often the primary key performance indicator (KPI) to determine value and define future allocations of capital or labor. This feedback signal is underdeveloped in impact markets and Impact Evaluators address this.

Reward high-potential/high-uncertainty work: Existing funding systems, particularly in public goods, typically prospectively fund projects that have clear paths to outcomes. This funding is also typically based on present and future cash flows. High uncertainty projects with high upside, particularly those without easily projected cash flows, appear to have a gap in funding systems. This manifests as missing incentives for the best talent to pursue these projects.

While we explore Impact Evaluators for “impact funding,” this mechanism can be very powerful for incentivizing uncertain tasks in general where typical financial methods fall short.

2. Designing an Impact Evaluator

To build intuition around Impact Evaluators, we will use an illustrative example throughout Section 2 in addition to the core theory. Note to the reader that our model systems used to explore IEs will draw heavily from open source software and blockchain experience, but that does not limit broader applicability.

Illustrative example: The WidgetMakers community

Suppose there exists a group of individuals that creates widgets, The WidgetMakers. These widgets are a classic “public good” - they are not sold, but many people in the broader community use them and find them valuable. The WidgetMakers have received a generous donation in the form of a $2M USD grant, and now want to grow the impact of their community, which they want to accomplish by bringing in new members. They allocate 50% through prospective funding over 10 years (e.g. referral fee, onboarding grants) but these options do not align incentives of new members with the success of the WidgetMaker community - this can lead to bad actors that will take funding but not perform as valuable members of the community.

To counteract this, WidgetMakers announces it will reward the remaining 50% to “valuable members” with a bonus based on the value of their contributions each year, starting with the previous year. This incentivizes existing members to consistently contribute and more members to join the community. Setting up this mechanism will require the ecosystem to:
● Define their Objective → intermediate goal of “more valuable members” in their community
● Define the Scope → which community members are eligible and the eligible indicators (actions / outcomes) counting toward the objective
● Define an Interval to evaluate the value created (in this case the last year)
● Measure the indicators (actions / outcomes) during the interval that provide the most information on value created, and identify entities (members) that contributed
● Evaluate the indicators measured and attribute to the entities (members) responsible
● Disburse an appropriate Reward to incentivize more valuable outcomes
● Apply this measure → evaluate → reward process repeatedly for future intervals

We will now develop each step in detail as we arrive at a formal definition of Impact Evaluators. More examples of IE design in practice are found in section 3.

Impact Evaluators: The Formal Definition

An Impact evaluator can be generally described by the tuple of objects:

𝐼𝑚𝑝𝑎𝑐𝑡 𝐸𝑣𝑎𝑙𝑢𝑎𝑡𝑜𝑟 𝐼𝐸 = {𝑟, 𝑒, 𝑚, 𝑆}
where
r is the reward function,
e is the evaluation function,
m is the measurement function on S, and
S is the scope of the IE.
The scope, S, constrains the domain of all past actions, outcomes, and entities to the subset that will be considered for the specific IE. The ordered application of these functions is used to compute the reward allocated to each entity. Additionally, all activities can be repeated in additional epochs or intervals, where i is the interval (time or other discrete milestone measuring the IE cadence).

This (simplified) ordered application yields the reward disbursement

$$ R = r(e(m(S))) $$ for which the IE is engineered to incentivize a set of desired objectives (O) within a scope S. The goal of an impact evaluator is that, over time, the value of objectives (O) achieved are greater than the value of resource inputs (operational cost + incentive rewards).

The remainder of Section 2 will expand on the individual components of an IE, concluding with a restatement of the reward disbursement that elucidates relative allocation between entities rewarded by IE.

Scope (S)

We start with the set of actions taken by entities (individuals or organizations) over all time and the outcomes of those actions. Within this set, we use Scope 𝑆 to identify a subset that is intended to be measured, evaluated, and incentivized by the impact evaluator.
This subset could be constrained by any boolean constraints on this set, including but not limited to: temporal constraints on the action or outcome (e.g. specifying hours or months), type of action (e.g. specifying a project name or type of activity), location (e.g. the amazon rainforest or github.com/ipfs/kubo), or any other designation.
To convey our intuition, we suggest but do not intend to formalize or prove the following
● This set could be considered a space by adding the additional metric where two actions are considered “close” if they have overlapping impact (or at least logically correlate with the same or similar, likely counterfactual, outcomes).
● We expect that all practical scopes will be convex subsets of this space.

Measurement function (m)

The measurement function takes a scope, S, as input, and outputs a set of ordered pairs describing indicators within the scope and the entities that contributed to those indicators. Namely,

$$ m(S_{i}) = ext{{ { (indicator{1}, entity_{1}), (indicator_{2}, entity_{2}), ... } }} $$ Where indicators can be attributable to one or more entities.
The indicators for each interval should be the direct result of activities/outcomes within the scope of that interval (potentially building off of activities within previous intervals).
Practically speaking: This step is highly non-trivial, requiring systems to objectively capture the state of 𝑆 (entities, indicators contributing value during interval i). It’s hard to perfectly specify a single comprehensive indicator of impact, so a combination of intermediate activities/metrics/outcomes are typically required.

Evaluation function (e)

The evaluation function combines the outputs of the measurement function and one or more evaluators to ascribe some impact to the various outputs. This converts the set of outputs into a measure of value and attributes it to the entities that created it.

$$ eig( ext{{ { (indicator, entity_{i}), evaluator ext{ } } }} ig) = ext{{ { (entity, eval score)_{i} }} $$
The output is a vector of entities and their respective evaluation score(s).

Reward function (r)

Once we have a completed evaluation, indicating a magnitude of value generated by each entity, we can convert the evaluation into a suitable reward in our reserve resource using a reward function.

$$ Rewards R = r igg( ext{{ { (entity, evalscore) }}, reserve reward igg) = ext{{ { (entity, reward) }}} $$

3. Impact Evaluators in practice

In this section we translate theory to practice by observing emerging paradigms for IEs, walking through considerations for operationalizing an IE, and discussing case studies which have informed our early perspective.

Emerging paradigms

Through our experiments, we’ve seen a few patterns forming with IE applications. We will introduce here then explore a few key considerations and examples below:

General operational considerations

IEs can be constructed in a variety of ways, enabling different degrees of freedom within the system to match the solution required.
Human (subjective) vs. automation (quantitative) tradeoffs
IE designs can range from computational protocols to voting mechanisms leveraging subject matter experts, depending on the solution desired. This comes with tradeoffs between:

1. Up-front setup cost to design and deploy the IE
2. Ongoing operational cost of experts and other human administration
3. Adaptability of an IE to adjust to unforeseen circumstances and effects

Reward function design

We will start by recalling the major needs that IEs address from Section 1:
● Evaluate value creation
● Reward high-potential/high-uncertainty work

4. Interoperability with other funding mechanisms

IEs are a powerful generalizable funding mechanism but not a “one-size fits all” approach to funding public goods and high uncertainty projects. This is particularly relevant during early IE rounds when rewards have not been realized or distributed.

5. Conclusion

Impact Evaluators are an evolving concept that has spawned out of conversations in the web3 public goods/regenerative economics space. This paper is an early attempt to bring more structure to the discussion around this mechanism and share the knowledge from early experiments and learning.