diff --git a/content/practitioner guides/practitioner-guide-demonstrating-org-value.md b/content/practitioner guides/practitioner-guide-demonstrating-org-value.md index 94da4e3..18b3a49 100644 --- a/content/practitioner guides/practitioner-guide-demonstrating-org-value.md +++ b/content/practitioner guides/practitioner-guide-demonstrating-org-value.md @@ -5,14 +5,12 @@ type: page date: 2025-07-22T15:25:31+00:00 ppma_authors_name: - elizabeth - - elizabeth nectar-metabox-portfolio-display: - all nectar-metabox-portfolio-display-sortable: - off url: /practitioner-guide-demonstrating-org-value --- -\[vc\_row type=”in\_container” full\_screen\_row\_position=”middle” column\_margin=”default” column\_direction=”default” column\_direction\_tablet=”default” column\_direction\_phone=”default” scene\_position=”center” text\_color=”dark” text\_align=”left” row\_border\_radius=”none” row\_border\_radius\_applies=”bg” overlay\_strength=”0.3″ gradient\_direction=”left\_to\_right” shape\_divider\_position=”bottom” bg\_image\_animation=”none”\]\[vc\_column column\_padding=”no-extra-padding” column\_padding\_tablet=”inherit” column\_padding\_phone=”inherit” column\_padding\_position=”all” background\_color\_opacity=”1″ background\_hover\_color\_opacity=”1″ column\_shadow=”none” column\_border\_radius=”none” column\_link\_target=”\_self” gradient\_direction=”left\_to\_right” overlay\_strength=”0.3″ width=”1/1″ tablet\_width\_inherit=”default” tablet\_text\_alignment=”default” phone\_text\_alignment=”default” column\_border\_width=”none” column\_border\_style=”solid” bg\_image\_animation=”none”\] # Practitioner Guide: Demonstrating Organizational Value @@ -20,7 +18,7 @@ If you haven’t already read the [Practitioner Guide: Introduction - Things to # Audience / Scope -The audience for this guide is open source teams and Open Source Program Offices working within organizations where there is a need to demonstrate the value of the work and justify the resources being used for open source initiatives. +The audience for this guide is open source teams and Open Source Program Offices working within organizations where there is a need to demonstrate the value of the work and justify the resources being used for open source initiatives. # Introduction @@ -28,8 +26,8 @@ Many people contribute to open source projects as part of their employment. In a **A few notes about terminology:** - * This guide uses the term "**organization**" deliberately because demonstrating organizational value is important for a wide variety of different types of organizations (e.g., university / government OSPOs, research groups, corporations). - * Throughout this guide, the term “**goals**” is being used to describe what the organization is planning to achieve. Depending on the organization, the goals might be described as part of the vision, mission, or strategy. In the context of a company business unit, the goals might be part of a product line strategy or plan. These goals are a key component of demonstrating **value** to your organization because organizational leadership is more likely to see the value in work that is clearly aligned with the goals of the organization. +- This guide uses the term "**organization**" deliberately because demonstrating organizational value is important for a wide variety of different types of organizations (e.g., university / government OSPOs, research groups, corporations). +- Throughout this guide, the term “**goals**” is being used to describe what the organization is planning to achieve. Depending on the organization, the goals might be described as part of the vision, mission, or strategy. In the context of a company business unit, the goals might be part of a product line strategy or plan. These goals are a key component of demonstrating **value** to your organization because organizational leadership is more likely to see the value in work that is clearly aligned with the goals of the organization. # Challenges @@ -41,10 +39,10 @@ In other cases, open source teams provide an overview of what they use internall Another challenge for many organizations is that they have no overall open source strategy for their contributions. Employees are frequently encouraged to contribute to open source directly or indirectly without proper guidance about how to demonstrate the value of these efforts, and this can create a negative feedback loop: - * Employees are encouraged to contribute. - * The value is not understood, so leadership asks: “Why are we spending time on something that doesn’t help us?” - * Employees are told to spend less time on open source, creating the feeling of being unrecognized and undervalued, which leads to burnout. - * Both the open source project AND the organization begin to suffer. +- Employees are encouraged to contribute. +- The value is not understood, so leadership asks: “Why are we spending time on something that doesn’t help us?” +- Employees are told to spend less time on open source, creating the feeling of being unrecognized and undervalued, which leads to burnout. +- Both the open source project AND the organization begin to suffer. This negative feedback loop can be even more pronounced in organizations where there is no official open source team, and demonstrating the value of this work falls to unofficial teams made up of individual contributors who may not be in a position or have the time to justify this work. In some cases, these individuals might be contributing because an open source project needs bug fixes or features that would help with their job duties, but this isn’t seen as part of their official role. @@ -52,37 +50,39 @@ This negative feedback loop can be even more pronounced in organizations where t Leadership wants answers to the following questions: - * What is the criticality of the open source projects the organization is helping maintain? - * What is the organization getting out of contributing to open source projects? - * How much software engineering time is being spent on maintaining open source projects? +- What is the criticality of the open source projects the organization is helping maintain? +- What is the organization getting out of contributing to open source projects? +- How much software engineering time is being spent on maintaining open source projects? Creating an open source contribution strategy can help organizations frame their discussions with leadership to demonstrate the value of their open source efforts in ways that resonate with leadership. At a minimum, the open source strategy should contain details in the following areas, which are each addressed in the “How to Take Action” sections below: - * Supporting your organization’s **goals** - * Determining which open source projects are the most **critical** for your organization - * Assessing open source project **health risk** - * **Prioritizing** within your organization’s limited resources - * Measuring & framing **value** +- Supporting your organization’s **goals** +- Determining which open source projects are the most **critical** for your organization +- Assessing open source project **health risk** +- **Prioritizing** within your organization’s limited resources +- Measuring & framing **value** # How to Take Action -By creating a robust open source strategy that demonstrates the value of your organization’s open source efforts, these efforts and outcomes can be described in ways that resonate with leadership by clearly articulating how the work in open source helps your organization achieve its goals. This helps to frame the rest of the strategy that describes the influence of open source project health, how your organization’s resources are being used, and finally how the value is being measured. +By creating a robust open source strategy that demonstrates the value of your organization’s open source efforts, these efforts and outcomes can be described in ways that resonate with leadership by clearly articulating how the work in open source helps your organization achieve its goals. This helps to frame the rest of the strategy that describes the influence of open source project health, how your organization’s resources are being used, and finally how the value is being measured. ## Supporting your organization’s _goals_ Taking some time and effort to make sure that your open source strategy supports the overall organization’s goals makes it much easier to justify continuing organizationally-relevant open source projects. Explaining how open source contributions support the goals of the organization can help the executive team understand the strategic importance of this work. -Having a solid open source strategy is important because successful participation in open source requires a long-term commitment. Individuals and organizations spend months and years building trust and reputations within specific open source communities, and all of that hard work is wasted if those people keep getting pulled off of open source projects to work on internal efforts. Keeping people working on the open source projects that are critical to your organization requires planning and strategic thinking to prioritize these open source efforts and align them with the overall goals for your organization. +Having a solid open source strategy is important because successful participation in open source requires a long-term commitment. Individuals and organizations spend months and years building trust and reputations within specific open source communities, and all of that hard work is wasted if those people keep getting pulled off of open source projects to work on internal efforts. Keeping people working on the open source projects that are critical to your organization requires planning and strategic thinking to prioritize these open source efforts and align them with the overall goals for your organization. However, every organization has a limited number of people and other finite resources, so **_prioritizing_** your open source efforts can help you focus on the activities that demonstrate the most value to your organization and most directly help the organization achieve its goals. This prioritization helps to facilitate efficient use of limited resources. For this guide, priority is defined as a formula made up of criticality multiplied by health risk. -Priority = Criticality X Health Risk +
+Priority = Criticality X Health Risk +
**_Criticality_** is where these efforts align with the goals of your organization. By showcasing the criticality of certain open source projects for your organization, these critical projects can be used to frame the strategies in a way that leadership will understand how important it is to continue this work over the long-term to ensure that the company can achieve its goals. -**_Project health risk_** is important in this equation because some open source projects are healthier and more sustainable than others, so some projects are riskier and need more help. A healthy open source project that is critical to your organization is likely to be a lower priority than an equally critical, but more risky project that needs your organization’s help to become healthy and reduce the risk of using it. +**_Project health risk_** is important in this equation because some open source projects are healthier and more sustainable than others, so some projects are riskier and need more help. A healthy open source project that is critical to your organization is likely to be a lower priority than an equally critical, but more risky project that needs your organization’s help to become healthy and reduce the risk of using it. -Thus, the priority for investment (allocating staff or other resources) is a combination of criticality and project health risk that is unique to your organization. +Thus, the priority for investment (allocating staff or other resources) is a combination of criticality and project health risk that is unique to your organization. ## Determining which open source projects are the most _critical_ for your organization @@ -90,801 +90,344 @@ For all of the important open source projects used, it’s important to determin To better illustrate this concept, here are some example categories with descriptions and questions that might help determine criticality. -**Dependency**: Determine how your organization depends on this open source project. +**Dependency**: Determine how your organization depends on this open source project. - * Is it a core component to the function of your organization? - * Does it have high usage rates across the organization? - * What are downstream impacts to your organization’s projects, products, services, or teams? - * How difficult would it be to switch to something else? Or fork and maintain internally? And what would that cost be? - * What would the impact of an unplanned security event be? +- Is it a core component to the function of your organization? +- Does it have high usage rates across the organization? +- What are downstream impacts to your organization’s projects, products, services, or teams? +- How difficult would it be to switch to something else? Or fork and maintain internally? And what would that cost be? +- What would the impact of an unplanned security event be? **Opportunities**: There are opportunities to drive or influence the open source project to better suit your organization’s needs. - * Does it currently have features or initiatives on the roadmap that would be beneficial? If yes, how so? - * If it went in another direction, how would it affect your organization? - * What opportunities make it worth investing resources in? - * Could development create a competitive advantage? - * Does your organization want to be seen as a leader in this space? Or be strongly associated with it? +- Does it currently have features or initiatives on the roadmap that would be beneficial? If yes, how so? +- If it went in another direction, how would it affect your organization? +- What opportunities make it worth investing resources in? +- Could development create a competitive advantage? +- Does your organization want to be seen as a leader in this space? Or be strongly associated with it? **Supportability**: Determine your organization’s ability to support using the open source project. - * How easy is it to support? Is it well documented? - * How difficult is it to skill-up employees or backfill on expertise? - * Does it help reduce or manage costs (e.g. right-sizing pods)? If so, by what factor? - * Would using and committing resources to it be a better option than a vendored option? - * Does using it enable picking from multiple solutions and avoid vendor lock-in? +- How easy is it to support? Is it well documented? +- How difficult is it to skill-up employees or backfill on expertise? +- Does it help reduce or manage costs (e.g. right-sizing pods)? If so, by what factor? +- Would using and committing resources to it be a better option than a vendored option? +- Does using it enable picking from multiple solutions and avoid vendor lock-in? The answers to these questions can help you map each open source project on a criticality scale. Here are some examples that can be used as a starting point to assign criticality using a 1 (low) to 10 (high) scale, which you’ll see again in the prioritization section later in this guide: - - - - - - - - - - - - - - - - - - - - - - -
- Example Criticality Scale -
- High (8 - 10) - -
    -
  • - Critical to core functions -
  • -
  • - High usage rates across the organization -
  • -
  • - Extremely difficult to swap or maintain internally -
  • -
  • - Difficult to backfill expertise -
  • -
  • - Roadmap has features that would be very beneficial to organization -
  • -
-
- Moderate (4 - 7) - -
    -
  • - Software that supports core functionality -
  • -
  • - Could be swapped out with reasonable effort -
  • -
  • - Easy to backfill expertise -
  • -
-
- Low (1 - 3) - -
    -
  • - Non-essential tools/apps with minimal organizational impact -
  • -
  • - Could be swapped out with minimal effort (API compatible) -
  • -
-
+| Example Criticality Scale | | +| :--- | :--- | +| **High (8 - 10)** | | +| **Moderate (4 - 7)** | | +| **Low (1 - 3)** | | ## Assessing open source project _health risk_ -Now that the criticality has been determined, it’s time to think about the other part of the equation, open source project health risk (Priority = Criticality x Health Risk). +Now that the criticality has been determined, it’s time to think about the other part of the equation, open source project health risk (Priority = Criticality x Health Risk). There are many ways to think about open source project health risk, but it can help to start with a few things: - * [Quality codebase][4]: Sound architectural design, static code analysis, sufficient testing - * [Responsive][5] to issues / PRs: Time to first response / review - * Healthy [organizational participation][6]: No single vendor drives the whole project - * Has future plans and a roadmap: They are planning and have a design review process - * [Quality documentation][7] for [both users & contributors][8]: Docs are critical for both adoption and [contributor sustainability][9] and growth - * Good [security][10] and releasing controls: Do they have a history of triaging and resolving security issues, are they using/investigating supply chain security best practices +- [Quality codebase][4]: Sound architectural design, static code analysis, sufficient testing +- [Responsive][5] to issues / PRs: Time to first response / review +- Healthy [organizational participation][6]: No single vendor drives the whole project +- Has future plans and a roadmap: They are planning and have a design review process +- [Quality documentation][7] for [both users & contributors][8]: Docs are critical for both adoption and [contributor sustainability][9] and growth +- Good [security][10] and releasing controls: Do they have a history of triaging and resolving security issues, are they using/investigating supply chain security best practices -Similarly to criticality, the health risk of a project can be scored while also potentially looking even deeper at core areas that matter the most to your organization. If a project is unhealthy and it’s critical, it’s likely worth allocating resources to improve the project to bring it into a healthier state. +Similarly to criticality, the health risk of a project can be scored while also potentially looking even deeper at core areas that matter the most to your organization. If a project is unhealthy and it’s critical, it’s likely worth allocating resources to improve the project to bring it into a healthier state. -As an example, when the CNCF Helm project was found to have several project health concerns due to having a single maintainer, the risk was escalated to the CNCF. Jorge Castro, a CNCF staff member, got involved and helped the project to realign on a target ([the launch of helm v4][11]), as a focal point for contributor growth. This led to building a better contributor ladder, engagement with other projects, organizations, and vendors to invest and make the open source project more sustainable. +As an example, when the CNCF Helm project was found to have several project health concerns due to having a single maintainer, the risk was escalated to the CNCF. Jorge Castro, a CNCF staff member, got involved and helped the project to realign on a target ([the launch of helm v4][11]), as a focal point for contributor growth. This led to building a better contributor ladder, engagement with other projects, organizations, and vendors to invest and make the open source project more sustainable. ## _Prioritizing_ within your organization’s limited resources With insight on how to score both criticality and health, the following example combines them to set priority. The example uses two made up open source projects “Foo” and “Bar” with some information about their status, a rough scoring, and how that would map to establishing a priority. Assuming each section is worth 10 points, they’re scored with comments in each, and then the criticality score is multiplied by the health risk score. Some important notes: - * The three components of criticality (Dependency, Opportunity, and Supportability) are averaged to create a single criticality score, but using a weighted average might be a better solution depending on your organization’s needs. A higher score means that the project is more critical. - * Health is scored as health risk, which means that lower numbers are better, since a healthy project would have a low risk score. +- The three components of criticality (Dependency, Opportunity, and Supportability) are averaged to create a single criticality score, but using a weighted average might be a better solution depending on your organization’s needs. A higher score means that the project is more critical. +- Health is scored as health risk, which means that lower numbers are better, since a healthy project would have a low risk score. Example: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
- Project - - Dependency - - Opportunity - - Supportability - - Health Risk -
- Foo (50) - - Criticality: 8.3 (average of 10, 7, and 8) - - Health Risk: 6 -
-
    -
  • - Score: 10 -
  • -
  • - Critical to core functions -
  • -
  • - Migrating to an alternative would be extremely difficult -
  • -
-
-
    -
  • - Score: 7 -
  • -
  • - Roadmap has very useful features -
  • -
  • - High barrier of entry to contribute and will require more time to engage -
  • -
-
-
    -
  • - Score: 8 -
  • -
  • - Extremely difficult to backfill expertise -
  • -
  • - Not well documented internally -
  • -
-
-
    -
  • - Score: 6 -
  • -
  • - Project has a large contributor base, but few senior maintainers and contributor ladder needs work -
  • -
  • - Time to first response is too long -
  • -
-
- Bar (21) - - Criticality: 7 (average of 6, 9, and 6) - - Health Risk: 3 -
-
    -
  • - Score: 6 -
  • -
  • - Not critical to core function -
  • -
-
-
    -
  • - Score: 9 -
  • -
  • - Project is a better option than and easy to drive features -
  • -
  • - Popular, strong branding opportunity to be associated with it -
  • -
-
-
    -
  • - Score: 6 -
  • -
  • - Easy to backfill expertise -
  • -
  • - Could be swapped out with reasonable effort -
  • -
-
-
    -
  • - Score: 3 -
  • -
  • - Project has enough maintainers and a decent contributor pipeline. -
  • -
  • - Codebase has solid quality processes -
  • -
  • - Documentation could be improved. -
  • -
-
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ProjectDependencyOpportunitySupportabilityHealth Risk
Foo (50)Criticality: 8.3 (average of 10, 7, and 8)Health Risk: 6
+
    +
  • Score: 10
  • +
  • Critical to core functions
  • +
  • Migrating to an alternative would be extremely difficult
  • +
+
+
    +
  • Score: 7
  • +
  • Roadmap has very useful features
  • +
  • High barrier of entry to contribute and will require more time to engage
  • +
+
+
    +
  • Score: 8
  • +
  • Extremely difficult to backfill expertise
  • +
  • Not well documented internally
  • +
+
+
    +
  • Score: 6
  • +
  • Project has a large contributor base, but few senior maintainers and contributor ladder needs work
  • +
  • Time to first response is too long
  • +
+
Bar (21)Criticality: 7 (average of 6, 9, and 6)Health Risk: 3
+
    +
  • Score: 6
  • +
  • Not critical to core function
  • +
+
+
    +
  • Score: 9
  • +
  • Project is a better option than <Baz> and easy to drive features
  • +
  • Popular, strong branding opportunity to be associated with it
  • +
+
+
    +
  • Score: 6
  • +
  • Easy to backfill expertise
  • +
  • Could be swapped out with reasonable effort
  • +
+
+
    +
  • Score: 3
  • +
  • Project has enough maintainers and a decent contributor pipeline.
  • +
  • Codebase has solid quality processes
  • +
  • Documentation could be improved.
  • +
+
The final step in determining priority is mapping the resources your organization has to the open source projects that have just been assessed for criticality. The example in the table above shows that Foo has a priority score of 50, making it a higher priority than Bar with a priority score of 21. In this case, more resources should be allocated to Foo, but it might make sense to allocate some resources to both. It can help to think about this in terms of software engineers and software engineering hours, but if your organization can’t commit people, donating funds or hiring contractors to help in these areas might be helpful. -Priority = Criticality X Health Risk +
+Priority = Criticality X Health Risk +
## Measuring & Framing the _Value_ Now that the most critical open source projects have been identified, this information can be measured and framed in ways that resonate with the leadership team. One of the most common pitfalls is that organizations tend to bias towards easy to measure metrics such as total contributions, but these do not hold up under scrutiny. Metrics should support the goals and be able to be tied back to “value” and the types of resources being invested in the open source project: - * Are the features we’re interested in progressing? - * Are the issues we’re concerned about being addressed? - * Is stability being improved and bugs being fixed? - * Is the project itself healthy? +- Are the features we’re interested in progressing? +- Are the issues we’re concerned about being addressed? +- Is stability being improved and bugs being fixed? +- Is the project itself healthy? Ultimately, your organization needs to decide if it is this worth the resource investment vs. maintaining internally or accepting the risk, and it can help think about these across three focus areas: - * **Classifying Features & Initiatives**: How to determine and prioritize what features & initiatives to track - * **Issues & Change Requests**: Tracking the value of contributions to organizational goals - * **Overall Project Health**: Tracking overall open source project health and how it can benefit your organization +- **Classifying Features & Initiatives**: How to determine and prioritize what features & initiatives to track +- **Issues & Change Requests**: Tracking the value of contributions to organizational goals +- **Overall Project Health**: Tracking overall open source project health and how it can benefit your organization ### Classifying Features and Initiatives When investing or contributing to an open source project and that project has a roadmap, it can help to start with the features in a project and evaluate them to determine alignment with your organization’s needs. These classifications are useful when reporting about where people are allocated and why. - * **Invest**: Direct benefit to your organization that warrants allocating more staff to drive the feature or initiative. Frequently these are important features to organization implementation or initiatives deemed important to overall health of the open source project. - * **Watch**:Feature or initiative that has the potential to either be disruptive or beneficial depending on implementation. - * **Ignore**: No features or initiatives that are expected to benefit or impact how the project is used by your organization. +- **Invest**: Direct benefit to your organization that warrants allocating more staff to drive the feature or initiative. Frequently these are important features to organization implementation or initiatives deemed important to overall health of the open source project. +- **Watch**: Feature or initiative that has the potential to either be disruptive or beneficial depending on implementation. +- **Ignore**: No features or initiatives that are expected to benefit or impact how the project is used by your organization. ### Issues and Change Requests Bugs, security, and issues are useful because most people understand them without having to know a lot about the details of the open source project itself. These are a few example metrics that are not too difficult to get from GitHub and other code platforms. Note that within the CHAOSS project, we use the term Change Request (CR) to include pull requests (GitHub), merge requests (GitLab), patches, and similar processes for merging changes into a repository, since the different tools have different names for this. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- Type - - Metric - - Source -
- Bugs - - - - Bugs closed this year (GitHub Query):

- -

- is:issue closed:2024-01-01..2024-11-01 label:kind/bug

- Security - - - - Time to resolve security issues (GitHub cli tool): \
gh issue list –search “is:closed label:security” –json “id,openedAt, closedAt”
-
- Issue / Change Request - - - - GitHub queries can be tailored with author: keyword look at organization created items to track own items vs. project as a whole. \
\
Issues/PR queries can be tailored to track initiatives -
- -

- Overall Project Health -

- -

- Improving the project health for your organization’s critical open source projects is crucial for reducing the risk associated with using those projects. Within the CHAOSS project, we have Practitioner Guides and metrics that can help identify areas within those open source projects that aren’t as healthy as they could be and take steps to improve in those areas. This is not an exhaustive list, but it should provide a good starting point. -

- - - -

- There are also a number of other important roles in open source projects that can provide value to your organization and improve the health of a project. These are covered in more detail in the Appendix and include, but aren’t limited to: -

- - - -

- Conclusion -

- -

- To tie all of this back together, here is an example report for the 2 open source projects, Foo and Bar described earlier that can be delivered as a quarterly report. This report highlights the percentage of bugs and security issues fixed by others along with the average time to fix them. Below the table shows a breakdown of Software Engineer (SWE) investment, along with the notes related to the categories that were identified as important. -

- -

- Telling the story and communicating to leadership might include examples like these: -

- - - -

- Summary of a report communicating the value of contributions to Foo and Bar -

- -

- One guide can’t possibly cover everything about demonstrating the value of your organization’s open source efforts, so this is just a fraction of the potential ways that open source investments can be tracked and tied back to goals while effectively describing the value of the contributions. It requires the right framing and ensuring that resources are allocated where they may have the most value; however, the framing and resource allocation will be different in almost every situation, since it depends on your organization’s unique needs. -

- -

- Cautions and Considerations -

- - - -

- Additional Reading -

- - - -

- Contributors -

- -

- The following people contributed to this guide: -

- - - -

- References -

- - - -

- CHAOSS Practitioner Guides are MIT licensed, living documents, and we welcome your feedback and input. You can suggest edits to this document at https://github.com/chaoss/wg-data-science/blob/main/practitioner-guides/demonstrating-org-value.md. -

- -

- Appendix -

- -

- There are a number of important roles in open source projects that can provide value to your organization and improve the health of a project. These include, but aren’t limited to: -

- - - -

- Triage and Program Management -

- -

- Having a product manager or program manager can help cut down on engineering time spent on things that aren’t really engineering efforts and that engineers might not be able to as efficiently as someone with expertise managing products and programs. Doing triage can provide awareness of open source project activities and issues that are important to your organization much earlier than if you were not involved. By reviewing incoming issues, assigning labels and priorities, and aiding in roadmap planning, your organization gets a number of benefits: -

- - - -

- Key Metrics: -

- - - -

- Long Term Health Metrics: -

- - - -

- Documentation -

- -

- Many people rely on an open source project’s documentation when getting started, and good documentation can help reduce support requests for common questions or issues that can be headed off with good documentation. In particular, a well documented contributor guide can help with onboarding new people and creating a good start for a contributor funnel. They can increase adoption and general overall positive sentiment with improved site traffic, search engine findability, and other web statistics. Contributors rarely want to do the big initial heavy lift to create documentation from scratch, but if a project has good docs to start, folks are much more open to keeping them updated. “Having good docs leads to better docs.” Well-documented open source projects are easier for users to understand and utilize, thus aiding in contributor sustainability, and your organization gets the following benefits: -

- - - -

- Metrics: -

- - - -

- Long Term Health Metrics: -

- - - -

- Communications and Marketing -

- -

- Open source projects often lack both the knowledge and skills regarding communication & marketing best practices. Maintainers and engineers do not always have the skills required to craft the right message to get attention, and they often appreciate help with marketing and outreach, which allows them to focus on the technical aspects that are more comfortable for them (Linåker et al., 2024). As an example of unclear and overcomplicated messaging, the description for an enhancement proposal was originally, “This KEP proposes to allow mutating spec.completions for Indexed job if spec.completions equals to spec.parallelism before and after the update, meaning that spec.completions is mutable only in tandem with spec.parallelism.” It was rewritten as, “Enables autoscaling for Indexed Jobs.” By providing resources for communications and marketing, your organization can see the following benefits: -

- - - -

- Metrics -

- - - -

- Long Term Health Metrics -

- - - -

- Events -

- -

- Events serve many purposes, from new contributor workshops to help with onboarding, to summits with many high-bandwidth conversations that can unblock development. Many open source projects resolve blockers and bigger issues at events (Foster, 2018), and being in the room can be a huge advantage to make sure your organization’s needs are considered. They’re also useful for workshops and training new contributors. Events are often underestimated, but they help with creating a healthy project, and are important. Your organization benefits in several ways: -

- - - -

- Metrics: -

- - - -

- Long Term Health Metrics: -

- - - -

- [/vc_column][/vc_row] -

+| Type | Metric | Source | +| :--- | :--- | :--- | +| Bugs | | Bugs closed this year (GitHub Query): **is:issue closed:2024-01-01..2024-11-01 label:kind/bug** | +| Security | | Time to resolve security issues (GitHub cli tool): **gh issue list –search “is:closed label:security” –json “id,openedAt, closedAt”** | +| Issue / Change Request | | GitHub queries can be tailored with **author:** keyword look at organization created items to track own items vs. project as a whole.

Issues/PR queries can be tailored to track initiatives | + +### Overall Project Health + +Improving the project health for your organization’s critical open source projects is crucial for reducing the risk associated with using those projects. Within the CHAOSS project, we have [Practitioner Guides](https://chaoss.community/about-chaoss-practitioner-guides/) and [metrics](https://chaoss.community/kbtopic/all-metrics/) that can help identify areas within those open source projects that aren’t as healthy as they could be and take steps to improve in those areas. This is not an exhaustive list, but it should provide a good starting point. + +- [Organizational Participation](https://chaoss.community/practitioner-guide-organizational-participation/) +- [Contributor Sustainability](https://chaoss.community/practitioner-guide-contributor-sustainability/) +- [Responsiveness](https://chaoss.community/practitioner-guide-responsiveness/) +- [Security](https://chaoss.community/practitioner-guide-security/) +- [Adoption](https://chaoss.community/?p=3573) +- [Documentation Usability](https://chaoss.community/?p=3532) + +There are also a number of other important roles in open source projects that can provide value to your organization and improve the health of a project. These are covered in more detail in the Appendix and include, but aren’t limited to: + +- Triage and program management +- Documentation +- Communications and marketing +- Events + +# Conclusion + +To tie all of this back together, here is an example report for the 2 open source projects, Foo and Bar described earlier that can be delivered as a quarterly report. This report highlights the percentage of bugs and security issues fixed by others along with the average time to fix them. Below the table shows a breakdown of Software Engineer (SWE) investment, along with the notes related to the categories that were identified as important. + +Telling the story and communicating to leadership might include examples like these: + +- For Bar, we have allocated 1 SWE per quarter, and have seen a good improvement in overall security with bugs trending down and the bugs we filed are all being fixed. +- For Foo, feature baz is set to release next month, and it should drastically improve our developer productivity. We estimate a lag of 2 weeks from release before we can roll it out internally. We made a strategic hire, and they are helping drive that feature upstream and are also spending time upskilling our own team to better engage the project. + +
+Summary of a report communicating the value of contributions to Foo and Bar +
+ +One guide can’t possibly cover everything about demonstrating the value of your organization’s open source efforts, so this is just a fraction of the potential ways that open source investments can be tracked and tied back to goals while effectively describing the value of the contributions. It requires the right framing and ensuring that resources are allocated where they may have the most value; however, the framing and resource allocation will be different in almost every situation, since it depends on your organization’s unique needs. + +# Cautions and Considerations + +- Demonstrating organizational value is highly dependent on your organization’s needs, so these activities should be interpreted in light of your organization’s goals. +- Return on Investment (ROI) is notoriously difficult to measure, but it’s even more difficult to measure when talking about investments in open source software. Coming up with a single ROI number is unlikely, so telling stories and giving examples will be an important part of helping your organization understand the value of your open source efforts. + +# Additional Reading + +- We have a [short video](https://www.youtube.com/watch?v=Gy9ZI-1_SMA&list=PL60k37cxI-HSHV4-rEsWMzExw2y2Oq79Z&index=8) (6 minutes) devoted to this guide on the CHAOSS YouTube channel. +- [CHAOSScast Episode 120: Practitioner Guide - Demonstrating Organizational Value](https://podcast.chaoss.community/120) +- KubeCon talk by Bob Killen: [Why is this so hard? Conveying the Business Value of Open Source](https://kccnceu2024.sched.com/event/1YeQH/why-is-this-so-hard-conveying-the-business-value-of-open-source-bob-killen-google) (slides and video) and the [White Whale](https://static.sched.com/hosted_files/lfms24/f9/Chasing%20the%20White%20Whale%20of%20Open%20Source%20-%20ROI.pdf) talk from the Linux Foundation Member Summit. +- [CHAOSS Practitioner Guides](https://chaoss.community/about-chaoss-practitioner-guides/) on topics that include improving responsiveness, contributor sustainability, organizational participation, and security. + +# Contributors + +The following people contributed to this guide: + +- Bob Killen +- Dawn Foster +- Justin Gosses +- Ria Schalnat +- Matt Germonprez +- Mike Gifford + +# References + +- Eghbal, N. (2016). [Roads and Bridges: The Unseen Labor Behind Our Digital Infrastructure](https://www.fordfoundation.org/work/learning/research-reports/roads-and-bridges-the-unseen-labor-behind-our-digital-infrastructure/). New York, NY: Ford Foundation. +- Foster, D. (2018). Understanding Collaboration in Fluid Organizations, a Proximity Approach [Doctoral Dissertation, University of Greenwich]. +- Linåker, J., Link, G., & Lumbard, K. (2024, October). Sustaining Maintenance Labor for Healthy Open Source Software Projects through Human Infrastructure: A Maintainer Perspective. In *Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement* (pp. 37-48). +- Linux Foundation (2023). [Open Source Maintainers](https://project.linuxfoundation.org/hubfs/LF%20Research/Open%20Source%20Maintainers%202023%20-%20Report.pdf): Exploring the people, practices, and constraints facing the world’s most critical open source software projects. +- Vargas, Sophia (2023). [What brings you to open source?](https://opensource.google/static/documentation/publications/WhatBringsYouToOpenSource_2023.pdf) How can projects encourage sustainable open source participation? Google Open Source. + +CHAOSS Practitioner Guides are MIT licensed, living documents, and we welcome your feedback and input. You can suggest edits to this document at [https://github.com/chaoss/hugo/blob/main/content/practitioner%20guides/practitioner-guide-demonstrating-org-value.md](https://github.com/chaoss/hugo/blob/main/content/practitioner%20guides/practitioner-guide-demonstrating-org-value.md). + +# Appendix + +There are a number of important roles in open source projects that can provide value to your organization and improve the health of a project. These include, but aren’t limited to: + +- Triage and program management +- Documentation +- Communications and marketing +- Events + +**Triage and Program Management** + +Having a product manager or program manager can help cut down on engineering time spent on things that aren’t really engineering efforts and that engineers might not be able to as efficiently as someone with expertise managing products and programs. Doing triage can provide awareness of open source project activities and issues that are important to your organization much earlier than if you were not involved. By reviewing incoming issues, assigning labels and priorities, and aiding in roadmap planning, your organization gets a number of benefits: + +- Prioritized triage of issues & bugs +- Decreased time to resolve +- Early awareness of potential breaking changes & security issues +- De-risk usage of project by improving overall project health +- Introduces better data to answer questions & track trends. + +Key Metrics: + +- Decreased [time to first response](https://chaoss.community/?p=3448) on issues and change requests +- Decreased time to assignment +- Decreased [issue resolution duration](https://chaoss.community/?p=3630) / [change request duration](https://chaoss.community/?p=3587) + +Long Term Health Metrics: + +- Increased contributor engagement and sustainability across a variety of metrics (e.g., [number of contributors](https://chaoss.community/?p=3630), [project velocity](https://chaoss.community/?p=3572), frequency of contributions, retention) +- Positive sentiment on issues / change requests and other communication channels + +**Documentation** + +Many people rely on an open source project’s documentation when getting started, and good documentation can help reduce support requests for common questions or issues that can be headed off with good documentation. In particular, a well documented contributor guide can help with onboarding new people and creating a good start for a contributor funnel. They can increase adoption and general overall positive sentiment with improved site traffic, search engine findability, and other web statistics. Contributors rarely want to do the big initial heavy lift to create documentation from scratch, but if a project has good docs to start, folks are much more open to keeping them updated. **“Having good docs leads to better docs.”** Well-documented open source projects are easier for users to understand and utilize, thus aiding in contributor sustainability, and your organization gets the following benefits: + +- Better developer experience +- Increases ability to self-service / decreased engineering time +- De-risk usage of project by improving overall project health + +Metrics: + +- Decreased support questions opened +- Increased site traffic & accompanying site metrics + +Long Term Health Metrics: + +- Increased [new contributor](https://chaoss.community/?p=3613) engagement & retention +- Positive sentiment on issues and change requests + +**Communications and Marketing** + +Open source projects often lack both the knowledge and skills regarding communication & marketing best practices. Maintainers and engineers do not always have the skills required to craft the right message to get attention, and they often appreciate help with marketing and outreach, which allows them to focus on the technical aspects that are more comfortable for them (Linåker et al., 2024). As an example of unclear and overcomplicated messaging, the description for an enhancement proposal was originally, “This KEP proposes to allow mutating spec.completions for Indexed job if spec.completions equals to spec.parallelism before and after the update, meaning that spec.completions is mutable only in tandem with spec.parallelism.” It was rewritten as, “Enables autoscaling for Indexed Jobs.” By providing resources for communications and marketing, your organization can see the following benefits: + +- Improved brand awareness & association +- De-risk usage of project by driving more adoption and conversion to contributors +- Early awareness of potential breaking changes + +Metrics + +- Increased traffic to website and accompanying site metrics +- Increased social engagement +- Increased share of voice + +Long Term Health Metrics + +- Growth in [adoption](https://chaoss.community/?p=3573) (e.g., downloads, stars / forks, share of voice) +- Positive sentiment on communication channels + +**Events** + +Events serve many purposes, from new contributor workshops to help with onboarding, to summits with many high-bandwidth conversations that can unblock development. Many open source projects resolve blockers and bigger issues at events (Foster, 2018), and being in the room can be a huge advantage to make sure your organization’s needs are considered. They’re also useful for workshops and training new contributors. Events are often underestimated, but they help with creating a healthy project, and are important. Your organization benefits in several ways: + +- Workshops can serve as onboarding or additional skill-up opportunities +- Being “in-the-room” can ensure organizational needs are addressed +- De-risk usage of project by improving overall project health + +Metrics: + +- Tracking contributions from attendees +- Conversions from attendees / contributors to maintainers +- Unblocked issues/features + +Long Term Health Metrics: + +- Increased contributor engagement and sustainability across a variety of metrics (e.g., [number of contributors](https://chaoss.community/?p=3630), [project velocity](https://chaoss.community/?p=3572), frequency of contributions, retention) +- Increase in conversion from contributors to maintainers + +CHAOSS Practitioner Guides are MIT licensed, living documents, and we welcome your feedback and input. You can suggest edits to this document at [https://github.com/chaoss/hugo/blob/main/content/practitioner%20guides/practitioner-guide-demonstrating-org-value.md](https://github.com/chaoss/hugo/blob/main/content/practitioner%20guides/practitioner-guide-demonstrating-org-value.md). [1]: https://chaoss.community/practitioner-guide-introduction/ [2]: https://project.linuxfoundation.org/hubfs/LF%20Research/Open%20Source%20Maintainers%202023%20-%20Report.pdf?hsLang=en @@ -896,4 +439,4 @@ Bugs, security, and issues are useful because most people understand them withou [8]: https://chaoss.community/?p=3534 [9]: https://chaoss.community/practitioner-guide-contributor-sustainability/ [10]: https://chaoss.community/practitioner-guide-security/ - [11]: https://helm.sh/blog/the-road-to-helm-4/ \ No newline at end of file + [11]: https://helm.sh/blog/the-road-to-helm-4/ diff --git a/content/practitioner guides/practitioner-guide-funding-impact.md b/content/practitioner guides/practitioner-guide-funding-impact.md index bf9a824..3eeaef2 100644 --- a/content/practitioner guides/practitioner-guide-funding-impact.md +++ b/content/practitioner guides/practitioner-guide-funding-impact.md @@ -11,7 +11,6 @@ nectar-metabox-portfolio-display-sortable: - off url: /practitioner-guide-funding-impact --- -\[vc\_row type=”in\_container” full\_screen\_row\_position=”middle” column\_margin=”default” column\_direction=”default” column\_direction\_tablet=”default” column\_direction\_phone=”default” scene\_position=”center” text\_color=”dark” text\_align=”left” row\_border\_radius=”none” row\_border\_radius\_applies=”bg” overlay\_strength=”0.3″ gradient\_direction=”left\_to\_right” shape\_divider\_position=”bottom” bg\_image\_animation=”none”\]\[vc\_column column\_padding=”no-extra-padding” column\_padding\_tablet=”inherit” column\_padding\_phone=”inherit” column\_padding\_position=”all” background\_color\_opacity=”1″ background\_hover\_color\_opacity=”1″ column\_shadow=”none” column\_border\_radius=”none” column\_link\_target=”\_self” gradient\_direction=”left\_to\_right” overlay\_strength=”0.3″ width=”1/1″ tablet\_width\_inherit=”default” tablet\_text\_alignment=”default” phone\_text\_alignment=”default” column\_border\_width=”none” column\_border\_style=”solid” bg\_image\_animation=”none”\] # Practitioner Guide: Funding Impact Measurement @@ -25,13 +24,13 @@ The audience for this guide is organizations who seek to demonstrate and/or just Open source software has become ubiquitous and can be found in almost every codebase, but sustaining those open source projects and communities over the long-term can be a challenge. Historically, much of the funding provided to open source projects has been for development of new projects and/or new features with the goal of fostering novel innovation. However, only funding innovation is not enough to ensure the sustainability of open source projects (Osborne, 2024). Several high profile, widely spread security vulnerabilities in open source projects (e.g., Log4Shell, Heartbleed) have also highlighted the need to fund maintenance and security and maintenance work in critical projects. With the push toward digital sovereignty and the recognition of the role that open source software can play, governments are now funding the maintenance of popular open source projects (e.g., [Sovereign Tech Agency in Germany][4], the work toward an [EU Sovereign Tech Fund][5]). Similarly, many companies are also seeking to support the open source projects that they rely on and use in their business operations. The role of open source software for supply chain security is also becoming increasingly important as governments, corporate procurement teams, and other groups start to create Software Bill of Materials (SBOMs) mandates. -Much of the critical infrastructure that we all rely on is made up of open source projects that lack the resources to be properly maintained over the long term. Companies, public institutions, and philanthropic organizations are beginning to fill this gap, but measuring the impact of this funding is an ongoing challenge (Osborne et al., 2024). Funders need to be able to understand the impacts of past funding in order to secure buy-in for future funding as well as to adapting and/or innovating [funding approaches][6] whilst mitigating ineffective or even harmful approaches. We all benefit from more public institutions, philanthropic organizations, and companies giving money to open source; when done in a way that positive impact is the ultimate goal and objective. +Much of the critical infrastructure that we all rely on is made up of open source projects that lack the resources to be properly maintained over the long term. Companies, public institutions, and philanthropic organizations are beginning to fill this gap, but measuring the impact of this funding is an ongoing challenge (Osborne et al., 2024). Funders need to be able to understand the impacts of past funding in order to secure buy-in for future funding as well as to adapting and/or innovating [funding approaches][6] whilst mitigating ineffective or even harmful approaches. We all benefit from more public institutions, philanthropic organizations, and companies giving money to open source; when done in a way that positive impact is the ultimate goal and objective. # Challenges Interest in open source funding is increasing, but there is little consensus on how to measure them in a meaningful way. Not only is the measurement of the impacts of open source funding challenging, but it is complicated by the fact that introducing funding into open source projects may change contributor incentives and the balance of voluntary and paid participation when some contributors become funded for project work. Funding may or may not be the right solution for all projects, and funders need to think about the best way to achieve their goals when funding projects. For example, general funding may not improve the security posture of open source projects as well as targeted support designed to strengthen security (Brackett, et. al., 2025) By better understanding the impacts of funding (or lack of), funders can learn more about what works well for different types of open source projects so that the funding provided can have the greatest benefit for fundees and more widely across the entire ecosystem. -Developing standardized funding impact measurement frameworks also presents challenges because the goals and objectives for the funding varies widely by funding organization and open source project, and how impact is measured depends on those goals and objectives. For example, providing funding for new feature development vs. funding for security fixes and maintenance require different funding impact measurement approaches. +Developing standardized funding impact measurement frameworks also presents challenges because the goals and objectives for the funding varies widely by funding organization and open source project, and how impact is measured depends on those goals and objectives. For example, providing funding for new feature development vs. funding for security fixes and maintenance require different funding impact measurement approaches. # Lessons Learned @@ -53,7 +52,7 @@ Different funding instruments and procurement – whether milestone-based contra ### Consider Project Life Stages and Social Structures -The impacts of funding vary depending on both a project's life stage and its social structure. For example, a new prototype project will have very different needs and potential outcomes compared to a mature project with an established community. While the prototype might need funding to build initial features and attract contributors, the mature project might require support for security improvements or maintenance work. +The impacts of funding vary depending on both a project's life stage and its social structure. For example, a new prototype project will have very different needs and potential outcomes compared to a mature project with an established community. While the prototype might need funding to build initial features and attract contributors, the mature project might require support for security improvements or maintenance work. Similarly, considering the social structure of a project is crucial for understanding funding impacts. For instance, Nadia Asparouhova (was Eghbal) (2020) identifies four distinct models of open source projects based on their contributor and user ratios: federations, clubs, stadiums, and toys. Funding for a federation-type project might need to account for complex governance processes and multiple working groups, while funding for a stadium-type project might focus on supporting its small core of maintainers who serve a large user base. These structural differences affect not only how funding should be allocated but also how its impact should be measured. @@ -65,257 +64,78 @@ Salary structures and cost factors across regions and organizations should also Funding can have many different types of impacts, which funders might want to consider (see also the [Superbloom toolkit][7]). The potential social, economic, and technological impacts can be both positive and negative, direct and indirect, internal (i.e. within a project) and external (i.e. among a project’s ecosystem of dependents and users), and manifest over various time horizons. -When assessing the impacts of open source funding, there may be a tendency to focus on technological impacts, which may in part be due to the technological nature of open source development or the relative ease of measuring technological impacts due to data availability from repositories. However, the potential impacts of funding extend beyond the code itself, and it is crucial to consider economic and social impacts on projects and their wider ecosystems of dependents and users. It is also important to note that impacts are not linear or unidirectional; funding can lead to both improvements and degradations across different metrics. This multidirectional nature of impacts means that measurement frameworks need to be flexible to capture both positive and negative changes, rather than assuming funding inherently leads to improvements or that metrics only move in one direction. Illustrative examples of various impacts can be found in the table below. +When assessing the impacts of open source funding, there may be a tendency to focus on technological impacts, which may in part be due to the technological nature of open source development or the relative ease of measuring technological impacts due to data availability from repositories. However, the potential impacts of funding extend beyond the code itself, and it is crucial to consider economic and social impacts on projects and their wider ecosystems of dependents and users. It is also important to note that impacts are not linear or unidirectional; funding can lead to both improvements and degradations across different metrics. This multidirectional nature of impacts means that measurement frameworks need to be flexible to capture both positive and negative changes, rather than assuming funding inherently leads to improvements or that metrics only move in one direction. Illustrative examples of various impacts can be found in the table below. **Examples of Social, Economic, Technological Impact Areas of Open Source Funding** - - - - - - - - - - - - - - - - - - - - - -
- - Internal Impacts (Project-level) - - External Impacts (Ecosystem-level) -
- Direct

- -

- Impacts

- - -
    -
  • - Social: User trust, ecosystem community, ecosystem events -
  • -
  • - Economic: Cost savings for adopters, integration/support costs, shared maintenance burden -
  • -
  • - Technological: Stability of APIs, ecosystem-wide security updates, interoperability -
  • -
-
- Indirect

- -

- Impacts

- - -
    -
  • - Social: Cross-project collaboration, training & education resources, ecosystem engagement -
  • -
  • - Economic: Market growth, job creation, industry cost reduction, start-up creation -
  • -
  • - Technological: Standardisation, research papers, patents, ecosystem-wide security improvements -
  • -
-
- -

- When measuring the impact of funding on open source development, it is crucial to consider the effects across different time horizons. The influence of funding can manifest differently in the short, medium, and long term, which is essential for a comprehensive understanding of funding impacts. -

- -
    -
  1. - Short term (<1 year): These are the immediate effects of funding, which may be easier to observe and attribute to the funding received. Short-term impacts might include immediate increases in development activity, improvements in code quality, or rapid growth in community size. -
  2. -
  3. - Medium term (1-3 years): Medium-term impacts may reveal how the project adapts and grows with the resources provided. This period might show the development of new features, expansion of the project’s reach, or evolution of the project’s governance structures. -
  4. -
  5. - Long term (>3 years): Long-term impacts are the enduring effects of funding that manifest over an extended period, which can be the most challenging to measure and attribute directly to funding. -
  6. -
- -

- Methods -

- -

- When it comes to measuring the impacts, funders can consider a breadth of methods for impact measurement. We recommend using a mixed methods approach, which provides a way to combine scalable quantitative measures along with contextual depth from qualitative data to better understand the funding impact. -

- -

- Quantitative methods offer a structured, scalable approach for measuring relationships between funding and outcomes in open source projects and their ecosystems. Starting with the funding objectives provides a focus for metric selection, rather than trying to cast a wide net that includes metrics that aren’t as relevant. These quantitative metrics can be gathered from repositories, surveys, and other sources, but there are also challenges. Some data isn’t often available for open source projects (e.g., usage, non-code contributions), and open source projects have different ways of working (e.g., commit strategies, code review processes) that make analysis more difficult. -

- -

- Qualitative methods, such as case studies, interviews, and participant observation, are useful strategies for collecting contextual data about the history, role, and impacts of funding on open source projects. A major strength of interviews lies in their ability to provide rich, contextual understandings by gaining insights into the perspective of the funding recipients, revealing why funding was needed and how it has influenced project dynamics, individual motivations, and community structures. Qualitative methods are useful for capturing intangible outcomes and hard-to-quantify impacts, such as changes in project culture or enhancements in the overall sustainability of the project, which often play a crucial role in the long-term success of open source projects but can be challenging to measure through quantitative means alone. Challenges can include securing the right people for interviews, ensuring representation, and the time required to conduct and analyze qualitative data. -

- -

- Mixed-Methods approaches that combine qualitative and quantitative techniques offer the best of both words: scalability and contextual depth. Mixed-methods approaches are particularly useful for examining complex open source ecosystems, where development spans multiple channels and goes beyond repositories. There are several ways to conduct mixed methods approaches. By starting with quantitative data the subsequent qualitative phase can be used to better explain the earlier data, which can be especially useful when looking at a large number of projects. On the other hand, starting with qualitative data offers a more exploratory approach that can be used to decide which quantitative measures to use, which may offer a better way to discover unanticipated impacts. They can also be collected and analyzed concurrently using a maturity model approach to create a holistic view of the development of an open source project, but this can be time-consuming and challenging to get agreement on what constitutes maturity. -

- -

- A much more detailed assessment of the strengths and weaknesses of qualitative, quantitative, and mixed methods approaches can be found in Section 3.4 of Osborne et al., (2024) with a summary in Table 3 on page 10. -

- -

- Conclusion -

- -

- While measuring the impact of funding open source projects can be challenging, this guide provides a framework for ways that impact(s) can be measured based on lessons learned from organizations already doing this work. By starting with the organization’s funding objectives, understanding the open source projects being funded, and accounting for cost factors, the funding initiatives can be put in the proper context before starting to measure the impact. This can be followed by an assessment of the economic, social and technological impacts that consider direct / indirect impacts along with project-level / ecosystem level impact to think about the metrics and timelines that might be important in your context. The final step is to consider how using a mixed-methods approach can combine scalable quantitative measures along with contextual depth from qualitative data to better understand the funding impact. The case studies section includes examples of organizations who are already doing this work. -

- -

- Funders need to be able to understand the impacts of past funding in order to secure buy-in for future funding as well as to adapting and/or innovating funding approaches whilst mitigating ineffective or even harmful approaches. We all benefit from more public institutions, philanthropic organizations, and companies giving money to open source; when done in a way that positive impact is the ultimate goal and objective. We hope that this guide helps organizations measure the impact of their funding initiatives so that we can increase the funding to open source projects to drive future improvements and allow these projects, and the people working on them, to become healthier and more sustainable over time. -

- -

- Cautions and Considerations -

- - - -

- Additional Resources -

- - - -

- Contributors -

- -

- The following people contributed to this guide: -

- - - -

- References -

- - - -

- CHAOSS Practitioner Guides are MIT licensed, living documents, and we welcome your feedback and input. You can suggest edits to this document at https://github.com/chaoss/wg-data-science/blob/main/practitioner-guides/funding-impact.md -

- -

- [/vc_column][/vc_row] -

+| | Internal Impacts (Project-level) | External Impacts (Ecosystem-level) | +| :--- | :--- | :--- | +| **Direct Impacts** | | | +| **Indirect Impacts** | | | + +When measuring the impact of funding on open source development, it is crucial to consider the effects across different time horizons. The influence of funding can manifest differently in the **short, medium, and long term**, which is essential for a comprehensive understanding of funding impacts. + +1. Short term (<1 year): These are the immediate effects of funding, which may be easier to observe and attribute to the funding received. Short-term impacts might include immediate increases in [development activity](https://chaoss.community/?p=3444), improvements in code quality, or rapid growth in [community size](https://chaoss.community/?p=3484). +2. Medium term (1-3 years): Medium-term impacts may reveal how the project adapts and grows with the resources provided. This period might show the development of new features, expansion of the project’s reach, or evolution of the project’s governance structures. +3. Long term (>3 years): Long-term impacts are the enduring effects of funding that manifest over an extended period, which can be the most challenging to measure and attribute directly to funding. + +## Methods + +When it comes to measuring the impacts, funders can consider a breadth of methods for impact measurement. We recommend using a mixed methods approach, which provides a way to combine scalable quantitative measures along with contextual depth from qualitative data to better understand the funding impact. + +**Quantitative methods** offer a structured, scalable approach for measuring relationships between funding and outcomes in open source projects and their ecosystems. Starting with the funding objectives provides a focus for metric selection, rather than trying to cast a wide net that includes metrics that aren’t as relevant. These quantitative metrics can be gathered from repositories, surveys, and other sources, but there are also challenges. Some data isn’t often available for open source projects (e.g., [usage](https://chaoss.community/?p=3573), non-code contributions), and open source projects have different ways of working (e.g., commit strategies, [code review](https://chaoss.community/?p=4712) processes) that make analysis more difficult. + +**Qualitative methods**, such as case studies, interviews, and participant observation, are useful strategies for collecting contextual data about the history, role, and impacts of funding on open source projects. A major strength of interviews lies in their ability to provide rich, contextual understandings by gaining insights into the perspective of the funding recipients, revealing why funding was needed and how it has influenced project dynamics, individual motivations, and community structures. Qualitative methods are useful for capturing intangible outcomes and hard-to-quantify impacts, such as changes in project culture or enhancements in the overall sustainability of the project, which often play a crucial role in the long-term success of open source projects but can be challenging to measure through quantitative means alone. Challenges can include securing the right people for interviews, ensuring representation, and the time required to conduct and analyze qualitative data. + +**[Mixed-Methods](https://arxiv.org/pdf/2411.06027)** approaches that combine qualitative and quantitative techniques offer the best of both words: scalability and contextual depth. Mixed-methods approaches are particularly useful for examining complex open source ecosystems, where development spans multiple channels and goes beyond repositories. There are several ways to conduct mixed methods approaches. By starting with quantitative data the subsequent qualitative phase can be used to better explain the earlier data, which can be especially useful when looking at a large number of projects. On the other hand, starting with qualitative data offers a more exploratory approach that can be used to decide which quantitative measures to use, which may offer a better way to discover unanticipated impacts. They can also be collected and analyzed concurrently using a maturity model approach to create a holistic view of the development of an open source project, but this can be time-consuming and challenging to get agreement on what constitutes maturity. + +A much more detailed assessment of the strengths and weaknesses of qualitative, quantitative, and mixed methods approaches can be found in Section 3.4 of Osborne et al., (2024) with a summary in Table 3 on page 10. + +# Conclusion + +While measuring the impact of funding open source projects can be challenging, this guide provides a framework for ways that impact(s) can be measured based on lessons learned from organizations already doing this work. By starting with the organization’s funding objectives, understanding the open source projects being funded, and accounting for cost factors, the funding initiatives can be put in the proper context before starting to measure the impact. This can be followed by an assessment of the economic, social and technological impacts that consider direct / indirect impacts along with project-level / ecosystem level impact to think about the metrics and timelines that might be important in your context. The final step is to consider how using a mixed-methods approach can combine scalable quantitative measures along with contextual depth from qualitative data to better understand the funding impact. The case studies section includes examples of organizations who are already doing this work. + +Funders need to be able to understand the impacts of past funding in order to secure buy-in for future funding as well as to adapting and/or innovating funding approaches whilst mitigating ineffective or even harmful approaches. We all benefit from more public institutions, philanthropic organizations, and companies giving money to open source; when done in a way that positive impact is the ultimate goal and objective. We hope that this guide helps organizations measure the impact of their funding initiatives so that we can increase the funding to open source projects to drive future improvements and allow these projects, and the people working on them, to become healthier and more sustainable over time. + +# Cautions and Considerations + +- There is no one-size-fits-all approach to measuring impact. Measuring the impact(s) of open source project funding is something that we’re all figuring out as we go along, so we expect this space to evolve over time as we learn more about the best ways to measure impact. +- Funding may or may not be the right solution for all projects and can result in both positive and negative impacts for projects. Funding can introduce unanticipated challenges for open source projects, especially the first time they receive funding, and some projects struggle to use their funding. +- Open source projects are made up of people, and money can create complications. Funding into open source projects can change contributor incentives and the balance of voluntary and paid participation, which should be considered when measuring impact. + +# Additional Resources + +- [A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development](https://arxiv.org/abs/2411.06027) - this paper was the starting point for this guide, and it provides many more details on the subject. +- [CHAOSS Funding Impact Measurement Working Group](https://github.com/chaoss/wg-funding-impact) - join us! +- [Roadwork ahead: Evaluating the needs of FOSS communities working on digital infrastructure in the public interest](https://recommendations.implicit-development.org/) +- [Podcast: Funding Impact Measurement Working Group on CHAOSScast](https://podcast.chaoss.community/106) in March 2025. +- [Toolkit for Measuring the Impacts of Public Funding for OSS](https://25.foss-backstage.de/schedule/#session/7UJYUR/) at FOSS Backstage in Berlin in March 2025 ([video](https://25.foss-backstage.de/session/toolkit-for-measuring-the-impacts-of-public-funding-for-foss/)). +- [Panel: The Impact of Funding for Sustainable Open Source Projects](https://sched.co/1zfhR) with Georg Link, Andrew Nesbitt, and Alyssa Wright at the Open Source Summit NA in Denver in June 2025 ([video](https://www.youtube.com/watch?v=8sD81Dqw0Os)). +- [FOSS Funders](https://fossfunders.com/) +- [OSS Research-Software: Incentivizing Investment in and Adoption of Open Infrastructure for Research through Community Health Assessments](https://infrastructureinsights.fund/projects/oss-research-software/) +- [Building bridges: How trust and community health frameworks can strengthen open infrastructure decision-making](https://investinopen.org/blog/building-bridges-how-trust-and-community-health-frameworks-can-strengthen-open-infrastructure-decision-making/) +- [Building Blocks: Digital Infrastructure Funding Toolkit](https://buildingblocks.superbloom.design/about/) + +# Contributors + +The following people contributed to this guide: + +- Dawn Foster +- Cailean Osborne +- Paul Sharratt +- Mirko Boehm +- Serkan Holat +- Katharina Meyer + +# References + +- Brackett, S. A., Scott, S., and Chen, C. (2025). [Buying Security: Open Source Software Funding and Security Posture](https://infrastructureinsights.fund/wp-content/uploads/2025/12/CSINTPolicyPaperNo22025BrackettScottandCheng.pdf). CSINT Policy Paper Series. Fall 2025. +- Casari, A., Ferraioli, J., & Lovato, J. (2023). [Beyond the Repository](https://dl.acm.org/doi/pdf/10.1145/3605160). *Communications of the ACM*, *66*(10), 50-55. +- Asparouhova, N. (2020). *Working in public: the making and maintenance of open source software*. Stripe Press. +- Osborne, C., Sharratt, P., Foster, D., & Boehm, M. (2024). [A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development](https://arxiv.org/abs/2411.06027). *arXiv preprint arXiv:2411.06027*. +- Osborne, C. (2024). [Open Source Software Developers' Views on Public and Private Funding: A Case Study on scikit-learn](https://dl.acm.org/doi/abs/10.1145/3678884.3681844). In Companion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing (CSCW Companion '24). Association for Computing Machinery, New York, NY, USA, 154–161. [https://doi.org/10.1145/3678884.3681844](https://doi.org/10.1145/3678884.3681844) + +CHAOSS Practitioner Guides are MIT licensed, living documents, and we welcome your feedback and input. You can suggest edits to this document at https://github.com/chaoss/hugo/blob/main/content/practitioner%20guides/practitioner-guide-funding-impact.md [1]: https://chaoss.community/practitioner-guide-introduction/ [2]: https://chaoss.community/practitioner-guide-demonstrating-org-value @@ -323,4 +143,4 @@ When assessing the impacts of open source funding, there may be a tendency to fo [4]: https://www.bundeswirtschaftsministerium.de/Redaktion/EN/Pressemitteilungen/2022/10/20221018-the-sovereign-tech-fund-launches-funding-an-investment-in-europes-digital-sovereignty.html [5]: https://eu-stf.openforumeurope.org/ [6]: https://recommendations.implicit-development.org/ - [7]: https://buildingblocks.superbloom.design/about/ \ No newline at end of file + [7]: https://buildingblocks.superbloom.design/about/