Open Source Observer: Measure the impact of open source contributions

Public Good Projects Discussion
1

Project Description

The goal of Open Source Observer is to measure the ROI of funding open source teams.

Specifically, we help open source communities and funders:

  • Track the full downstream impact of grants programs on key ecosystem metrics like user growth and developer activity over time
  • Create reward functions that allocate funding to projects based on impact analysis
  • Deep dive well-known projects and minimize bias

We’re putting all the metrics needed to inform better decision-making and grantmaking in one place, including:

  • Impact: trace a project’s downstream contribution to end users, revenue, ecosystem growth, and sustainability by linking it to platform metrics
  • Activity: graph a project’s team, momentum, and dependencies from GitHub and package managers
  • Financing: estimate funding and burn rates from grant metadata and analysis of public financial reports

We are building Open Source Observer entirely as a public good, not just with open source and open data, but with truly open infrastructure. We want to make the data and analyses as widely available as possible and are committed to sharing our infrastructure directly with the community.

Main Project Funding Sources

Open Source Observer is grant-funded.

So far, we’ve received grants and inkind support from Optimism, Arbitrum, Protocol Labs, and Gitcoin. As our infrastructure costs grow, we may look for cost-sharing opportunities with individuals and team who make use of the data to supplement what we receive from grants.

Seeking project-specific funding or funding for general operations

We are seeking general operations funding. We are currently a team of three building this as a public good. Apart from our team costs, our main expenses are related to our data infrastructure.

Roadmap

In the near term, we’re building a dashboard so people can explore and start to use our impact metrics. We’re also working on automated data ingestion pipelines for different blockchains and open source activity metrics (eg, GitHub, package managers, dependency networks, etc).

Next, we want to build a data collective and release an API for developers to experiment with and define their own impact metrics.

In the long run, we want to see composable, impact-based funding power the internet economies of the future.

Team Information

Ray, Reuven and I began Open Source Observer while at Protocol Labs, working on Hypercerts. We left in October and are now pursuing this full-time. We registered a company called Kariba Labs. Open Source Observer is the first product we’ve released.

  • Raymond Cheng is the engineering lead. He has started multiple projects that are used by millions of people, including co-founding security company Oasis Labs, and creating a networking project that was productionized by Google into Outline.
  • Reuven Gonzales is the senior software engineer. He has been a founder and early engineering hire and a variety of web2 and web3 companies, including Oasis Labs and GainForest.
  • Carl Cervone is the product and community lead. He is a Gitcoin steward, previously co-founded a startup that generates sustainability and transparency data for commodity supply chains, and frequently writes about public goods funding.

Social Credibility

We are still a young project and just launched our Twitter and Substack in November.

Our GitHub is the best place to see our work progress. We had various speaking events Devconnect, including dPact, Funding the Commons, Protocol Labs Summit, and Schelling Point.

Eligibility Criteria

  • Do you have a commitment to open-source (i.e. every open-source license accepted by the Open-Source Initiative) technology and sharing results publicly?
  • Have you provided transparency about how exactly funding will be used?
  • Are you advancing values of freedom and privacy (no surveillance and handling of personal data)?
  • Are you supporting decentralization in various fields (for example building Web3 projects)?
  • Have you provided social media channels to the extent that we can confirm social proof of your project?

Discord contact

Carl Cervone (@cerv1)

2 Likes
2

OSO x Octant Epoch 2 Update

Achievements since the last Octant Epoch

We completed a HUGE amount since the last epoch. Here are some of the highlights:

  • Migrated to a new, more scalable infrastructure for our data warehouse

  • Released the V1 of our API and are already working on integrations with Buidl Guidl, Gitcoin, and other teams

  • Added a complete set of docs for contributors, data scientists, and developers

  • Deployed our first set of impact metrics for code and onchain contributions

  • Expanded our OSS coverage to include projects building on Arbitrum

  • Completed a longitudinal study of the impact of Gitcoin Grants

  • Went to ETH Denver and co-hosted a Schelling Point session all about impact metrics

Upcoming milestones on your project roadmap

Now that our docs and infrastructure are in place, we’re focused on building a network of impact data scientists. More than 30 people have already signed up to join the Kariba Data Collective. We’re creating learning materials and “data challenges” to help people get started with our data and tools. Our goal is to run the first 2-3 challenges by the end of Q2.

We’re also working on expanding our coverage of onchain data (more chains!), onchain funding (including an integration with Drips), onchain identity (including collaboration with Gitcoin Passport), and adding more impact metrics for code contributions.

Finally, we’re going to be giving our project dashboards on opensource.observer some long-overdue love. We’re going to be adding more metrics, more visualizations, and more ways to explore the data.

Funding

How funds from the previous Epoch were utilized and plans for any funds raised from Epoch Two

We have claimed the funds and kept 100% on our multisig as of 2024-03-11 but will be moving them soon. We have not spent any of the funds yet. As stated in our application, we will use the funds to pay the core team and support the infrastructure costs of running our data warehouse and API.

Grant funding received since the last Epoch (e.g., RetroPGF, Gitcoin, etc.), including funding amounts

We started receiving our RetroPGF3 stream of 127,914 OP in February 2024. We also received a grant of 75K ARB from Arbitrum.

Other non-grant funding sources (like user payments, donations, staking or LP contributions to treasury, VC investments, etc.)

n/a

1 Like
3

OSO x Octant Epoch 3 Update

Although we were disappointed to miss out on Epoch 3 funding, our motivation to continue working with Octant is higher than ever :rocket:

Achievements since the last Octant Epoch

  • Our coverage is growing. We currently have 1,682 projects; 36,578 open source repos; 47,329 smart contracts on OSO. Here’s a snapshot report of what’s happening among the projects on Octant.

  • We made a number of schema updates to OSS Directory and are getting lots of community submissions, including some massive ones from our friends at growthepie.

  • We entered a range of strong new data partnerships, including with GoldSky, Karma3, Gitcoin Passport, ecosyste.ms and are onboarding tons of new public datasets (including Farcaster, Lens, all Superchain traces, and much much more). These datasets are open for anyone to build with and connect to!

  • We are building metrics for Optimism Retro Funding 4. Learn more here.

  • We participated and reviewed FIL RetroPGF 1. See our recent blog post with reflections here.

  • We are wrapping up our first community data challenge. Pairwise has been one of the most enthusiastic participants!

  • We released a stable V1 of The Open Source Observer API. Our free API lets you fetch data and impact metrics about any open source project (or collection of projects) you care about it. Our first public user was Gitcoin during GG20, for eligibility screening and impact widgets.

  • We co-organized the first day of Funding the Commons in the SF Bay Area, April 13-14, which is all about open source funding across web3, traditional FOSS, and government. Be on the lookout for recorded talks on the FTC YouTube channel. We also got to hang IRL with including Tor, growthepie, Drips, Hypercerts, Gitcoin, MetaGov, and Octant friends :slight_smile:

Upcoming milestones on your project roadmap

We are focused on helping with Optimism Retro Funding 4 and shipping a suite of new impact metrics that can help with onchain and off-chain impact assessments.

We’re planning our next data challenge with Karma3 Labs (OpenRank), using EigenTrust on the OSO events table. If this doesn’t make any sense to you, then imagine airdropping GLM to a subset of users who are “trusted nodes” in the Octant social graph on Farcaster and active Gitcoin donors. Or getting a personalized drips list based on developers you watch on GitHub.

We are pushing more metrics models to our API and connecting more and more data. The next big boulder is package dependencies.

Funding

How funds from the previous Epoch were utilized and plans for any funds raised from Epoch Three

n/a (we did not get funded in Epoch 3)

Grant funding received since the last Epoch (e.g., RetroPGF, Gitcoin, etc.), including funding amounts

We received $8,806.21 from Gitcoin GG20. We received $30.28 in direct donations from Giveth’s recent QF round.

Other non-grant funding sources (like user payments, donations, staking or LP contributions to treasury, VC investments, etc.)

n/a

4

Was excellent to meet you all there, and props to the success of the event. I can’t wait to see these partnerships continue!

1 Like
5

OSO x Octant Epoch 4 Update

  • Total Funds Received from Octant so far: 31.9731 ETH
  • Funds Spent: 12.5 ETH
  • Detailed Utilization: We claimed our Epoch 2 funds on 2024-03-11. We have since used these to pay our core team of three full-time developers (Carl, Ray, Reuven). We have moved our Epoch 4 funds to our Coinbase Prime account as of 2024-08-11 but have not spent any of these funds yet.

Milestones and Deliverables

  • Milestone 1:
    • Description: Coverage growth.
    • Immediate Outcome: We have continued to expand our coverage of projects, repos, and contracts in the Etheruem ecosystem. As of July, our coverage included: 2,385 projects (+10% from June), 46K open source repos (+23% from June), and 127M smart contracts. The smart contract figure includes “discovered contracts” downstream from project deployers or factory contracts. In June, there were 10,619 active contributors across these projects — up from 10,396 in May.
    • Funding Used: na (however, this was approximately 15% level of effort for the team)
  • Milestone 2:
    • Description: Launch of data exchange.
    • Immediate Outcome: We launched the OSO Data Exchange, a free public data exchange that in includes every model in the OSO pipeline, raw blocks/transactions/traces across 7 chains in the OP Superchain (OP Mainnet, Base, Frax, Metal, Mode, PGN, Zora), GitHub repo and developer activity, Gitcoin Grants and Passport, OpenRank, Farcaster, and Lens datasets. In the first month, more than 5 developer teams signed up for access and consumed a combined 90 TB of data.
    • Funding Used: na (however, this was approximately 50% level of effort for the team)
  • Milestone 3:
    • Description: Protocol metrics v1.
    • Immediate Outcome: We powered Optimism’s latest Retro Funding round with protocol-level impact metrics. OSO worked with Optimism’s Citizen House to develop 16 onchain impact metrics and apply them consistently across 230 projects and 6 OP stack chains. Read all about it here. This work directly informed the allocation of 10M OP.
    • Funding Used: na (however, this was approximately 35% level of effort for the team)
  • Karma GAP: Karma GAP - Open Source Observer

Challenges

  • Previous Challenges:
    • Description: In the last submission, we highlighted the challenge of scaling our data processing infrastructure to handle the increasing volume of data from various sources. We tripled our data volume since the last epoch, and now have more than 200 models running on top of it. This put a huge strain on our existing infrastructure, leading to performance bottlenecks and slower query times.
    • Updates: We have made significant progress by deploying a new GraphQL API service built on a scalable Clickhouse cluster, which has notably improved query performance.
    • Resolution: The resolution involved a lot of backend work to upgrade our infrastructure, specifically by swapping out CloudSQL with Clickhouse, and wiring up the new GraphQL API endpoint. We also enhanced our documentation for developer teams to better understand how to interact with the new API service.
    • Lessons Learned: One of the key lessons learned is how rapidly data processing costs can escalate as data volumes grow. We are now more mindful of the need to continuously optimize our infrastructure to ensure efficient data processing and delivery.

Outputs and Outcomes

Outputs (Last Epoch):

  • Output 1: Launched the OSO Data Exchange, offering a free public data exchange that includes models and datasets from 7 chains within the OP Superchain.
  • Output 2: Powered Optimism’s Retro Funding round with 16 on-chain impact metrics, applied across 230 projects and 6 OP stack chains.

Short-term Outcomes:

  • Outcome 1: Increased community engagement, particularly among developers integrating our data into their projects, as evidenced by new integrations like Pairwise, Retrolist, and RetroPGF Hub.
  • Outcome 2: Observed high demand for the OSO Data Exchange, with over 5 developer teams signing up for access and consuming a significant amount of data (90TB) within the first month.

Long-term Outcomes:

  • Outcome 1: Helped Optimism allocate 10M OP based on the impact metrics provided by OSO, demonstrating the potential of comparable impact metrics for other funders in the ecosystem.

  • Outcome 2: Increased coverage to 2,385 projects, 46K open-source repos, and 127M smart contracts, providing the most comprehensive, fully open source / open infra view of the Ethereum ecosystem for developers and data scientists.

    Metrics for Outcomes:

    1. Funding Allocation: 10M OP allocated by Optimism based on OSO’s impact metrics.
    2. Data Utilization: 90TB of data consumed by developer teams within the first month of launching the OSO Data Exchange.
    3. Coverage Expansion: 10% increase in projects, 23% increase in open-source repos, and 127M smart contracts covered by OSO.
    4. Project Integrations: total of 9 teams currently integrating OSO data into their projects (five via the OSO Data Exchange and four via direct API access).

Plans for Potential Funds Raised in the Next Octant Allocation Window

  • Goal 1:
    • Description: Expand the OSO Data Exchange to include additional data sources (eg, software dependencies) and integrations with other L2s.
    • Expected Outcome: Increased adoption of the OSO Data Exchange across multiple ecosystems, leading to greater coverage, more ecosystem-specific insights, and more integrations with developer partners.
    • Funding Needed: approx $10K per month per chain for indexing and storage costs, plus metrics for ~200 projects per chain.
  • Goal 2:
    • Description: Develop advanced analytics and visualization tools to better showcase the insights derived from OSO data.
    • Expected Outcome: Reboot of the OSO website to include timeseries datasets, artifact-level page views, and dynamic ecosystem reporting (eg, live stats for all Octant Projects).
    • Funding Needed: $50,000 (this will be the focus of our Epoch 4 funding)

Other Funding

  • Grant Funding Received Since the Last Epoch: We have not received any new liquid grants since the last epoch.
  • Other Non-Grant Funding Sources: n/a

Future Plans for Financial Sustainability

We are lean, and our primary costs are infrastructure and developer salaries. We are exploring recurring revenue streams and backstopping grants with some of our L2 partners. Octant funding makes a huge difference in our ability to continue to grow and scale our impact!

2 Likes
6

Thanks for the detailed update @cerv1
Great to see all the progress with the OSO Data Exchange launch and the expanded coverage. Powering Optimism’s Retro Funding round is a huge win! Looking forward to seeing the Data Exchange grow and the new analytics tools coming together. :slight_smile: