Meet the teams: Hypha Worker Co-op — stewarding Cosmos Hub Testnets and helping scale Replicated Security
As stewards of the interchain, the Interchain Foundation (ICF) advances the creation of an interoperable, sustainable, and community-owned decentralized ecosystem. We do this by building basic protocols and funding teams working on developing and furthering the adoption of the Interchain Stack.
Hypha Worker Co-op is one of these Distributed Development Teams that the ICF works with. Since 2021, Hypha has been instrumental to the Cosmos Hub — from governance to validator relations, and is currently administering the Cosmos Hub Testnets Program.
In this series, written by the teams that the ICF works with, we explore the work that each of them do and how it benefits the interchain.
Hypha Worker Co-op was incorporated in 2019 in order to help communities build better relationships with technology. Hypha’s team of technologists, designers, and community organizers advise on sustainable practices, build capacity within organizations, and develop custom technologies to empower friends and neighbors — in the Great Lakes region and our virtual neighborhoods.
Hypha started working on the Cosmos Hub in late 2021. We began with analyzing the state of governance in the Cosmos ecosystem and continued to improve governance on the Hub by reinvigorating the Cosmos Hub forum. Some of our work was funded by Proposal #63, and we published reports on our deliverables and recommendations for ongoing stewardship of Cosmos Hub’s governance.
Cosmos Hub’s Testnets Program
In 2022, Hypha began coordinating the Cosmos Hub Testnets Program as a team funded by the Interchain Foundation. The first testnet to be set up was the release testnet, which provided a production-like environment to test out the Vega and Theta Gaia upgrades. To this day, the Release Testnet is often referred to as the ‘theta’ testnet even though it remains up-to-date with current Gaia releases.
In 2023, the Replicated Security testnet was created to help new consumer chains onboard, as well as to help validators level up at operating infrastructure for multiple chains. We also facilitated the Cosmos Hub’s third incentivized and adversarial testnet (Game of Chains), which comprised 32 tasks for 90 validator teams to hone their skill with the new technology and uncover bugs and edge cases ahead of the launch of Replicated Security.
Today, we continue to maintain both of these testnets, for both release testing and consumer chain onboarding. We also work closely with the Cosmos Hub development team at Informal Systems to verify and test Gaia releases on a battery of local testnets.
To help steer the direction of the Testnets Program and better meet the needs of the validator community, we’ve recently launched the Testnet Working Group with a subset of validators who are active in the testnet. To learn more about the working group and how to join, reach out to Lexa (lexa@hypha.coop).
Almost every week, we coordinate upgrades, launches, and rehearsals during Testnet Wednesdays, which was borne out of feedback from validators who needed a more predictable testnet participation cadence. Hypha is responsible for scheduling events, coordinating between stakeholders (consumer chain teams, validators, community contributors, and Informal’s Cosmos Hub team) right up to a chain’s mainnet launch. All of this work has supported past and existing chains, and is part of building the continuous feedback loops that future consumer chains will benefit from.
Scaling up Replicated Security
Since March 2023, the Hypha team has worked closely with the teams building Neutron and Stride to successfully launch two new consumer chains using Replicated Security. Together we’ve been through seven launch rehearsals, three synchronized upgrades, one re-launch, and many hours of behind-the-scenes planning. Drawing upon our learnings and reflections gathered during retrospectives with consumer chain teams, we’ve operationalized complex sociotechnical processes, written documentation to clarify key procedures like consumer key assignment, and iterated our joint communications strategy with various stakeholder teams to more effectively prepare validators for launch day.
Prior to the launch of Replicated Security, we spoke with upcoming consumer chains like Neutron and Noble, Cosmos Hub validators of all sizes, and the Hub development team at Informal Systems to put together the first major essay on what Replicated Security might look like on the Hub, “Preparing for Replicated Security”. That essay was closely followed by the publication of the Swiss Booklet: Launch Neutron on Replicated Security addressed community concerns about launching the first consumer chain. Hypha continues to be part of the conversation about economic sustainability under Replicated Security, with ideas such as conditional basic income.
Supporting Gaia Upgrades
A large chunk of our work consists of working closely with the Informal team to provide additional testing for upcoming Gaia releases. Using private local testnets, we simulate and verify major release candidates via automation using both fresh state and exported mainnet state before scheduling a coordinated upgrade on the Release Testnet. When a release candidate passes all tests locally, we apply the upgrade on the public Release Testnet, followed by the provider chain on the Replicated Security testnet. This process has been used for four major upgrades in 2023 (v8 through v11).
Our testing has uncovered various bugs and edge cases over the last year. Some examples include discovering that bank denom metadata was being mishandled by the upgrade handler in v8, verifying the multisig key assignment fix ahead of the emergency upgrade for v9, locating an error in the slash packet data in ICS v3, and uncovering a chain-stopping error in the liquid staking module in v12. We’re also working closely with our partners at Informal to reduce the lead time for each round of testing, which will enable more frequent (and boring) major version releases.
Looking Forward
In the coming year, we’ll continue to steward the Cosmos Hub Testnets Program, increasing validator engagement and improving the accuracy and speed of each testnet event. We’ve already started stacking multiple events on the same day, and we’re hoping to experiment with running Game Days on testnets soon. We’ll continue to grow the Testnet Working Group and pave the path for new validators to join The Hub.
If you’re interested in getting involved with the testnet, you can learn more at github.com/cosmos/testnets. You can also follow our work on our Cosmos blog, and say hello to us in the testnet channels on Discord!
