21 posts tagged with "monthly"

This is a monthly report on the progress of the 🐲 Hydra and 🛡 Mithril projects since July 2024. This document serves as both preparation for and a written summary of the monthly stakeholder review meeting, announced on our Discord channels and held on Google Meet. This month, the meeting took place on 2024-08-28, using these slides and you can see the recording here.

Mithril​

Issues and pull requests closed in August

Roadmap​

Here is an update on our current roadmap:

• Cardano transactions signature/proving MVP #1457: the feature is finalized and the last step to completion is to release on the mainnet.
• Cardano Stake Distribution certification #955: the feature is finalized and the last step is to release on preview, preprod and mainnet.
• Mithril signature diffusion with Cardano network layer PoC #1837: the proof-of-concept is completed.
• Decentralization of signature orchestration #1777: we have started implementing the feature with some preliminary works.
• CIP for Mithril signature diffusion through Cardano network #1775: the draft CIP has been published on the Cardano foundation CIPs repository and is under review.

Distributions​

We have released the new Mithril distribution 2430.0. This distribution includes several critical updates and enhancements:

• Support for Cardano node 9.1.0
• Support for Cardano transactions certification in release-preprod and pre-release-preview
• Bug fixes and performance improvements.

Future distributions​

We plan to release new distributions in September following the Chang hard fork:

• 2436:
  • Activation of the certification of Cardano transactions in the release-mainnet network
  • Activation of the certification of Cardano stake distribution in pre-release-preview, release-preprod, and release-mainnet.

Protocol status​

Latest status of the Mithril protocol on Cardano mainnet

Cardano stake distribution certification​

We have completed the implementation of the Cardano stake distribution certification with Mithril and are currently preparing for the rollout of this feature to production:

• We have implemented a new type of data certified for the latest snapshot of the Cardano stake distribution done by the Cardano node at the end of each epoch.
• Under the hood, we use a Merkle tree representation of the stake distribution and sign its Merkle root.
• The Mithril clients (library, CLI and WASM) have the capability to download and verify a snapshot of the Cardano stake distribution.
• We are preparing to activate the feature in the release-preview, release-preprod, and release-mainnet networks with the release of the next Mithril distribution.
• We have created documentation and some example code implementation for developers.

Hydra​

Issues and pull requests closed in August

Snapshot of the new roadmap with features and ideas

Notable updates on our roadmap this month include:

• Incremental decommits, Conway support fixes, and more were released in 0.18.0
• New landing page, SSL support, and bug fixes around transaction pruning were released in 0.18.1
• Incremental commits (coming in a 0.19 release) have pivoted to a deposit-based scheme based on open discussions (thanks @GeorgeFlerovsky!)
• Ready to shift ledger to Conway when hardfork is complete #1338
• Settled on Partial Fanout as a good solution to a few known problems
• Initial work on a suite of network-resiliance tests #1552
• The Hydra Doom demo yielded several ideas that need to be refined into features

Hydra Doom​

At the Rare Evo event in Las Vegas, we showcased the Hydra Doom demo, a technology demonstration of Hydra where users could play the 1993 id software game Doom and for each frame of the game, a smart contract transaction was submitted to a Hydra head which recorded game state, user inputs, and validated game state transitions (as a proof of concept, using only some minor logic pertaining to the movement of the player and dying in-game).

People could connect remotely via https://doom.hydra.family or in person on two custom-made arcade cabinets we had constructed for the occasion. Each game session contributed 35 transactions per second. In aggregate, we processed 106 million transactions with Hydra during the 78 hours of operation, which is more than the entire history of Cardano mainnet. The peak was around 7,200 tps.

The demo was very positively received on social media and seems to have rekindled the flames around the Hydra narrative both internally and externally, which hopefully bodes well for both future adoption interest among builders, as well as the prospects of securing funding for the continued development of Hydra from Intersect.

Network tests with fault injection​

We have set up a simple yet effective workflow capable of simulating a Hydra cluster under high transaction loads and packet loss network failures.

With this, we aim to consistently reproduce and resolve issues that are causing the head to become stuck, as outlined in #1436.

The workflow specifies a set of high-load scenarios, each of which spins up the same Hydra cluster on devnet used during the getting started tutorial and injects network faults using Pumba.

The idea is to allow and encourage everyone to experiment with this workflow and collect summary results, which contain details about the performance within the head. Because these tests are performed with the hydra-node being a 'black box,' it also allows us to explore alternative network stacks as a whole.

New landing page​

For the Rare Evo event with the public Hydra doom demo (see above), we also figured it would be a good idea to improve the first impression new visitors get from the https://hydra.family website.

The marketing team at IOG collaborated on creating new content and design for the landing page. In addition to highlighting the key features and explaining the importance of the Hydra Head protocol, the updated landing page also includes an analogy of how Hydra works, using a cargo plane as an example.

Screenshot of the new landing page

The original design included an alternative color scheme using Teal as the primary accent color and Plum as the secondary. An example is included below. Do you think a new color scheme would be appealing or do you prefer the Purple main color as-is? Shoot us a message on Twitter (X) or #ask-hydra on Discord with your thoughts.

Alternative color scheme example

Bug fix: layer 2 transactions during decommits​

After releasing the decommit feature that enables users to transfer funds from layer 2 to layer 1 while the head is operational, we quickly identified a bug. When a decommit is pending ( funds have not yet been withdrawn from the head) and a user attempts to create a new layer 2 transaction, the hydra-node attempts to re-apply the pending decommit to the local ledger state, resulting in an error.

We promptly realized that any pending decommit needs to match with the decommit in the new snapshot and should be preserved in the next snapshot(s) until it is observed. Without this fix, our decommit feature would be extremely fragile, so we made sure to quickly draft a failing test and do a proper fix.

This is a pull request that fixes this bug and we also demonstrated this fix in one of our monthly meetings.

Conclusion​

The monthly review meeting for August 2024 was held on 2024-08-28 via Google Meet, presenting these slides and this recording.

This month we saw demonstrations from both projects about new features and improved testing capabilities. Optimization results and test results underline the teams' commitment to evidence-based engineering. The Hydra Doom demo was a great success and we received good feedback from users and use cases.

While roadmaps have not moved much over the last couple of months, but more re-orientation is likely as plans for 2025 are coming together in the working groups and other Intersect committees.

This is a monthly report on the progress of 🐲 Hydra and 🛡 Mithril projects since June 2024. This document serves as a preparation for and a written summary of the monthly stakeholder review meeting, which is announced on our Discord channels and held on Google Meet. This month, the meeting was held on 2024-07-22 using these slides and you can see the recording here.

Mithril​

Issues and pull requests closed in July

Distributions​

2423.0​

We have released the new Mithril distribution 2428.0. This distribution includes several critical updates and enhancements:

• Support for Cardano node 9.0.0.
• Database cleanup and optimization once per epoch.
• Bug fixes and performance improvements.

Future distributions​

We have planned to release two new distributions in August:

• 2430 (before the Chang hardfork):
  • Activation of the certification of Cardano transactions in the release-preprod network.
  • Support for Cardano node 9.1.0 on all networks.
• 2434 (after the Chang hardfork):
  • Activation of the certification of Cardano transactions in the release-mainnet network.

Protocol status​

Here is the status of the Mithril protocol on the Cardano mainnet:

Transaction verification in Nami​

As Blockfrost now supports the Mithril endpoints within their API, they have presented an implementation example.

• The demo shows transaction verification within the Nami light wallet (using Blockfrost as its backend).
• This allows the light wallet user to verify that their transaction is on-chain with the high certainty provided by Mithril security.
• While it is just an experimental implementation, it might be accepted by the Nami upstream in the future.

Transaction certification​

We have continued working on the implementation of the Cardano transaction certification with Mithril and are currently preparing for the rollout of this feature to production:

• We are finalizing the minimization of the signer footprint on the SPO infrastructure.
• We are preparing for activating the feature in the release-preprod network with the following release.
• We are preparing thorough documentation about the mechanism used to certify the transactions
• We have selected final parameters for signing the transactions:
  • 100 blocks from the tip of the chain (~30 minutes).
  • 30 blocks between snapshots certifying the transactions (~10 minutes).
  • With these parameters, we can consider the transactions final with very high probability.
  • These parameters are hot parameters that can be adjusted without requiring a new distribution

Decentralized Message Queue CIP​

We have been collaborating with the Cardano networking team to design a mechanism for implementing a decentralized message queue to decentralize the signature diffusion from Mithril signers to Mithril aggregators based on the Cardano network layer. The idea is to create a separate process for the decentralized message queue implementation:

• A new dedicated process (aka "Mithril network node") based on the Ouroboros network stack
• More efficient handling of resource consumption (network and CPU) and reduced impact on the Cardano node performance and availability
• Enhanced security with no impact on the Cardano node
• Different release pace for both nodes
• Both nodes will be bundled in the Cardano node distribution release
• A hard fork for SPO registration of the port of their Mithril network node is likely needed.

Hydra​

Issues and pull requests closed in July

Snapshot of the new roadmap with features and ideas

Notable updates on our roadmap this month are:

• Completed incremental decommits, to be released separately as 0.18.0 after some remaining cleanup tasks and documentation updates
• Release 0.19.0 will be incremental decommits and, depending on the hard-fork date of Cardano mainnet, switching to Conway on the Hydra L2 ledger (see March 2024 for more details)
• Refined scope of "what is Hydra V1" through the Hydra working group
  • Added SDK for wallet integration with Hydra #1509 to provide a better developer experience
  • Decided to implement Directly open heads #1329 and Partial fanout to overcome our known issue & limitations and supersede several other 💭 ideas.
  • Prioritized Optimistic head closure #198 higher based on user feedback.
  • Declared React to protocol parameter changes #195 as not essential for a V1.

Incremental decommits completed​

This month we finally finished the implementation of the Incremental decommit #1057 feature.

As evident from the GitHub feature description, the number of comments and linked pull requests, this was a complex change.

Last month, we discussed why versioning the open state on-chain is necessary and the subsequent changes to the specification and implementation. After addressing these in hydra#1473 and in hydra#1474, a thorough review and requiring tests for all changes (compared to master) in hydra#1483 concluded in a last cleanup PR, which made the feature consistent in specification, implementation and documentation.

In the monthly review meeting we demonstrated the feature and the video below also shows how it works using a small extension to the hydra-tui:

Repository moved​

This month, the Hydra repository was moved to the cardano-scaling GitHub organization: https://github.com/cardano-scaling/hydra.

This allows for a more flexible use and sharing of GitHub runners between the main repository and for example hydra-explorer.

Most links should be automatically redirected, but if you encounter broken links, just reach out on Discord or create an issue 🙏 https://github.com/cardano-scaling/hydra/issues/new/choose

Working group updates​

There were two Hydra working group meetings this month and multiple people showed interest in joining. Current member list can be found in the charter and meeting notes of all meetings are kept in the repository too.

The working group started work on the first item "What is Hydra Head V1" and gathered valuable feedback in discussions between fellow layer 2 architects and Hydra users.

Notable inputs were the need for an SDK, wallet integration and the ability to operate without a full cardano-node, which got captured in hydra#1509 and hydra#1305 respectively. Even brand new ideas emerged from these discussions like "resumable Hydra heads".

The immediate goal is to achieve a common understanding of what must be included for a Hydra Head V1 and consequently when/how this could be audited?

Discussions also revolved around discoverability of Hydra heads, specifically, but also general interoperability with off-chain ledgers (any layer 2). In that regards, the group is considering to work on a CPS/CIP in the mid term.

Conclusion​

The monthly review meeting for July 2024 was held on 2024-07-22 via Google Meet, presenting these slides and this recording.

Despite the likely impact of summer breaks on attendance, it was great to have Blockfrost demonstrate their experiments using Mithril certificates delivered through Blockfrost in a forked Nami version. This demo clearly illustrates how light clients can benefit from Mithril verifiable transactions.

On the Hydra side, we are very pleased to have successfully merged and prepared the incremental decommits feature for release. Although this process took some time, it marks the first substantial feature not originally covered by the Hydra head paper, demonstrating the protocol's capacity to grow and meet user needs. We appreciate all the valuable input from the Hydra working group regarding the features we should develop next and are excited to see the growing community interest in Hydra's future.

This is a monthly report on the progress of 🐲 Hydra and 🛡 Mithril projects since May 2024. This document serves as a preparation for and a written summary of the monthly stakeholder review meeting, which is announced on our Discord channels and held on Google Meet. This month the meeting was held on 2024-06-24 using these slides and you can see the recording here.

Mithril​

Issues and pull requests closed in June

We have released the new Mithril distribution 2423.0. This distribution includes several critical updates and enhancements:

• BREAKING changes in the Mithril client CLI:
  • The deprecated snapshot command has been removed
  • It has been superseded by the cardano-db command
• Bug fixes and optimizations.

We have also created a developer blog post about the removal of the Mithril client CLI 'snapshot' command.

Additionally, we have released a draft version of the Mithril threat model analysis. We expect to receive external feedback and contributions before considering it final.

Transaction certification​

We kept working on the implementation of the Cardano transaction certification with Mithril and made good progress with the development:

• Transactions are now retrieved with the chain sync mini-protocol using the Pallas library
• The performance of the prover route of the aggregator has significantly improved.

Low latency certification​

The transactions are imported using the native chain sync mini-protocol of the Cardano node. This has been made possible with the Pallas library, which fully supports this mini-protocol. This allowed us to retrieve transactions much closer to the tip of the chain and at more frequent intervals. To achieve this, we have enhanced the transaction importer to handle chain rollbacks, where some blocks previously recorded may be discarded at a later time, with an increased probability as we get closer to the tip of the chain.

We are currently calibrating the system parameters (depth from the tip and certification pace) before activating it on the mainnet.

Increased prover performance​

We have identified some bottlenecks in the computations of the merkelized proof for transaction membership of the transaction set as well as in the database access. These bottlenecks have been fixed, allowing us to achieve a new order of magnitude in the performance of the prover route. It is now able to consistently deliver 10,000 transaction hashes certification per second.

Mithril/Cardano integration​

We have been working with the Cardano networking team to design a closer integration of Mithril into Cardano. In particular, we have focused on decentralizing the signature diffusion from Mithril signers to Mithril aggregators based on the Cardano network layer.

Here is the current roadmap:

• Assessing Cardano network bandwidth capacity to support Mithril signature diffusion
• Designing a node-to-node mini-protocol for signature diffusion
• Designing two node-to-client mini-protocols for signature submission and notification
• Preparing the submission of a CIP draft to the community.

Hydra​

Issues and pull requests closed in June

Here is a high-level overview of the team's work over the past month:

• Incremental decommits. We continued our work on incremental decommits, focusing on investigating adversarial attacks on the head script contract.
• Specification update. We updated our specification with the latest changes following our exploration of adversarial attacks. These updates aim to enhance the security of the protocol.
• Agda specification for the Hydra protocol. Work has begun on writing an Agda specification for the Hydra protocol. You can follow the progress here.
• Incremental commit. We discussed various options for the incremental commit work, including potential designs for a hydra-ledger to improve system efficiency and security.
• Node compatibility. The node has been successfully updated and is now compatible with cardano-node version 8.11-pre.
• Website. We updated our website tutorials and documentation to enhance expressivity and ease of use.
• Committing internal wallet UTXOs. We made breaking changes to the hydra-node API /commit endpoint #1463 to remove the check that prevented committing UTXOs from an internal hydra-node wallet.

We have not planned the next release, so our roadmap remains largely unchanged. Our current focus is on incremental decommits and commits and specifying our protocol in Agda.

The latest roadmap with features and ideas

Incremental decommit​

We were about to release the incremental decommit feature when we discovered a bug during a demo. This bug related to handling the decommit snapshot in close/fanout. When one decided to decommit funds from the head and everyone signed the decommit snapshot, the same snapshot was used to close/fanout. If this snapshot contained some UTXOs to decommit but one already decommitted funds from the head, the fanout would distribute the decommit funds again.

This issue was significant, and although we have many tests around the decommit feature, we had not checked user balances upon head closure.

We have implemented a fix that involves adding versioned snapshots. This allows us to distinguish between two cases:

• The decommit snapshot was signed, but the decommit was never observed
• The decommit snapshot was signed, and the decommit was also observed.

When we observe the decommit transaction locally, we increment the version in the local state. This allows us to compare the snapshot version against the expected local version on-chain.

We also needed to keep extra data in the Close redeemer to successfully check for a valid snapshot signature while altering the close datum to remove the UTXOs to decommit if the versions do not match. This ensures that the fanout behaves correctly, as it is only concerned with the close datum and the actual transaction output hashes.

The remaining work involves updating the specification with the changes in the implementation. We expect to release this feature in the coming days.

Community​

Subbit.xyz​

Algy Wallis presented 'Subbit', a layer 2 solution focused on subscription-based services, aimed at addressing the complexities of building blockchain products. Subbit is designed to be easier to integrate and manage compared to layer 1 solutions, reducing latency and transaction fees. The system allows providers to verify consumer IOU notes, facilitating periodic payments and withdrawals with minimal on-chain interactions. This model mirrors Bitcoin's Lightning Network and aims to enhance scalability and interoperability for subscription services on Cardano.

Hydrozoa​

George Flerovsky introduced the Hydrazoa proposal, currently in Catalyst Fund12. Hydrazoa aims to enhance the existing Hydra framework by adding dynamic capabilities for incremental commits and decommits, along with the ability to add or remove participants in state channels while they are still open. The goal is to simplify layer 1 scripts and lower costs, with fallback mechanisms similar to Hydra's in case of layer 2 consensus failure. George highlighted the differences and potential integrations with Hydra, emphasizing a more flexible and simplified approach to state channel management.

Technical working groups​

Reza Baram provided an update on the Hydra working group, which recently had its first meeting to discuss its vision and collaborative processes. The primary goal is to create an environment for sharing knowledge and working towards scaling Cardano beyond just Hydra. The group is currently focused on defining and delivering Hydra version one, prioritizing incremental commits and robustness improvements. Feedback from the community and working group members will be sought to refine and enhance the roadmap, ensuring effective scaling solutions for Cardano applications. The Hydra working group is open to new members and contributions from the community. Read more here.

Conclusion​

The monthly review meeting for June 2024 was held on 2024-06-24 via Google Meet, presenting these slides and this recording.

This is a monthly report on the progress of 🐲 Hydra and 🛡 Mithril projects since April 2024. This document serves as a preparation for and a written summary of the monthly stakeholder review meeting, which is announced on our Discord channels and held on Google Meet. This month the meeting was held on 2024-05-28 using these slides and you can see the recording here.

Mithril​

Issues and pull requests closed in May

We have released the new Mithril distribution 2418.1. This distribution includes several critical updates and enhancements:

• A breaking change introduced in the Mithril client / Mithril client CLI:
  • The certificate chain structure has been modified to remove coupling with the immutable file number
  • The client must be updated to verify the certificate chain
• We have switched the memory allocator to jemallocator on the signer and the aggregator to avoid memory fragmentation
• We have enabled the BLST portable feature by default to benefit from runtime check of intel ADX instruction set.

Also, we have started the process of removing the deprecated snapshot command from the client which is superseded by the cardano-db command. This change will be effective with the release of the next Mithril distribution.

Transaction certification​

We have kept working on the implementation of the Cardano transactions certification with Mithril and we have achieved a new milestone with running the certification in a test network (aka testing-mainnet) which operates on the Cardano mainnet.

This allowed us to refine our roadmap with clear objectives before the MVP can be released:

• Keep working on the low latency implementation of the certification
• Optimize the aggregator prover route to reach good throughput
• Optimize the signer footprint to limit the impact on the SPO infrastructure
• Optimize the signer and aggregator warmup to avoid network instability/disruption once the MVP is released.

A first optimization of the prover route has been implemented, drastically improving performance. By implementing resource pooling on the Merkle tree that signs transactions by block range, we have achieved a 100x increase in throughput.

We have also implemented a pruning mechanism on the signer that retains only the transactions needed to compute upcoming signatures. This has reduced the storage requirements from 32GB to 100MB.

In/Out SPOs dashboard​

We have added a new page to the Mithril explorer that lists the newly registered and recently de-registered signers in the Mithril protocol.

The protocol insights dashboard​

A new Mithril protocol insights dashbord has been released. It provides metrics such as the number of SPOs and total stake involved, the daily number of Cardano database restorations, and the breakdown of running signer versions.

Feel free to request some new metrics on the dashboard!

Hydra​

Issues and pull requests closed in May

This month, Hydra version 0.17.0 was released which includes

• breaking changes of the /commit endpoint, yielding full flexibility when committing from scripts (see this how-to for details)

• new GET /snapshot/utxo HTTP endpoint

• detecting network protocol mismatches

• Full release notes and a list of delivered features

Besides corresponding features being released through 0.17.0, the roadmap did not change much this month. Current focus is on incremental decommits and commits, while we also wait for the Conway hard-fork to happen before we can switch the L2 ledger also to Conway (see March report for more details).

The latest roadmap with features and ideas

Transaction trace testing​

In the process of implementing the incremental decommits feature, we constantly considered how to ensure our test suite covered all potential scenarios. For example, what happens if we request a decommit, have a signed snapshot, but never post a decommit transaction? Or how do we handle close and fanout with decommit UTXO in the snapshot? There are various scenarios we are interested in testing, and we quickly realized that we would need to test at different levels to achieve the coverage we expect, especially for our on-chain code.

Our mutation test suite was also very helpful in testing whether our on-chain code produces the expected errors when manipulating the transaction to make it invalid. However, this framework tests individual transactions, and we needed to test if multiple transactions in sequence produce valid results.

To address this, we decided to utilize quickcheck-dynamic once again to model simple snapshots, which are used to construct Hydra transactions. Our hope was that if the model is simple enough, we would be able to reason about it without delving into many details and decide if a particular UTXO/snapshot should be valid in the current transaction context or not. In the quickcheck-dynamic world, our 'Action' contains snapshots, which are then used to construct different transactions, evaluate, and observe them. Our model keeps the extra information we need to evaluate these transactions, and all snapshots are signed by everybody to keep things simple.

This test suite has already exposed some bugs we had and enabled us to reason about our code using a simpler model without burdening us with real-world details, which are more involved and harder to reason about.

Hydra-related Catalyst proposals​

Catalyst Fund12 is in full swing, with over 30 proposals this round mentioning 'Hydra' in one way or another. While some directly draw inspiration from Hydra and tackle similar challenges, others aim to enhance transaction processing scalability but take different approaches (such as rollups and sidechains). Additionally, some proposals focus solely on scaling payment transactions, offering refreshingly simple solutions.

After a quick screening, we also engaged with some of the proposal authors this month to address questions about Hydra, discuss similarities, and explore opportunities for mutual learning and collaboration:

Use cases​

• Wine Supply Chain Tracking and Reporting System. Great use case that requires the seamless integration of recording data as transactions off-chain, with resulting states and tokens accessible from the Cardano layer 1. The team is already developing a proof of concept using Hydra, and this proposal aims to advance it into a minimum viable product (MVP).

• Ikigai + MLABS - Grabbit / Hydra auctions: Running auctions within Hydra heads is a popular use case, allowing for fast and cost-effective bids on layer 2, while the final purchase of auction items seamlessly occurs on the Cardano layer 1. This proposal, in particular, will utilize incremental commits and decommits to facilitate fully-backed auctions on layer 2.

Using / extending hydra​

• Hydra-Enabled Micropayments System. The same team behind the wine supply chain tracking system wants to also provide a decentralized payment system that offers great point-of-sales experience using Hydra at its core. While they aim to use Hydra Head as a core component, this would also be a great use case for the more payment-focused variants in the long run (see below).
• Open-source Hydra Head L2 Web-based Explorer and API. Improving the visibility of what's going on in a Hydra Head is a great idea. Besides exploring what heads are available on the network, this would make transactions inside a head more visible and is very much related to this idea on our roadmap.
• Cardano Layer 2: Hydrozoa protocol for lightweight and flexible Hydra Heads for Cardano. An exciting re-interpretation of the core Hydra Head protocol and ambitious simplifications of roadmap features like directly open heads, incremental de-/commits and optimistic head closure. This was also discussed in the past with core contributors, and we would love to explore how those protocol variants can find their way into the upstream Hydra Head protocol.

Alternative scaling approaches​

• Subbit.xyz : Cardano's featherweight L2. Refreshingly simple take on how to process payments off-chain. Very much inspired by lightning payment channels and maybe even Chaumean eCash systems. Having payment channels that work in such a simple and flexible way is exciting and could be a great basis for any payment channel networks, ie Cardano Lightning.
• µgraph: Instant, Untraceable Payments in Cardano. Ambitious, but promising, take on what the Hydra tail could be – an asymmetric construction that benefits individual users by using zero-knowledge proofs to ensure correct operation of the node.
• Anastasia Labs Cardano Layer2 - Midgard. A layer 2 construction, similar to Hydra Head, that seeks to utilize the Cardano ledger isomorphically, but operates with a limited validator set and employs a distinct consensus mechanism to offer an experience akin to sidechains. We always wondered what you would get if you ran a Hydra Head with a non-total consensus and checkpointing state on layer 1!

Technical working groups​

As the age of Voltaire unfolds, it fosters the formation of community alliances and member-based organizations. At the same time, there's a popular belief that governance over core Cardano projects should become more open, with decisions regarding the inclusion or exclusion of elements on the roadmap of core infrastructure being transparent.

The current strategy devised by Intersect and its members to address this involves establishing technical working groups that operate openly and transparently on roadmaps and high-level backlog items. This approach mirrors what we're already doing here (see also last months starmap update section), but with a more concrete democratic approach to ratifying a roadmap. Presently, roadmaps for both Hydra and Mithril projects are expected to be governed in this manner.

Specifically for the Hydra working group, initial steps were taken this month by drafting a barebones charter and brainstorming first agenda items to engage potential members and get first sessions going as we gather more interested stakeholders. The charter and any future outputs of the working group will be hosted on cardano-scaling/wg-hydra and we also registered the working group at Intersect.

Conclusion​

The monthly review meeting for May 2024 was held on 2024-05-28 via Google Meet, presenting these slides and this recording.

May was a short month because of the public holidays and some people being off, but we still made good progress on both projects. The new dashboards for Mithril will visualize existing participation and surely garner more interest in the protocol. We are also excited about the progress on the Cardano transactions certification with Mithril, especially as multiple projects building bridges or layer 2s are keeping a close eye on this feature.

While we are busy on the Hydra side wrapping our head around incremental decommits and ensuring they are correctly implemented through transaction trace testing, we are also excited about the upcoming Catalyst Fund12 proposals and the discussions coming along with it as others are also looking into building similar, yet different scaling solutions and tackle the same challenges.

We are also making progress on making Hydra and Mithril less IOG-driven, but more community-owned through working groups and Intersect. We are excited to see how this will turn out.

This is a monthly report of progress on 🐲 Hydra and 🛡 Mithril projects since March 2024. This document serves as a preparation for and a written summary of the monthly stakeholder review meeting, which is announced on our Discord channels and held on Google Meet. This month, the meeting was held on 2024-04-30, using these slides and you can see the recording here.

Logo design work​

During the months of March and April, we focused on refreshing the logos of Hydra and Mithril. We also created a separate logo for the shared workstream aimed at scaling Cardano. It's crucial to have a logo for this workstream as it will help bring various projects together to enhance the scalability of Cardano.

Latest logo drafts for Hydra, Mithril and Cardano Scaling work group

Above, you can see the latest logo drafts created by our designer, Alexander Wende. Let us know your thoughts through any of our communication channels like Discord or Twitter.

Mithril​

Issues and pull requests closed in April

As a part of our strategy to increase the participation in the Mithril protocol we have been reaching out to SPO's who don't yet run a Mithril signer, providing them with the support they need to set up and run their signer. We appreciate the SPO's such as Cardano Foundation who took the step in joining the protocol. As of epoch 481, the number of stake pools registered in the Mithril protocol as signers has reached 258. This represents approximately 5 billion ADA, which is approximately 22% of Cardano's active stake.

We have prepared the pre-release of the new Mithril distribution 2418.1-pre. This release includes several critical updates and enhancements:

• A breaking change is introduced in Mithril client / Mithril client CLI:
  • The certificate chain structure has been modified to remove coupling with immutable file number.
  • The client must be updated to verify certificate chain.
• We have switched the memory allocator to jemallocator on the signer and the aggregator to avoid memory fragmentation.
• We have enabled the BLST portable feature by default in order to benefit from runtime check of intel ADX instruction set.

Signer registration decentralization PoC​

We have kept working on the decentralization experiments for Mithril networks. The signature diffusion decentralization has already been implemented in a previous proof-of-concept, and we have created a new proof-of-concept for decentralizing the signer registration process.

In order to do so, we have created a new peer-to-peer pubsub topic using libp2p and adapted the relay so that:

• When a signer relay receives a signer registration from a signer, it broadcasts it to the new P2P topic.
• When an aggregator relay receives a new signer registration from the P2P topic, it calls back the aggregator with it.
• A repeater mechanism has been implemented so that a signer relay sends the signer registration multiple times during an epoch (at regular pace) to prevent missed messages.

We don't need strong guarantees or consensus for the diffusion of signatures: different aggregators can create valid multi-signatures from different subsets of individual signatures as long as the quorum is reached. We could also implement the repeater mechanism so that a signature is broadcast a few times in order to prevent missed messages and reach more aggregators.

However, we need stronger guarantees for the signer registration: in order to create a valid multi-signature a minimum number of signers and aggregators must use exactly the same set of signer registrations (enough signers and aggregators so that the individual signatures created allow to reach the quorum). In case of a partition, the network could get in a situation where the quorum is never reached during an epoch: this would lead to a gap in the certification and the security of the protocol could not be guaranteed. Adversarial behaviors from some signers could for example be responsible for such partitions. Moreover, with the current design, as the number of aggregators is much lower than the number of signers, the risk of partition could be even higher.

This will lead us to:

• Reach for a consensus or quasi-consensus for the signer registrations.
• Adapt the topology of the network so that there are approximately the same number of signers and aggregators (i.e. all signers are also aggregators) or modify the aggregation process so that aggregators don't verify the message that is certified.

The next steps for the decentralization of Mithril networks are:

• Draft a CIP for leveraging the Cardano network layer and implement mini-protocols that will operate the diffusion of messages in a pubsub topic.
• Draft a CPS for stating the problem about signer registration.
• Get some feedback from the community, feel free to contribute!

Hydra​

Issues and pull requests closed in April

This month, the roadmap received updates to reflect an upcoming 0.17.0 release, which will include several API updates and two notable features:

• Revise POST /commit endpoint interface #1350 and
• Use versioned handshake when connecting hydra-nodes #1010

The latter was distilled out of the much broader and long-standing Hydra node software, scripts & protocol updates #191 as it was the last actionable piece of the included scenarios on how to handle change. Having this will make the hydra-node detect incompatible network protocols and improve robustness of the node implementation.

The latest roadmap with features and ideas

Besides feature ideas which are to be discussed and refined in the upcoming weeks, our focus lies on the two major features of incremental commits and decommits are currently being worked on by the team.

Why are incremental commits hard?​

Both, Incremental decommit #1057 and Incremental commit #199 turned out to be non-trivial and are in progres still.

While the plan is clear by now (see issue comments for a history), changes to the protocol are substantial and can be considered literally Hydra Head V2 (but any change to the scripts is a major change). Currently, the off-chain workflow is mostly implemented end-to-end, but the on-chain part is still in the works and requires a lot more testing.

As these features will allow to update the Open state on-chain, rollbacks and interleavings of so-called increment, decrement and the close transactions need to be considered. This is a complex problem that requires a lot of testing and validation to ensure that the protocol is secure and correct.

In particular, we need to be testing (ideally proving) that:

• funds already commited / not yet decommitted are always returned (completeness)
• funds are never returned twice (soundness)
• the head can process transactions throughout (liveness)

Our approach to testing is similar to when we implemented the basic protocol: end-to-end tests ensure overall integration, while property-based mutation tests ensure individual validators work correctly. This time, however, we are also looking at stateful property-based testing of many sequences of transactions (see this task and this module on the feature/incremental-decommit branch).

Hydra /commit endpoint enhancements​

We are about to release a refactor on the /commit endpoint which provides greater control for drafting commit txs, allowing for adjustments to the transaction context, which is crucial for real-world scenarios involving transaction validity checks, required signers, etc.

This refactor changes the endpoint to now accept a "blueprint" tx, CBOR-encoded, along with the UTxO, JSON-encoded. The UTxO is used to resolve inputs spent in the transaction, and the "blueprint tx" serves as the foundation for creating the resulting drafted commit tx, retaining anything that complies with the requirements of a valid head commit transaction.

This enhancement was initiated following a discussion led by the MLabs team, as documented here. The need arose to unblock a sub-validator utilized in the hydra-auction for checking spending conditions to move a standing bid to L2. The discussion was promptly converted into an issue and subsequently added to our roadmap after internal grooming by the Hydra team.

While this refactor represents a breaking change, the hydra-node can still be utilized as before if the provided UTxO is at a public key address. However, to spend from a script UTxO and unlock more complex use-cases, users must provide an additional unsigned transaction that accurately specifies required data, such as redeemers and validity ranges.

Cardano Buidler Fest​

This month the who-is-who of Cardano builders got lured to Toulouse, France for a two-day tech-only conference. It was a great occasion to meet fellow contributors and build bridges (literally) to new projects and their creators.

Two of many notable presentations and how they relate to the Hydra and Mithril projects are detailed as contributed by Pi (Sundae Labs) and Sebastian (Cardano Foundation).

IBC bridge​

Cardano Foundation has started to work on an implementation of the Inter Blockchain Communication protocol (IBC) to allow builders to connect other blockchains to the Cardano network that also implement IBC like e.g. Cosmos SDK or Substrate based blockchains. Whereas IBC has many similarities with other communication protocols like e.g. TCP and is based on the pretty straight forward delivery of data packets from one blockchain to the other by so called Relayers the general security model is based on the assumption that the consensus of both blockchains to be connected can be trusted. The latter resolves in the challenge that so called Light Clients need to be implemented that allow one network to check the validity of the consensus of the other, to verify if a specific operation like e.g. the locking of certain assets has been done as communicated via IBC. This typically implies that a blockchain network is able to generate so called state proofs that can be presented to the Light Client which then is able to perform the validation.

Unfortunately, the Cardano blockchain does not generate such state proofs as the underlying architecture and Ouroboros consensus do not rely on such a primitive. This leads to the the challenge that implementing a Light Client purely based on available data from the consensus (e.g. block data) has several shortcomings with regards to security or if traded for it liveness of the client causing significant latency in the message delivery between both networks (as the Light Client would have to wait until a sequence of blocks is presented for validation), which is not a feasible option for a real world production deployment.

However, there are other networks out there with a similar setup in that specific regard that recently claimed to have implemented IBC like e.g. Solana. Their approach is based on overlay networks that take care of collecting additional signatures from network validators for state proof data (like blocks, transactions, etc.) increasing the trustworthiness of those proofs. Fortunately, there is a similar solution currently under development in the Cardano ecosystem, which is Mithril.

The Cardano Foundation Engineering team has recently concluded the first phase of the IBC implementation demonstrating the technical feasibility of connecting Cardano to a Cosmos SDK based blockchain taking the disadvantages mentioned above into account. The work has been made public in a repository that can be found here: https://github.com/cardano-foundation/cardano-ibc-incubator

In a second phase that is scheduled to run from April till roughly beginning of July the team is working on replacing the existing supposedly weak Light Client implementation of the Ouroboros consensus on the Cosmos based blockchain with a Light Client that leverages the functionalities of Mithril. As a first approach we are intending to use specifically a recently introduced method that allows to check if a specific transaction is part of the global set of transactions of the network.

Gummiworm​

Gummiworm is a low-divergence fork of the Hydra node being built by Sundae Labs, which teases apart execution and custody; By sacrificing some control over the opening and closing of heads, it dramatically strengthens the security model specifically in the case when non-head participants are transacting in the head.

At Buidler-fest, we demo'd a very early, duct-tape and bubble-gum demo of depositing into a smart contract, minting those funds in the hydra head, and achieving auditability of the transactions in the head.

We also spoke about our longer term plans, such as changes to the ledger to accomodate a better user experience, and making the custody model "plug and play", so you could provide different mechanisms for different use cases.

Starmap update​

Every now and then, roughly quarterly, we like to take a step back from individual project roadmaps and think on an even higher level of what should be the priorities for the next couple months.

This time, the themes we would like to focus on are open governance, use cases and adoption, as well as measure and visualize. Across Hydra and Mithril we believe the following items, which ideally are backed by technical features on individual roadmaps, are suitable next steps:

As one can see from past quarters, opening up the governance of scalability projects is a recurring theme and of highest priority. Ironically, the proposed work items above were mostly shaped and prioritised in private discussions between IOI and PaloIT. We will need to open discussions of what is valuable to build, to gather feedback and ensure individual projects tie into a compelling and fundable portfolio across the whole ecosystem.

A first glimpse of this already happened at the Buidler Fest where a spontaneous (open space) session on "the future of Hydra and Mithril" spawned great interest and new ideas like

• Shallow node would be interesting
• Once per epoch Mithril-signed ledger state
• BLS accumulators for Hydra partial fanout; also relevant for Mithril?
• The need for another audit of latest Hydra Head protocol

As also listed on the starmap, a logical next step is to create an open working group with interested experts from the Cardano community. A working group to discuss whether and how a Hydra Tail protocol should be built out? What it would entail (pun intended)? Built by whom? How would it relate to Gummiworm, Cosmex, Subbit, zkFold, ... which should maybe prioritised by the Cardano community instead?

Conclusion​

The monthly review meeting for April 2024 was held on 2024-04-30 via Google Meet, presenting these slides and this recording.

While we saw some demos on Mithril and Hydra projects' progress, the real highlight this month was the Cardano Buidler Fest with its many great talks and discussions - often involving scalability limitations and technology to overcome it. Exciting discussions around zero knowledge primitives, infrastructure projects, and open source likely inspired everyone attending to keep building and improving Cardano.

There is still some uncertainty in how exactly governance of Hydra and Mithril can be made more open, collaborating on consistent high-level plans across multiple projects in the realm of improving Cardano scalability. However, we might have found with cardano-scaling.org a welcoming home not only for Hydra and Mithril, but for contributors and projects from various backgrounds, while first steps of involving the community more actively in a Hydra working group at Intersect are being made.

Besides this, the monthly review meeting held some sneak peeks of a new paper on fresh scalability tech (Ouroboros Leios) and fun conversations on games and other great ways to potentially show off a definite Hydra V1 at Rare Evo this summer - "Can it run doom?" 😈.

This is a monthly report of progress on 🐲 Hydra and 🛡 Mithril projects since February 2024. These reports have recently moved to https://cardano-scaling.org/monthly (more details here).

This document serves as a preparation for and a written summary of the monthly stakeholder review meeting, which is announced on our Discord channels and held on Google Meet. This month, the meeting was held on 2024-03-27, using these slides and you can see the recording here.

Mithril​

Issues and pull requests closed in March

This month, there has been a noticeable increase in SPOs joining the Mithril protocol. In particular, many new multi-pool operators have set up the Mithril signer in their infrastructure. At the end of epoch #475, there were 230 pools registered with 4.6B₳ stake involved representing approximately 20% of the total active stake. We will continue to provide regular updates to the community to encourage their participation in the protocol.

We have released the new Mithril distribution 2412.0. This release includes several critical updates and enhancements, such as support for Prometheus metrics endpoint in signer, deprecation of the snapshot command in the client CLI, full Pallas based implementation of the chain observer, and support for Cardano node version 8.9.0.

Transaction signatures​

We have developed a signing structure that enables quick proof generation while maintaining a light footprint on the signer for the mainnet. This is crucial as we need to handle a vast amount of transactions (100 million). To achieve this, we created a block range Merkle tree, which stores transactions in a Merkle tree and uses its root to create another Merkle tree. According to the our benchmark tests, a batch proof for 100 transactions can now be created in less than 250 ms.

We have also completed the roadmap of the MVP that will be released on mainnet:

Transaction verification in frontend​

The Mithril client WASM library has been used to implement a Cardano transaction verifier inside the Mithril explorer. This feature is currently available only on the testing-sanchonet network and will be progressively rolled out to the other Mithril networks. Wallet and DApp developers could leverage the client WASM library to implement a security layer that verifies transactions provided by a third party.

Prometheus metrics​

We have released a new Prometheus endpoint with the latest stable version of the signer. The feature can be easily activated by the SPOs as explained in this developer blog post.

Additionally, we created a Grafana template to easily set up a dashboard for this Prometheus endpoint: https://grafana.com/grafana/dashboards/20776-mithril-signer/

Pallas updates​

The latest stable distribution now includes a fully implemented chain observer powered by the pallas crate. Additionally, we have been collaborating with TxPipe to implement the chainsync mini-protocol with Pallas. This will significantly reduce the latency of signatures for Cardano transactions.

Deprecation warning​

We have deprecated the snapshot command of the Mithril client CLI as explained in this developer blog post.

We recommend using the replacement cardano-db snapshot command for users of the client CLI.

Hydra​

Issues and pull requests closed in March

Hydra project roadmap did not change much this month:

The latest roadmap with features and ideas

Notable updates​

• Evolved the user-requested feature idea #1337 into a new feature on the roadmap to revise POST /commit endpoint interface #1350. This will allow committing into a head directly from advanced smart contracts. It remains unclear whether this will be part of the upcoming 0.16.0 release.

• Captured the new feature idea of Directly open heads #1329 as item on the roadmap. This requires further grooming and could supersede other ideas like Always abortable head #699.

• Mostly completed Drop support for JSON encoded transactions #1213 and Streaming plugins #1325, which will be released soon as 0.16.0.

• Made substantial progress on Incremental decommit #1057, however, this is unlikely to be released as part of 0.16.0.

• Fully prepared Switch L2 ledger to Conway #1178, such that it can be released directly after the hard fork on the Cardano layer 1.

• As we improve our understanding of pivotal features such as incremental commits and decommits, the feature set required to address [known issues and limitations] (https://hydra.family/head-protocol/docs/known-issues/) is being narrowed down, and the scope for a convincing version 1.0.0 is becoming clearer.

Conway support​

As mentioned in previous roadmap updates, we separated work to support the upcoming Conway era on Cardano in two:

• Conway support #1177 will make the hydra-node support Conway and allow users to keep heads open across the hard fork.
• Drop Babbage support #1178 will eventually drop support for the Babbage era and retain maintainability.

While the former was already merged and is currently released in version 0.15.0 onwards, the team prepared the latter this month.

Ideally switching the transaction format in the hydra-node is as simple as setting the type alias type Era = Conway. However, the code was Babbage-specific in several areas and work on this resulted in various pull requests making switching easier in several places: #1326, #1327, #1328, #1334, and #1342.

The result is a relatively simple switch in this draft pull request and available on branch lc/conway-support, which we intend to bump and merge once the public Cardano networks hard fork.

Having full Conway era support, including PlutusV3, on the Hydra layer 2 already prepared on a branch allows early adopting users from the community to explore and use this already. Modulo-P has been on the forefront here by developing zero-knowledge hybrid DApps and wrapping the Hydra branch up in this example repository.

Streaming plugins​

SundaeLabs is working on a Hydra-based layer two protocol (gummiworm) that requires the usage of hydra-node components in a different way than the default setup. As part of this development, SundaeLabs submitted a Fund10 Catalyst proposal to enhance the Hydra node by adding features that would facilitate not only their work but also the work of everyone building on Hydra.

They have been working on a plugin architecture to enable the infiltration and exfiltration of events, which enables hygienic forks of Hydra that feature additional connections to the outside world. They are interested in consuming the Hydra event stream via Amazon Kinesis and storing transaction archives on S3 for later reference.

After an initial design phase resulting in ADR29, they contributed the event source and sink abstractions, and refactored the Hydra node to use these new extension points together with core contributors. The EventSource and EventSink interfaces, along with the configurable list of eventSinks in the main HydraNode handle, allow to extend the hydra-node with any external system. It also paves the way for more scalable persistence methods on the mainline Hydra node, and might even allow refactoring the existing API server as an EventSink.

Community​

Mithril office hours​

As the Catalyst project of TxPipe on Mithril is completed, we want to build upon this initial maintenance work in a more open format. For this, we are transforming our alignment calls into a public weekly office hour where we invite the community to join and discuss concrete pull requests, issues and questions on Mithril.

👉 Mithril office hours happen every Tuesday at 16:00 UTC on the #mithril-live Discord channel.

Hydra governance walkthrough​

This month, the Hydra team has been asked to provide a walkthrough of how we use GitHub to run and govern the project. We have prepared a video that explains how we use Github issues, pull requests, discussions, and projects to manage the development of the Hydra project. However, it is important to note that this information represents only a current view and we are enthusiastic about continuously improving the processes to make them more transparent.

We are excited to announce that we are looking to establish a Hydra working group at Intersect. This group will aim to involve more community members in proposing feature backlogs, roadmaps, and RFPs. Initially, the focus of this working group will be on the concrete Hydra Head protocol (state channels). However, variations such as Hydrozoa, and combined scalability protocols like Gummiworm, Lightning-like payment channel networks, or the Hydra Tail protocol (~ rollups) will also require more community involvement. Stay tuned for more updates on this in the upcoming months.

Conclusion​

We held the monthly review meeting for March 2024 on 2024-03-27 via Google Meet, presenting these slides and this recording.

This month we have made significant progress on Mithril transaction signatures, while the Hydra team has been busy working on the incremental commits/decommits features. Both projects are fully prepared for the upcoming Conway hard fork on Cardano and continuously demonstrate user experience improvements.

Besides the Hydra explorer user interface, the teams also showcased the Mithril transaction verification feature in the Mithril explorer frontend. This demonstrates that Cardano transactions can be additionally verified using Mithril certificates from within a web browser. Taking this even further, we believe that such a "Verified with Mithril" checkmark can provide additional level of confirmation for users of light wallets and Cardano DApps!

Besides these, Modulo-P showcased their early adoption of Hydra heads using Conway and PlutusV3 for their quite involved zero-knowledge protocols. Briefly after the meeting they released a first look on their zkMastermind DApp. Also, SundaeLabs presented their streaming plugins for Hydra. Both are examples of how the Hydra protocol can be extended and used in different ways, and we are excited to see more community contributions in the future.

As a closing remark, we see the trend of decentralizing governance on both projects, Hydra and Mithril, while at large the overall 'age of Voltaire' is unfolding. Consequently, we anticipate moves, merges, or restructuring of repositories, projects, and working groups. What an exciting time to be a part of this big Cardano community!

This is the first joint report of the 🐲 Hydra and 🛡 Mithril projects, which summarizes the teams' progress since January 2024.

Previously, both teams had been releasing progress updates in different formats and intervals within their respective repositories:

• Hydra monthly reports
• Mithril dev blog

Both sites are created from the individual project repositories. However, it is not ideal to keep 'procedural' documents as part of the code base, since it requires special handling when building versioned, published documentation. For example, this workflow in the Hydra project which publishes to https://hydra.family required several attempts to work around how Docusaurus websites are built. Consequently, we introduced a dedicated repository that builds into a website. This not only helps keep the original product repositories clean of inherently dated information, but also serves as a new home for joint reports, announcements, or occasional articles.

This monthly report serves as preparation for the monthly stakeholder review meeting (see slides and recording). At the meeting, the teams update stakeholders on recent developments and gather feedback on proposed plans.

Mithril​

Issues and pull requests closed in February

Release and publication​

This month, the Mithril team released distribution 2408.0 and published two blog posts:

• Mithril SanchoNet network released.
• Mithril client CLI output breaking change.

Signing Cardano transactions proof of concept​

The team has been working on implementing a proof of concept of a new data type that will enable the certification of Cardano transactions within Mithril networks. Last month, the team extended the Mithril client library to support the certification of Cardano transactions. This month, the team have upgraded the client CLI and introduced a new cardano-transaction command that can list snapshots and certify a list of Cardano transactions. Additionally, they have implemented the same API surface on the WASM client that now supports transaction certification in the browser.

Enhanced support for Conway and SanchoNet​

The team has conducted thorough testing on the upcoming Conway era and are confident that Mithril networks will transition smoothly to this new era. Additionally, the team have released a new testing-sanchonet network that has been opened to SPOs to enable collaborative testing of the upcoming and experimental features (eg the aforementioned signature of the Cardano transactions).

Hydra​

Issues and pull requests closed in February

This month, the Hydra project roadmap underwent a few changes:

The latest roadmap with features and ideas

Notable updates​

• New: Drop support for JSON encoded transactions #1213 due to breaking changes when upgrading cardano-node and cardano-ledger versions.
• Smoke tests on SanchoNet #1257 took longer than expected, revealing issues, but seeing smoke tests run on SanchoNet makes the team confident the hydra-node is ready for the Conway hard fork
• Build and deploy a Hydra heads explorer #696 is around 50% complete. Data is available for sanchonet at http://explorer.hydra.family/heads but a user interface is still needed.
• Started working on Incremental commit #199 as the UX will be similar to Incremental decommit #1057. Implementing both features off-chain allows early adoption while improving on-chain security.
• Added the Streaming plugins #1325 feature to cover SundaeLab's work on their Catalyst proposal

Contest after fanout bug hunt​

While working on hydra-chess, the team encountered a bug in the Hydra node. In the two-party Hydra Head, nobody was able to perform a Fanout. This happened because one of the nodes was in the initial Snapshot state and missed the request for a new Snapshot, while the other node already had one confirmed Snapshot locally.

When the Close was posted, the Head was closed with the initial Snapshot and then contested by the node that saw a newer Snapshot. After the contest, the team observed a strange behavior where both nodes could not Fanout. We expected this failure in the node that only had the initial Snapshot since it did not have the correct UTxO locally to Fanout. However, the failure in the other node with the newer Snapshot should not have happened.

While reviewing the code, the team discovered that the Contest observation was not extending the deadline after the contestation was made. As a result, the Fanout transaction was invalid because the validator checks that the Fanout transaction's lower validity bound is higher than the deadline to contest. This check ensures that one can only fan out after the contestation period is over.

Before fixing this bug, the team tried reproducing it using the MBT (Model Based Tests) framework, one of the tools in the test suite. However, to do that, they needed to complete modeling the entire head lifecycle and the specific scenario they observed in real life and arbitrary actions in all head states did not reveal this bug.

After also improving the framework itself to correctly validate transactions and a fix for simulating rollbacks, the team was able to fix the bug easily. However, it proves that being able to reproduce a bug is not only improving the code, but also allowed the team to gain knowledge and improve their test suites.

Hydra explorer supporting multiple versions​

This month, the team deployed the hydra-explorer to serve data under http://explorer.hydra.family/heads. Alongside this, they upgraded the Hydra scripts and switched to the latest pre-release version of cardano-node for Conway, and the system is currently running on the sanchonet network.

The Hydra explorer currently has the capability to support only one version on a single network. The reason for this limitation is that the Hydra scripts are compiled into two binaries- hydra-node and hydra-chain-observer - and the observer is connected to a single cardano-node that operates as a client on one network.

To support multiple versions of the protocol #1282, the team needs to compile the scripts of each version into individual chain observer binaries. These binaries can then communicate with the hydra-explorer components through an API. While Plutus validators change with any modification (thus affecting their hashes and addresses), this API can serve as an integration point that the team can support through multiple versions of the protocol.

Community​

Both projects saw community contributions this month:

TxPipe on Mithril​

The TxPipe team has kept working on the implementation of the Pallas Chain Observer for Mithril nodes and has contributed to retrieving the stake distribution and Mithril era markers (stored in the UTxOs of the chain) directly from the Cardano node with mini-protocols.

Ikigai auctions on Hydra​

As it also was briefly shown in the review meeting, Ikigai continued work on Hydra for Auctions in their Catalyst-backed project. Their use case is about starting auctions on the Cardano mainnet, but then collecting bids off-chain and only committing the final result on-chain. This is a perfect use case for Hydra, as it allows for a large number of off-chain transactions to be processed in parallel and then committed to the blockchain in a single transaction.

Conclusion​

We held the monthly review meeting for February 2024 on 2024-02-26 via Google Meet, presenting these slides and this recording.

This month, both projects focused on preparing for Conway and the upcoming hard fork combinator event. Preparations for the hard fork made the cardano-node a moving target, but the teams believe that early integration is key. Unexpected issues popped up in both projects, but they were able to address them.

New feature development is progressing well, with Mithril's transaction signing now available to early adopters. This feature opens up the possibility of making hydra-nodes more lightweight, as they won't need to rely on a single third party for trusted chain data. Instead, they can verify on the client (the Hydra node) using Mithril certificates. Additionally, Hydra's incremental decommit feature is in progress. While there's still significant work to be done on the on-chain parts, the teams have already implemented the off-chain workflow for incremental commits. Making both features available for users to try out will help validate the API changes. However, this cannot be released to mainnet without completing the required on-chain security work. During the review meeting, the teams sought feedback on a potential simplification of the Hydra on-chain protocol to have 'directly open' heads. Any thoughts or feedback on this fresh feature idea #1329 would be appreciated.

In general, if you read this far we would like to hear from you, dear reader, on any of our communication channels on Discord or a shout out on twitter. Maybe along with some feedback on the new, common website and the joint monthly report format? 🙏

We are looking forward to the next month and when it happens, the Conway hard-fork on the first public testnets (no date yet) when 🛡 Mithril serving certificates and 🐲 Hydra heads remaining open throughout.

This report summarizes the work on Hydra since November 2023. It serves as preparation for the monthly review meeting (see slides and recording), where the team updates project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

The Hydra team restructured several items on the project roadmap this month.

The latest roadmap with features and ideas

Notable updates​

• Released version 0.15.0, delivering offline mode and Conway support.

• @SundaeLabs contributed a new feature, offline mode #1254, which is related to this Catalyst project.

• Delivered Conway support #1177 preparing hydra-node for the upcoming Conway hard fork.

• A new feature, indicating preparation for running the latest version of Cardano nodes in P2P #1256. This is necessary as IOG's non-P2P relay nodes are to be shut down in January 2024.

• Similarly, a new feature - running smoke tests in SanchoNet #1257 - will serve as another proof point of being able to open/close heads in the new Conway era.

• Groomed and started work on building and deploying a Hydra heads explorer #696, see details below.

• Renamed 'Drop Babbage support #1178' to Switch layer 2 ledger to Conway #1178 to better capture its content.

Hydra development​

Issues and pull requests closed since the last report.

This month, the team worked on the following:

API changes: dropping JSON Tx​

The Hydra API now utilizes a TextEnvelope containing the upstream CBOR encoding from cardano-api for the Tx type and removes the JSON representation from the API #1240. This ensures that transaction types have a fixed canonical encoding that will not change between versions. As a result, consumers of the Hydra API will need to deserialize CBOR to inspect transactions.

This representation mirrors the TextEnvelope produced by the cardano-cli, with the addition of an txId field, which is guaranteed to match the transaction ID within the CBOR object.

Hydra Chess​

We team began developing a game on Hydra, leveraging dogfooding to uncover suboptimal elements or stumble upon bugs requiring resolution. Hydra Chess exemplified this process, offering valuable lessons during DApp development.

While refining the details, the team aims to improve the workflow of running hydra-node within a full peer-to-peer DApp framework. The objective is to ensure user-friendly accessibility while also providing an example for Hydra developers. You can access the source code here.

Hydra Explorer​

To measure the progress and success of Hydra, we require tools that provide insights into its usage. For this, the team initiated work on #696, enabling anyone to track Hydra heads throughout their entire life cycle and observe the growth of the Hydra ecosystem.

In this initial phase, the team has implemented a basic backend service #1235 that can track all heads on-chain within a devnet network. This service establishes a baseline and utilizes the hydra-chain-observer package, exposing a REST API for querying and retrieving information about all observed heads and their current states.

Community update​

TapTools recently published a summary of the project's progress over the last two months, which complements the updates provided above. Check it out here!

Besides the contributed offline mode mentioned above, Sundae Labs is also implementing a streaming API as designed in ADR29 in the course of this Catalyst project.

Also funded through Catalyst is a continuation of the Hydra auction project by Ikigai and MLabs. The Hydra team has not yet re-engaged in a regular exchange, but they expect to support the project with features as needed as they kick off this work.

As also pointed out in the review session, some of the team members have been involved in organizing the first Cardano Buidler Fest - happening on April 23-24 in Toulouse, France. This event will be a prime chance to connect with the people building (on) Cardano and inspire new ways of collaboration. At the time of publishing, tickets are already sold out, but we hope to see many Cardano builders there!

Conclusion​

We held the monthly review meeting for January 2024 on 2024-01-19 via Google Meet, presenting these slides and this recording.

This month, the teams demonstrated various Hydra and Mithril features. Additionally, Juan Magán and his team from Modulo-P showcased a compelling demo on utilizing zero-knowledge cryptography in a Hydra head for a mastermind game. This work is a Catalyst Fund 10 accepted proposal.

Besides providing the usual status and roadmap updates on both Hydra and Mithril, several interested community members, including Tudor, joined the discussion to share their use case ideas. Tudor presented his proposal for supply chain tracking of Moldavian wine, which is currently running as a Catalyst proposal in Fund11. The proposal aims to leverage Hydra for CIP-68 NFT updates in a head, requiring wallet/app support. We also discussed a potential CIP to standardize asset access if held on layer 2 ledgers.

Charles provided feedback on considering Cardano block extensions as a potential core protocol change, enabling various proof systems and new consensus algorithms to support partner chains. We further emphasized collaborative efforts with the community to explore and develop the user space, determining which aspects could evolve into marketplaces and what should be integrated into core protocols. Processes like CIPs play a vital role in standardizing and facilitating technology transfer from the 'user space' to the 'kernel space'.

These discussions will inform the team's roadmapping sessions for the year, focusing not only on building essential features for the Hydra Head protocol but also on experimenting with new approaches to enhance Cardano's efficiency and accessibility.

This report summarizes the work on Hydra since October 2023. It serves as preparation for the monthly review meeting (see slides and recording), where the team updates project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

This month, several items were restructured on the project roadmap:

The latest roadmap with features and ideas

Notable updates​

• Release of version 0.14.0 is imminent as most of the stateless chain observation #1096 feature is done and only some internal refactoring missing.

• The user-submitted feature idea on supporting extended keys, #1141, has been refined. The purpose of this item is now clear, and although it's not currently a priority, work on it can be started.

• Two new items were created to improve the developer experience of the Hydra build toolchain #1146. Additionally, a follow-up on things learned by hydra-poll aims to create a full end-to-end example of a two-player game in a Hydra head #1098.

• There was already an item about reacting to main chain parameter changes on the roadmap, which included not only protocol parameter updates, but also hard-fork events. With the upcoming hard fork into Conway, the team split off the changes needed to navigate the hard fork and clarified them:

  • Conway support #1177 will make the hydra-node support Conway and allow users to keep heads open across the hard fork.
  • Drop Babbage support #1178 will eventually drop support for the Babbage era and retain maintainability.
  • The original item is only about reacting to protocol parameter changes #195 now.

• Substantial updates were made to the design of the next major feature Incremental decommit #1057, and work on it can be started.

• In general, the team prepared several of these 💭 idea items and many of them will turn into 💬 features soon.

Hydra development​

Issues and pull requests closed since the last report

This month, the team worked on the following:

MeshJS + Hydra​

At the beginning of month, the team attended the Cardano Summit in Dubai where they held the Hydra MasterClass workshop. There they also presented the Hydra-Poll dApp which is using MeshJS to build Hydra transactions on the frontend.

The Hydra team collaborated closely with the MeshJS team and as the outcome of this work The MeshJS team added some Hydra specific features to the MeshJS SDK. There is a nice low level API for building custom transactions which is pretty similar to what cardano-cli does.

Now you can use isHydra field which you can pass to this API which then uses another set of protocol paramters suitable for Hydra (eg. usually in Hydra we want to have zero transaction fees, reduce script execution units etc.).

Big thanks to the MeshJS team on giving much needed support. They were presenting these changes in the monthly review meeting so make sure to watch the recording.

Rendering the Hydra specification​

Besides working on the design of incremental commits itself, the team also tidied up the way the Hydra protocol specification is written.

So far, the specification was written in LaTex and rendered to a PDF. LaTeX was a good choice because of its expressiveness on math expressions, but not so great a choice for pull requests and the entry barrier is quite high. The majority of documentation around Hydra and Mithril is written using Markdown and rendered into a webpage using a static site generator, currently Docusaurus. Most modern Markdown renderers do support LaTeX-style math nowadays, even the normal GitHub file preview.

Hence, the team experimented with using the more standard Markdown format to write the specification, but still being able to use LaTeX math and LaTeX macros (which are still heavily used in the spec). The aim is to have both, a PDF and HTML rendering.

While not yet finished, the GIF above shows the progress being made using pandoc to resolve macros, producing a PDF via xelatex and Docusaurus compatible markdown. Using powerful extensions of pandoc one can even convert TikZ, GraphViz and Mermaid into both target formats.

Community update​

Beside the great work done by the MeshJS team towards supporting Hydra in the browser, here are some news and updates on community-led work streams related to Hydra:

• SundaeLabs has made good progress on their offline mode PR which should hit the main branch soon,

• TxPipe has been collaborating with IOG's creative engineering team to build Cardaminal, a command-line wallet, and demonstrated it during the monthly review meeting. Cardaminal is built in Rust, leveraging Pallas libraries, and is aimed at being a fully-fledged scriptable wallet providing users with powerful stateful transaction edition capabilities, native assets management, chain synchronisation, etc.

  The plan is to make Cardaminal compatible with Hydra - to be able to use it to interact with a head - and Mithril - using snapshots to synchronise the wallet.

• The team behind ZK Snarks in Hydra Catalyst proposal is making good progress and should be able to give a demo at the next monthly review meeting

Conclusion​

The monthly review meeting for November 2023 was held on 2023-11-27 via Google Meet with these slides and the recording.

The Hydra team is thankful to have again had so many great demos from the numerous projects in and around Hydra and Mithril projects - we should really schedule it to be one hour planned duration. Also, if you have not watched it already, make sure to check out the recording, the written report is merely expanding on a couple of topics but cannot render the cool demonstrations.

At first look, the demonstrated projects may appear not related - we even had a minimal command-line wallet on the show after all. But as we went on, these impulses spark discussions and it becomes clear that all of these projects are connected.

For example, the terminal-based wallet 'Cardaminal' as it was presented the first time in public this month, could itself become a Mithril client for faster synchronizing or even light-mode using the recently published Rust client library for Mithril. Furthermore, such a minimal wallet would be a great testing ground for a Hydra-aware wallet. The MeshJS integration also has further potential of Hydra specific fetcher and submitter providers.

On the other hand, we also stumbled over not-so-optimistic sentiment even in much more mature scaling solutions like the Bitcoin Lightning Network. This tweet summarizes that creating non-custodial [scalability/LN] solutions 'feel like the most thankless thing in the world'.

It is an exciting ride still, but events like this monthly review meeting with its colorful demonstrations of small steps to improve the usability and scalability of Cardano piece by piece with often unknown synergies is also empowering. This month again made us feel not 'only a tribe', but more a working group across the Cardano community .. the Cardano Scaling group!?

This report summarizes the work on Hydra since September 2023. It serves as preparation for the monthly review meeting (see slides and recording), where the team updates project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

This month, the team released version 0.13.0, and updated the project roadmap to include the next planned version 0.14.0:

The latest roadmap with features and ideas

Release 0.13.0​

• Implemented security fixes to Hydra on-chain scripts, all of which are only exploitable by other head participants (not any attacker)

• Added support for (externally) committing inline datums

• Improved stability by querying the cardano-node at the tip

• Improved state persistence

• See full release notes and a list of delivered features.

Notable updates​

• Moved the network resilience #188 issue out of version 0.13.0 to allow for a quick release of the security fixes. This feature has been split in two parts, both of which are completed and will soon be included in the 0.14.0 release.

• A new user submitted a feature idea about supporting extended keys #1141. Initially urgent, it's no longer a priority and requires further clarification.

• Although not part of Hydra version 0.14.0, the team completed the integration of Hydra support in Kupo #1078, which can already be used with Kupo version 2.7.0

• Implemented a new Hydra Poll DApp feature, which is an example to be demonstrated at the Cardano Summit 2023.

• Realized stateless chain observation #1096 as a dedicated feature, which will add a hydra-chain-observer tool and simplify upcoming feature implementation related to layer 1 transactions.

• Created a feature idea item to migrate all validators to Aiken #1092. This was moved to 1.0.0 though, as it would require a complete re-audit of the scripts and and there is no urgent requirement for performance improvements at this stage.

Hydra development​

Issues and pull requests closed since the last report

This month, the team worked on the following:

Network resilience to node crash​

Last month, the team worked on making the Hydra network resilient to transient network connectivity failure (see #188). However, this was just one aspect of the larger effort to improve the overall stability and reliability of the network. By completing #1079, the team also made the network resilient to arbitrary node failures, which is a much more likely event to happen should the network run for a significant amount of time.

Hydra nodes now persist the network messages they send to peers and together with the already present acknowledgment mechanism, this means a node recovering from a crash can resend messages which were not received by their peers.

Hydra Poll DApp​

As the Hydra team seeks to adhere more to the principles of dogfooding, we wanted to upgrade our basic Hydraw application to something more interactive. We, therefore, built a poll DApp with the goal of demonstrating how one can build an application living entirely in the frontend that can interact directly with a hydra-node, using its websockets-based API. This application was also showcased at the Cardano Summit 2023, where the Hydra team delivered a masterclass.

Upgrade to GHC 9.6 & Brick 1.10​

This month, we were able to update our compiler toolchain from GHC 9.2.8 to 9.6.3. This also required us to upgrade the hydra-tui from Brick 0.73 to 1.10 to stay compatible with the new version of the mtl.

The update to Brick necessitated a complete rewrite of the hydra-tui which greatly simplifies the application logic, using Brick's new elm architecture style combined with microlens-mtl.

Community update​

This month, the monthly review meeting was almost completely covered by demonstrations from the various teams working on and with Hydra. Don't miss out on the demonstrations of the various community contributions and make sure to also view the recording.

Hypix​

Hypix is a continuation of the 'Hydraw' demonstration the team used throughout development. The project is spear-headed by Trym Bruset and will integrate CIP-68 NFTs with instant transaction processing in a Hydra head to realize collaboratively created art pieces. The project is progressing great with an open beta available soon.

The Hypix user interface

Kupo x Hydra​

Indexers of chain data, like Kupo, are useful to enable lightweight decentralized applications like Hypix. As the application state of Hypix, for example, is not only on the Cardano layer 1, but also in the Hydra head, Kupo was enhanced to run 'in front of' a hydra-node and provide the same API for UTXOs indexed from a Hydra head. This was a joint effort between @KtorZ, @v0d1ch and @ch1bo and has been released in version 2.7 of Kupo.

Kupo indexing a Hydra head

Offline-mode Hydra node​

Ledger-only mode for Hydra was one of the Catalyst Fund10 projects that got funded and work is underway to make it an integral part of Hydra software components. A new ADR has been proposed which details the rationale behind this new feature and an overview of what it brings and what impacts it will have on the hydra-node.

Moreover, this implementation is underway and should be part of the next release. This will enable running a single hydra-node as a kind of 'ledger-as-a-service', priming it with some UTXOs and allowing clients to submit and validate transactions without any relationship to layer 1.

Voting on Hydra​

The Hydra-for-voting project has been started earlier this year as a proof-of-concept to explore how a Hydra Head could help improve the voting or rather tallying process for Catalyst and other large-scale on-chain decision-making process. It's been on pause for a few months due to other priorities but it's been revived recently and put to use to tally votes for the Cardano Summit's awards recipients.

This development effort led to a couple of new features and fixes:

• The ability to submit a transaction for layer 1 using 'standard' CBOR-hex format
• Upgrade to the checked-in protocol parameters used by demo network which were out of sync with the internal ledger version
• Discussion on the usage of extended payment keys in in relation to the Hydra Head protocol

Hydra Tallying UI

ZK Snarks on Hydra​

The team has initiated discussions with another funded Catalyst Fund10 project, exploring the implementation of a zero-knowledge proof system for Cardano, utilizing Hydra and ZK Snarks. At this early stage, there is minimal progress to display, but the team is eager to start the development of this promising tool.

Conclusion​

The monthly review meeting for October 2023 was held on 2023-10-30 via Google Meet with these slides and the recording.

As mentioned above, the meeting was about 45 minutes of demonstrations - which clearly shows that we are in the wake of Catalyst Fund10 where several project proposals involving Hydra and Mithril got funded and at the same time everyone was preparing for the Cardano Summit 2023.

With this many community contributions and projects moving forward, there was not much time to discuss the Hydra roadmap or go into details about Mithril project status. This is great!

Besides, the project roadmaps seem to be well aligned where our next features in-line are most anticipated (incremental de-/commits) and the adoption of Mithril is showing a healthy level with clear next steps and features ahead.

It is getting increasingly hard to keep track of all the new developments around Hydra. However, that is also often not needed and the Cardano builders just set off and build things using Hydra these days.

November will start with the Cardano Summit, likely yielding even more feedback and feature ideas from users. Following this, there will be several productive weeks focused on feature development before the usual Christmas and New Year breaks across the teams.