Skip to main content

· 7 min read
Sebastian Nagel

This report summarizes the work on Hydra since January 2023. It serves as preparation for the monthly review meeting, where the teams update major project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

Looking at the roadmap, there are a few things that changed this month:

  • In general, there will be several items in flux as we align the technical roadmap to support our high-level objectives for the year.

  • Also note that we have not had a release in a while and we want to have smaller faster releasable items again.

  • Aligning implementation with HeadV1 specification #452 was more work than anticipated. There was quite some back and forth with the writing of the specification and discussions with researchers leading to scope creep.
    Most importantly, we made on-chain behavior consistent with specification.

  • Version 0.9.0 will be the first release to be developed against a full technical specification. We expect a release in early March.

  • Version 0.10.0 will be the first mainnet compatible release. For this, we created and groomed the main feature item Mainnet compatibility #713:

    • Not only making it technically compatible but also adding guard rails to limit risk (“not shoot yourself in the foot .. with a real gun now”) and documenting known issues.

  • For a fully maintained version 1.0.0 we still need to address some things, but we aim for an increased rate of planned releases in the next months as we march toward it.

The latest roadmap with multiple new items and starting to reflect Hydra objectives of 2023.

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

  • Aligning specification and implementation. This major feature item was the topic we worked on over the last couple of months. This time we could finish the major, must-have parts and can continue now on other features. Notable things from this month were:

    Pushing out contestation deadlines: The original paper defined that all participants need to be contesting within the contestation period (after close, before the deadline). In discussions with researchers last year, we identified better semantics: the contestation deadline gets "pushed out" on each contest. This has the benefit that the period needs to only accommodate the minimum time required to get a single contest transaction into the blockchain. Before it was depending on the size of the Hydra head - each participant can contest once.

    Align off-chain protocol logic: While most concerns addressed were about the on-chain scripts of the Hydra protocol, this is about the layer two protocol logic. That is, which messages are exchanged between Hydra nodes and how a node reacts to them. Here we realized that the implementation was slightly different from the formalism used in the paper and specification. This makes it non-obvious whether the behavior is exactly as specified or whether the specification can even be implemented into a working system.

    To clarify this, we put in some effort to align the implementation with the specification and the other way around. This allowed us to reason through our behavior tests whether the protocol as specified is sound as well. Below you see an excerpt of the Haskell code against the LaTeX pseudocode formalism:

  • Profiling Hydra transactions. We will want to maximize the capability of the Hydra Head protocol. This includes writing the on-chain scripts efficiently. Besides having transaction cost benchmarks, we worked on a profiling workflow for Hydra protocol transactions and published a tutorial on the Hydra website. This allows any contributor to Hydra to profile the scripts used in these transactions and identify code that uses a lot of the memory or CPU budget in flame graphs like this:

    [Interactive]Head not closable/limiting script size discussion + acceptance property

Community​

  • Audit guidelines. Hydra is an open-source project and, as such, should be freely used, reviewed and audited. The goal of audits and peer reviews is to increase confidence in the security of a system. However, conducting an audit is also very expensive and not in the scope of our current budget. All the while, the worth of an audit is not the same to all users, some will not even trust an audit they have not paid for themselves, while others do not care at all and would blindly trust the word of the maintainers.

    While the Hydra project continues to seek for funding to eventually pay for an audit of the protocol specification and the hydra-node implementation, it is a prerequisite that the necessary artifacts for such an audit exist. As we have been creating them in recent months, we also worked on guidelines for auditors, now published on our website. These guidelines should serve as a starting point and scope description of what would be most crucial to audit.

  • New tutorial. This month we received a great contribution from @perturbing, who created an end-to-end tutorial on using a hydra-node and opening a Hydra head on the testnet. We gladly adopted this tutorial and aim to keep it up to date with upcoming versions of the hydra-node. It will provide a great starting point for new users and makes the project more approachable. Thanks! 💚

  • Hydra for Payments demo. As we are about to kick off the next phase of the payments use case with Obsidian Systems, they demonstrated the use of hydra-pay in a new backoffice-like view, listing multiple payment channels and providing a great teaser of using payment channels for pay-per-use APIs or cashback use cases (micro-transactions).

  • Hydra for Voting. The project with Catalyst and Cardano Foundation on voting with Hydra (or rather tallying with Hydra) showed some progress. They have contributed Merkle Trees to Aiken, which will be useful to prove the correct tallying of votes in the L2. They started work on a hydra-java-client and provided valuable feedback on the hydra-node API over the last month.

  • Hydra on Demeter. The folks from @txpipe have been working on integrating Hydra into their platform-as-a-service offering, Demeter. This is a great way to get started with running a hydra-node as a service and we plan on integrating one instance into our long-running hydraw demo. It's still not fully released, but the teams are working together to make this integration great. Thanks for providing this service! 💚

  • New discord category. The Hydra project will be using Discord a bit more in the coming weeks and months. For that, a new category on the IOG Technical Discord server has been created. Look out for news and, most importantly, invites for the next month's review meeting in the new Hydra #announcements channel!

Conclusion​

The monthly review meeting for February was conducted on 2023-02-27 via Google Meet - slides / recording.

This month's installment featured two demos, one about the Hydra Payment channels, and another about the progress on the Hydra for Auctions project. It was great to showcase not only core development progress on the specification and protocol implementation in the hydra-node, but also to see contributions from the Community on tutorials and tooling (Demeter integration).

An exciting, but also challenging next step will be to aim for mainnet compatibility and at the same time deliver quick releases to enable these lighthouse projects and the increasing number of use cases explored by the builders of Cardano.

· 6 min read
Sebastian Nagel

This report summarizes the work on Hydra since December 2022. It serves as preparation for the monthly review meeting, where the teams update major project stakeholders on recent developments to gather their feedback on the proposed plan forward.

Roadmap​

Looking at the roadmap, there are just a few things to report this month:

The latest roadmap with minor changes, which needs a reflection of the latest objectives.

Development​

Issues closed since last report

This month, the team worked on the following:

  • First write-up of Hydra spec online. The team has been aggregating the latest definitions of on- and off-chain semantics into a new technical specification. The latest version can still be found on Overleaf, before its integration into the core Hydra repository. Should you have feedback, please send it our way through one of the communication channels.

  • Closed more gaps in the Head Contract. The team addressed two more gaps in the on-chain scripts based on discussions on the specification.

    • All validators are authenticated now and ensure contract continuity by checking for the state tokens in the output (or getting burned).

    • Abort now fully reimburses all committed UTXOs, even if two parties would have committed the exact same output (quite a theoretical attack vector).

    This work is captured in #452, but this work item is taking quite a bit longer than expected. Also, it's a single feature item, lacking some rationale on certain requirements, so we intend to break this down into smaller pieces over the next days and weeks.

  • Mutation test framework improvements. While closing gaps in the contracts, the team realized that some of the mutation tests were correctly tripping validators and making transactions invalid, but not always for the right reason. In fact, some of the mutations were ‘too harsh’ and made the even transaction not even pass phase-1 validation. The team addressed this by introducing a first (naiive) way to assert the right cause of the failure by checking strings in the validator log. #679

  • Add headId in API and TUI. The team not only added the unique headId to the appropriate server outputs and the TUI #678, but also added sequence numbers and timestamps to produced outputs #618. This feature was asked for by users and simplifies identifying of heads and integration with the hydra-node.

  • Hydra explorer experiment. Some of the team used the holiday season to conduct some experiments on summarizing Hydra Heads observed on a network. This quick hack already demonstrates the value of a Hydra explorer as a tool to measure adoption by open heads on a given network. Naturally, it would reuse code from the hydra-node and challenge the architecture used in it for tracking multiple heads. Also, it will drive discussion about the versioning of hydra-plutus and the possibility to track multiple versions of the Hydra protocol on the chain (hashes/addresses change in each version).

Hydra explorer first experiment UI

Community​

  • The Hydra for Voting project, which got kicked off in December is picking up steam with deep dives into Catalyst voting and Hydra technology. Discussions are currently held in various settings and the team is continuing to develop a picture of what is achievable.

  • Hydra for auctions lite paper. IOG is collaborating with MLabs on exploring how Hydra can improve auction use cases. At this stage, the teams discussed multiple approaches on structuring the problem space with various Hydra topologies and specific trade-offs. The full case study was published on Essential Cardano. The teams plan to upstream the findings back into the use case section on the Hydra website from which this project originated.

Themes for 2023​

After kicking off planning in a workshop and reflecting on what Hydra achieved in 2023, the team has progressed in fleshing out relevant themes and objectives for this year to position Hydra as a sustainable open-source project in the age of Voltaire:

  • A mainnet mature application

    • Why? The core contributors should be able to use the protocol on mainnet, so it should be maintainable and mature enough to lock some ada in a Hydra Head using a reference DApp (eg, hydraw). By creating and growing this DApp, the team will dogfood features and hence improve additional usability.
    • How to measure? Number of heads on mainnet > 0 and core contributors feel confident to lock at least 100₳ in a Hydra Head on mainnet.
    • Next steps:
      • Publish Hydra Head V1 specification
      • Close gaps in implementation and release 0.9.0
      • Create a request for proposals (RFP) to audit the specification and its implementation

  • Increase adoption

    • Why? The team wants to showcase what is possible with Hydra through benchmarks and lighthouse projects, enabling use cases and reducing friction to use Hydra. Also, ensuring interoperability through open standards and reference implementations.
    • How to measure? Number of third-party created heads on any network > 0
    • Next steps:
      • Build a basic Hydra Head explorer to measure adoption
      • Benchmark off-chain performance (of a selected scenario)
      • Demonstrate Hydra for payments to potential users
      • Support Hydra for voting project
      • Hydra for auctions project exploring a delegated voucher auction

  • Sustainable open-source-development

    • Why? The team wants to scale Cardano in the long run. Hence this project should not be owned by a single entity, but by the community and the Cardano network itself. It is thus essential to make contributions to the project possible and ultimately it should be easy to become a contributor.
    • How to measure? Number of contributors on GitHub
    • Next steps:
      • Open up monthly reviews
      • Add tutorials to the website
      • Publish and maintain a use-case-centric roadmap and feature map

Conclusion​

This monthly review was the first public one with not only stakeholders from IOG and the CF, but also collaborators of the latest projects. This was the first step to a more open and transparent development process.

The teams showed some classic demos – like the ‘hydraw’ application which has been running on the same Hydra Head for four weeks now – and some new developments on the Hydra explorer and minor improvements to hydra-node API and TUI.

It was very interesting to bring various teams together to exchange ideas and thoughts on the individual roadmaps for the new year. The teams also shared the relevant themes and objectives of the Hydra project for 2023 (as also listed above). They will now make sure to reflect these steps into this concrete roadmap such that it will contain marketable features bringing us closer to these goals.

· 4 min read
Sebastian Nagel

This report summarizes the work on Hydra since November 2022 and also serves as a preparation & write-up of the monthly review meeting, in which we update major stakeholders of the project on recent developments and gather their feedback on our proposed plan forward each month.

Roadmap​

Looking at our roadmap we can report the following updates:

Notable changes​

Latest roadmap with drafted 0.10.0 version.

Development​

Issues closed since last report

This month we have been working on:

  • Fixed AcquirePointTooOld errors by changing the way the internal wallet initializes its state #621. See the PR description for details. This did not strictly happen this month, but we have not mentioned it last time.

  • Closed first gap in Head Contract: We collected gaps between specification and current implementation in #452 and started now closing them. This first issue, was to prevent DoS / stalling of heads, by bounding the lower and upper transaction validity time of the close transaction. As always, time handling is hard and this required detailed discussion in the form of an ADR. See ADR21 for more details.

  • Reduced cost of commits by at least 30% using reference scripts also in the commit transaction. We only used reference scripts in abort so far. This reduces the min ADA cost of commit because the transactions are roughly 5.2KB smaller.

    Before:

    After:

  • Use nix flakes & build docker images using nix: Triggered by a migration to Cicero for a nix-based CI, we started using nix flakes for nix build and nix develop instead of the legacy nix-shell in #646. While we ended up sticking with Github Actions as the only CI and a Cachix cache, we extended this work to build our docker images more efficiently using nix in #654.

    Image sizes went down between 20-50%:

    While docker CI builds on master went down to 1-10min (from 10-30m+)!

Hydra V1 Specification​

Finally, we bit the bullet and started a write-up of the specification. Over the last couple of months we had worked with marked up versions (see below) of the original paper and a GDoc to discuss and review the protocol as it was implemented.

As we also updated the security properties and proofs, we started aggregating a LaTeX document on overleaf. While it holds a similar background and preliminaries as the original paper, it only specifies a single version of the protocol with no variants, less modularization, as well as explicit assumptions and important notes for implementors.

We aim to be publishing the spec as part of the main hydra repository soon.

Community​

  • Hydra for Payments project completed the first phase of this work stream by tagging version 0.1.0. This open source project conducted by Obsidian Systems and sponsored by IOG created a great starting point for developers to build services/APIs of payment channels using Hydra Head. This month we reviewed the improved documentation and are looking forward to see this work applied to eventually build fast & cheap payments for light-wallet users and power payment channel use cases from business-to-business (B2B) in the short-term.

  • Hydra for Voting project kicked off this joint project between IOG Research, Catalyst & Hydra Tribes, and the Cardano Foundation. The goal is to assess suitability of Hydra Head to support voting use cases by implementing parts of Catalyst voting system on Hydra Head and benchmark execution at scale. The kick-off meeting took place on December 15.

  • Blog post by CF: To round off the year, Matthias wrote a nice retrospective blog post about Hydra at the CF. Besides summarizing our progress made in 2022 it also gives an outlook where Hydra is headed for in 2023.

Conclusion​

Although Hydra is in essence an open source project right now, we would love to make it "even more open source" in the coming months. Our vision of Hydra requires to have the builders of the Cardano ecosystem to be part of this journey. For that matter, we want not only to open up these monthly reports, but the review meeting backing it as well!

This, besides our engagements with the community above, will allow the project to grow beyond IOG & CF contributions. To become the true Cardano open source project we believe will be worthy to serve as the scalability component required by the Cardano network.

· 9 min read
Sebastian Nagel

Introduction​

This report summarizes the activities of the Hydra team since October 2022 and also serves as a preparation & write-up of the monthly review meeting, in which we update major stakeholders of the project on recent developments and gather their feedback on our proposed plan forward each month.

Roadmap update​

Looking at our roadmap we can report the following releases and updates:

Release 0.8.0​

  • Long overdue feature release adding persistence to the hydra-node
  • Backup & restore the state of a Hydra Head #187
  • Improve user experience following hydraw experiment #518 from being only a discussion to a (to-be-groomed & planned) feature
  • Full release notes

Release 0.8.1​

  • Follow-up release addressing user feedback on persistence from 0.8.0
  • Allow clients to see latest state after restart #580
  • Bug fixes of following chain state #599
  • Full release notes

Notable changes​

  • Impact mapping from workshop (see below) and reviewing the starmap above had us discuss whether the scope for 1.0.0 is correct or not
  • Need to be conscious of scope creep vs. getting something audited
  • Focus on getting the spec done, 0.9.0 tagged and an audit under way now → minor re-ordering in priorities
  • Iterative releases afterwards still - even while getting audited, non-contract changes first
  • Promoted https://github.com/input-output-hk/hydra/issues/635 from being only a discussion to a (to-be-groomed & planned) feature

Latest roadmap with 0.8.0 and 0.8.1 already released and slight re-ordering on 0.9.0

Development​

Issues closed since last report: Issues - input-output-hk/hydra

Besides the things in the releases above (see roadmap update), we have been working on:

  • Flaky TUI CI tests: We have sunk quite some time again in investigating cryptic CI failures in our TUI end-to-end tests. They have been crashing abruptly without any information and thus hard-to-debug. This investigation was unsuccessful and we realized at some point that the TUI tests are not that important anyways - we have API-level end-to-end tests. So we disabled these tests for now.

  • Published hydra-cardano-api to CHaP: With the Cardano Haskell Packages (CHaP) becoming available now and wanting to be a good citizen, we pushed for getting our flavor of the cardano-api also published there. This is non-trivial though, as we are at an integration point even "further up" than the cardano-node #504

  • Implemented ADR18: While the first stints on persistence already make it possible to restart hydra-node without needing to close a Head, we have implemented this in a nicer way now only keeping a single state (ground truth) for both L2 and L1 information #541 of the Head protocol and the specification, we kept discovering bigger and bigger issues and hence realized the need for a discussion on transaction validity in context of closing / contesting Heads. #615

Formal verification & specification​

An important part of the project right now is the formalization and audit preparation:

  • Internal audit has started with involvement of two persons from IOG IT team, each one addressing different part of Hydra:

    • One auditor is working on the project as a whole, targeting potential vulnerabilities with the off-chain code, the infrastructure, dependencies, etc. This has not lead to any significant issue nor action plan so far.
    • Another auditor is specifically targeting the formal specification and the on-chain code to identify vulnerabilities in the protocol itself.

  • Discussions with researchers have led to the development of a joint Coordinated Hydra Head V1 specification defining formally the protocol as it is actually implemented.

    • Work on this document has already allowed us to identify gaps
    • It is the basis on which BCryptic's analyst is working to audit Hydra protocol
    • It allows us to make explicit a lot of implicit assumptions that are in the code but not in the original paper and "Shape" the language used to describe the protocol

  • We have drafted an RFP for submission to external auditors https://github.com/input-output-hk/hydra/blob/audit/rfp/security/RFP.md defining the scope and targets of the audit

  • We are still unsure about whether our approach is the right one as having a "formal specification" in a manually checked document spanning a dozen pages seems quite brittle.

    • Some progress has been made on "formalizing" properties to be automatically "QuickChecked" using an executable model of the expected behavior of the system but this approach seems more suited for "team-internal consumption", e.g. building confidence within the core committers and contributors

    • Ideally, we would want a proper formalization of the protocol, using an existing theorem proving/model checking framework, through which properties could be asserted. Quite a lot of work has already been done in blockchain space, including some work on Lightning and TLA+ or some other state-machine/temporal logic based language appear promising:

    • This effort should be done in accordance to the DApps certification process

Product​

Most updates on the product side of things have been addressed in the starmap update above.

  • Hydra for Payments lighthouse project: Latest demonstration shown by Obsidian Systems on how to use payment channels in a "light way" from a web frontend. The API evolved and we were able to use the preview testnet now. The project is progressing very nicely and we are optimistic to close it successfully soon with improved documentation and usability.

  • NFT auctions lighthouse project: After doing surveys and interviews, MLabs is currently concluding the discovery phase with a light / white paper on how NFT auctions could be implemented using Hydra (today or in the future with more features).

  • Drafting Voting on Hydra project: Work has started on building a Proof-of-concept for voting on Hydra targeting Catalyst’s use case but with an eye towards building a generic solution suitable for large scale voting systems based on Hydra Head

    This development should be undertaken jointly with the Cardano Foundation and SundaeSwap who are also interested in building such a system and recently demonstrated their capacity of doing voting via the Cardano Summit voting system.

  • TxPipe demonstrated integration with demeter.run: Without our help nor even knowing of this effort, TxPipe has recently shown an early prototype of hydra-nodes instrumented via their demeter.run platform. This has come a bit as a surprise and is exactly why we love open source ❤️ → Video

Team & open source​

Some notable developments this month have been:

  • Renamed the repository: from hydra-poc to hydra! This was revealed in the summit presentation (see below) and should indicate that the project is not only a Proof of Concept (POC) anymore, but has become more - as also demonstrated by the various early adopters and demos lately. 🐲

  • Inner source (IOG) contribution: The education team has been working on a Hydra Tutorial and we have been involved in reviewing and trying it out. This is a great effort and will help people get started with Hydra. Thanks folks 💚

Cardano Summit & workshop​

This month there was also the Cardano Summit, this time organized by the Cardano Foundation and the content was even voted on by the community! We have been both, invited and nominated as panelist and speakers and this makes us extremely proud! In general, the reception of Hydra seems to be very positive in the community from what we could experience first hand on the summit.

Agenda items we participated in:

After attending the summit, we also used the fact that the whole team is in one location for a team workshop. We booked a coworking space and spent 3 days together. Not much coding & hacking this time, but we are happy to have produced these results:

  • Timeline / year in review: As we had new team members with us, the request was to give a recap of how the Hydra project evolved over the last two years. The whole timeline can bee seen below, and this also sparked the idea of creating a "Year in review" blog post, which is currently in preparation.

  • Retrospective: In-person workshops are the perfect place for doing retrospectives to reflect on what was good, bad and collecting ideas & actions in how to improve our work environment and processes.

    Retrospective board from 2022-11-22

  • Impact map: Conscious about the fact that the project got defined about one year ago in a first project plan, and inspired by timeline and (short-term) ideas, we also set off to reflect on the Why, How and What of this project.

    After reviewing the project vision, we reached for the tool of our choice to ideate on tangible objectives / deliverables - impact maps! Also, we felt the need to use a different goal this time. One that is reflecting more closely (or, in fact, broadly) what our current mission is. So instead of the % of Cardano transactions are done on L2, we set a new goal: being the number one DApp on Cardano (by all known metrics: TVL, traffic, volume etc..) Fundamentally, both goals illustrate the same idea, but the latter better supports the narrative that Hydra is also just a DApp (not a network upgrade) and needs usage & adoption to reach our vision.

    Below you see the result of our session. We checked back to the old impact map after creating this one, and many things are still relevant / similar on the new sample (it's never complete!) - some of the deliverables we even achieved.

    Impact map with new goal

Conclusion

Reflecting on our original plans (about a year ago) and our current roadmap we can say we are behind our anticipated schedule. However, recent developments on the lighthouse projects and community members contributing & using Hydra are convincing us that we are on the right track. The summit paints a similar picture and we feel reinvigorated by the great people we met, discussions we had and relationships we created & solidified on the summit and during our Hydra workshop.