This report summarizes the work on Hydra since June 2023. It serves as preparation for the monthly review meeting (see slides and recording), where the team updates project stakeholders on recent developments to gather their feedback on proposed plans.

Starmap update​

Every couple of months, the team reflects on the themes identified for 2023 (as presented in January), and creates a high level list of possible deliverables, eg using impact maps. This time, the team has come up with the following items:

• Sustainable open-source 💚

  • Hydra master-class workshop: A concrete session we aim to conduct in an upcoming event or online as a fallback. This will also be a motivation to update our existing tutorials and improve the first-start experience.

  • Engage and support developer community on Catalyst proposals: There are around 30 proposals mentioning Hydra in the current Catalyst funding round and we want to review, comment, try to align, and maybe even promote some proposals. The goal is to provide a platform for many new ideas while avoiding fragmentation and maximizing the reuse of work.

  • Further open up GitHub repositories: We want to provide a welcoming experience to non-IOG contributors and provide an easy-to-navigate group of repositories relating to Hydra.

• Mainnet mature application 💜

  • Make Hydra trivial to use: To enhance user and developer experience, we aim to enable a quick and frictionless setup of Hydra, possibly involving Mithril to get started. Our goal is to facilitate the opening of a distributed Hydra head with two nodes on a testnet within just a few minutes.

  • Build a basic head explorer: We have previously experimented with the explorer, and now we are looking to set it up for the long term. The goal is not only to assess Hydra adoption but also to provide a valuable tool for DApp developers. Additionally, we aim to create a visualization that end users can use to answer important questions, such as 'who is running this head?'

• Drive adoption 💙

  • Deliver the first version of HydraNow to mainnet: It's important that the very promising use case of payments using Hydra is made available to first users on mainnet. As a concrete example, this would be a mobile application 'HydraNow', which uses a hydra-pay service to enable peer-to-peer payment channels.

  • Enable uni-directional payment channels: This will expand the capabilities of any payment channel created using Hydra, allowing deposits and withdrawals while keeping the head open.

Work packages related to these items appear on the technical roadmap (especially in this view) and in our community activities.

Roadmap​

This month, the team released version 0.11.0, which shifts the way Hydra clients need to interact with the hydra-node when committing funds. The project roadmap was also updated to reflect the latest developments and ideas from the starmap:

The roadmap without idea items

Release 0.11.0​

• Completes the layer 2 ledger isomorphism with timed transaction support

• Adds commit from external wallets API, which allows to:

  • Commit multiple UTXO
  • Commit from scripts
  • Commit directly from any key

• Deprecated: Internal commits via the WebSocket API

• See full release notes and a list of delivered features

Notable updates​

• Two new items _elevated from user ideas and improved the API capabilities:

  • The ability to read protocol parameters via API #735
  • The ability tosubmit layer 1 transactions through hydra-node #966

• [Removing commit from internal wallet #954](Submit layer 1 transactions through hydra-node#954) is a consequence of deprecating this functionality in the previous 0.11.0 release. This will make the code base and documentation cleaner as there will be only one way to commit funds to a head.

• The first new item derived from the starmap is to Update tutorial and include Mithril #997. The tutorial is already a bit outdated, this will slim it down and incorporate Mithril to get the fastest (secure) bootstrap possible of a hydra-node.

• As a result of focusing on the very concrete use case of payments for mainnet deployment (and hence 1.0.0 release), multiple items need to (and can) get reprioritized to match the requirements of this use case.

The latest roadmap with features and ideas

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

Authenticate network messages #727​

The Hydra Head protocol paper states:

Each party then establishes pairwise authenticated channels to all other parties.

Hydra: Fast Isomorphic State Channels, p.14

Guaranteeing the authenticity and integrity of the messages received from one's peers is critical to the security of the Head protocol. Initially, we considered deferring this responsibility to the transport layer, for example, leaving it to Hydra node operators to use TLS or other encrypted channels to communicate with their peers. However, this introduced an additional and unnecessary operational burden on operators.

It appeared simple enough to reuse the existing Hydra keys (using Ed25519 curve), which need to be passed to the node, to:

• sign all messages sent to peers
• verify all messages received from peers

The net benefit of this feature is to increase the life expectancy of a head, preventing some forms of denial of service attacks or wrongly configured nodes that would lead to a stalled head.

ReqSn only sends transaction IDs #728​

This feature aligns the current implementation better with the original paper, as it requests snapshots using only the transaction identifiers, rather than the full transactions.

The motivation to implement this feature came from different directions:

• Improve the performance of the protocol by reducing bandwidth usage
• Make it possible to have 'atomic' transaction sets submitted (#900),
• Prevent 'front-running' from the current leader which, in the previous implementation, could inject arbitrary (but valid) transactions that peers had never seen before into a snapshot.

ReqSn's snapshots now contain only the list of transaction IDs and each hydra-node validates the snapshot by resolving those ideas against the set of transactions they have seen (through ReqTx). A consequence is that this introduces the possibility of space leak in the event a submitted valid transaction never gets included in a snapshot, but this should not be a problem in practice.

Note that the specification has been updated to reflect this change.

GitHub security advisories​

There was a bug in the multi-signature verification which could compromise off-chain security of a Hydra Head protocol. Although the fix was quite simple, we decided to use a new GitHub feature to create the first GitHub security advisory for Hydra.

The process creates a separate GitHub repository where potential security issues could be discussed privately (between the reporter and the maintainers) since potential attackers should not be aware of any issues until they are fixed. The team scored this issue using CVSS (The Common Vulnerability Scoring System), the score was moderate (5.5), and they didn't request a CVE (Common Vulnerabilities and Exposures) for this (yet).

The process overall works well, but Hydra CI is not checking the builds in these private, one-off forks so there is a risk of merging code that doesn't compile.

Moving to GHC 9.2.7 #841​

GHC 9.2 series have been around for more than a year and the whole Cardano ecosystem is gradually moving to use this new compiler version. Moreover, GHC 8.10.7 does not work reliably on aarch64 architectures, eg Mac M1/M2.

Moving to this new version entailed some significant changes in the code and also in the Plutus scripts:

Name	After (bytes)	Before (bytes)	Δsize
νInitial	4289	4621	- 7%
νCommit	2124	2422	- 12%
νHead	9185	8954	+ 3%
μHead	4149	4458	-7%

Community​

Hydra master-class​

As already mentioned above in our starmap update, we aim to have an educational session at the upcoming Rare Evo event:

Whether you are building decentralized applications which need to be fast and scale well, or you are creating infrastructure for other builders on Cardano, we will show you how to use Hydra to make your applications and platforms utilize fast off-chain transaction processing via the Hydra Head protocol.

In this session you will learn what Hydra is all about, how to build scalable applications in a live coding session, get to know the hydra-pay and hydra-auction projects, and be guided through exercises to get your hands on these APIs and early SDKs.

We are currently putting together a detailed outline and will send out communications on how to register for this exclusive session within the next 1-2 weeks. Stay tuned.

Conclusion​

The monthly review meeting for May was held on 2023-07-26 via Google Meet with these slides and here is the recording.

Although we had planned an awesome demo for this month's review meeting, showcasing a Hydra head hack without the authenticated messages feature, we encountered some live demo problems (lesson learned: always record a backup!). While we might present it next time, this situation emphasizes the need for continuous improvement in developer and user experiences.

We haven't received many questions on our journey to further mature the Hydra software stack to support various use cases building on it. However, the recently kicked-off Catalyst Fund10 sparked a nice discussion on how to handle all these great new ideas while being mindful not to fragment the scaling efforts too much.

Open protocols mean that new ideas can be incorporated not only into the various code bases, but also in the specification and standards which ensure interoperability. We discussed means to ensure adherence to standards in a light-weight agile way and the challenges of educating potential contributors in specification-driven development.

Next month we'll be at RareEvo, but might have the monthly just before in a hybrid session from Colorado. Until then, we'll make Hydra trivial to use and get started on our new objectives.

This report summarizes the work on Hydra since May 2023. It serves as preparation for the monthly review meeting (see slides and recording), where the team updates project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

While there was no release this month, the team implemented several notable features, which will be released soon as version 0.11.0:

The roadmap without idea items

Notable roadmap updates​

• Realized allowing commit txs with multiple UTxO #774 as a dedicated roadmap item separate from the related commit from external wallet #215 (which grew too big).

• Revisited off-chain performance by doing benchmarks #186. Identified a bottleneck, groomed and planned a follow-up performance improvement feature for Event sourced persistence" #913

• Plan to release 0.11.0 without Authenticate network messages #727 to deliver enabling featues earlier.

• API improvements and exploring batched transactions on L2 showed that ReqSN only sends transaction IDs #728 is in demand, front-loads further API changes and is estimated to be a low-hanging fruit.

• Reprioritized items in <= 1.0.0 column, to do items with on-chain protocol impact earlier and not much feedback has been received on snapshotting items. In fact, if Support larger # of UTXO via split-fanout #190 is done slightly differently, that should avoid some of the problems of impossible to finalize snapshots.

The latest roadmap with featuresand ideas

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

Commits with multiple UTXO #774​

One of the early adopting projects is exploring how to move scripts from layer 1 to layer 2. For that purpose, it was necessary to not only commit the actual script UTXO, but also a "regular" UTXO holding only ada to be used as collateral (the cardano-ledger always requires collateral although it would not be necessary on a layer 2).

To enable this, the specification and on-chain protocol needed updating. Before a protocol participant could commit zero or one UTXO, which changed now to a list of UTXO. As the specification is now part of the repository, it could be kept consistent within the same pull request.

Despite being a breaking change, leading to new Hydra script hashes to be published and used starting with version 0.11.0, this change was suprisingly easy to do and demonstrated the amenability of the Head protocol and the system architecture.

Commits from external wallet #215​

The team started to mark fuel some time ago as it was an easy workaround to distinguish UTXOs that can be committed into a head apart from regular outputs holding ada to pay for fees - the so-called fuel. However, this required users to "send funds" they want to commit first to the hydra-nodes internal wallet and involved additional steps in tagging such outputs with a special datum hash

To commit from external wallets, a new API endpoint was introduced for the purpose of drafting a commit transaction. The clients would request such draft transaction by sending a POST request to /commit and the hydra-node would respond with a transaction already authorized by the internal wallet. If the commit involved user funds (empty commits are still possible), then the client application would need to sign the transaction using the corresponding signing key. Also, submitting this transaction has shifted from hydra-node to the client.

This removes direct custody of hydra-node over user funds since clients can now use whatever key they own, not known to the hydra-node, to do a commit step and no single hydra-node has access to user funds used in the Head protocol.

Within this work package, marking fuel became deprecated and all UTXOs owned by the internal wallet are considered fuel. Fuel marking wil be completely removed in the future. Furthermore, a good old HTTP-based API is now used for the new query (so far it was only WebSocket-based), which prompts a potential shift to using OpenAPI as API specification since AsyncAPI does not describe synchronous requests well.

Benchmark performance of Hydra Head #186​

Low confirmation time is a key feature of Hydra Head protocol and is needed for some use cases. The existing end-to-end benchmarks have been revived and tailored for a baseline scenario, one with a single hydra-node and a single client. While this scenario is not representative of target deployments, it's interesting as it gives us an upper bound on the performance of a Hydra Head network.

The results have been somewhat disappointing, and further investigations lead to a few insights:

• There was a misconfiguration in the RTS of the hydra-node executable which was therefore not taking advantage of parallelism
• We identified the main bottleneck to be persistence of the node's state, which is currently done by overwriting a single file with the JSON content of the full state on state change which is pretty inefficient. As a consequence, the team will work on improving the persistence strategy as described in this ADR
• More benchmarks:
  • The aforementioned End-to-end benchmarks,
  • Ledger Micro-benchmarks as a comparison basis.

Operating hydra nodes​

As a "dogfooding" exercise, the Hydra team is operating a Hydra Head on mainnet on top of which is running our hydraw application. The Head protocol got stuck a couple of times and post-mortem investigations lead to a few improvements in how to operate a hydra-node and network:

• Better reporting on the version of executable hydra-node and hydra-tui which now report the Git SHA1 at which they were built
• Reduction in the volume of logs emitted by hydra-node by removing some chatty network-related logs and using ids to link Begin/End pairs
• Rework the log-filter tool to provide timings for various events and effects keyed by transactions

Community​

Hydra hackathon / workshop​

The Hydra team is considering holding a workshop at/around Rare Evo at the end of August and is in contact with the event teams at IOG and the Rare Evo organizers. The concrete format, scope and agenda is still a bit unclear as we are contemplating whether to do a workshop/tutorial style or rather a introduction + challenge setting event. If it does not work out for Rare Evo, we will find another event or do it ourselves.

If you are reading this and would be interested in joining such an event please drop us a line on discord or DM @ch1bo! Ideally along with some thoughts on preferred format or what you would be interested in.

Hydra for Auctions contributions and closing of project​

One of the Hydra lighthouse projects is slowly coming to an end. The collaboration project between IOG and MLabs on using Hydra for auctions is currently finalizing documentation and creating demonstration running the whole thing on a public testnet. Although the demo video was not available at the copy deadline of this report, watch this space for more about this next month.

The project yielded multiple github issues containing ideas and sparked great discussions on improving Hydra like reported last month. It is also the first project which demonstrates how to move smart contracts from the layer 1 (L1) to the layer 2 (L2)! Overall it is a great example of establishing crucial state on L1 and achieving scalability through Hydra as L2.

The code is fully open source and available on Github hydra-auction.

Conclusion​

The monthly review meeting for May was held on 2023-06-21 via Google Meet with these slides and here is the recording.

It was a fairly straight-forward month and consequently unexciting review meeting this month. Unfortunately, we could not release 0.11.0 yet, but we wanted to get the quite significant change of supporting commits from external wallets done first. This feature in particular was more involved than expected, but as the demonstration in the meeting showed, we are in the final stages of getting this over the line.

Besides some nice findings to potential performance improvements for layer 2 transaction processing, there was not much to announce this time around. Behind the scenes, however, there have been great progress on the Hydra for Payments project and the next updates are going to be more interesting again.

Despite holiday season approaching, we will march on, steadily adding features and enabling more and more use cases to scale through Hydra.

This report summarizes the work on Hydra since April 2023. It serves as preparation for the monthly review meeting (see slides and recording), where the team updates project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

This month the team released version 0.10.0 which includes many important features. The project roadmap was only slightly updated this month and already saw one more feature completed:

The roadmap without idea items

Release 0.10.0​

• First mainnet compatible release of hydra-node including technical changes and documentation

• Mainnet exposure currently limited to 100₳ per participant

• Added some missing features to the client API (short-term solutions)

• Some quality of life improvements in running the hydra-node

• Full release notes and a list of delivered features

Notable roadmap updates​

• Got new input and a concrete scenario for benchmarking (off-chain) performance of a Hydra head. As it is a very basic one, we moved the item from an idea to a feature and started work on it.

• Added a new feature item to add Hydra as a tool to developer platforms within our agenda of promoting Hydra as an open-source platform for scalability on Cardano.

• Already completed a first feature for 0.11.0 and considering whether to release it as early as possible with latest advances.

The latest roadmap with features and ideas

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

Timed transactions #196​

To realize the Hydra Head protocol, a hydra-node observes and follows the chain progress on the layer 1 (L1). Until now, this component was only observing protocol related transactions. Now, on every block roll forward, a Tick event carrying the notion of time from L1 is observed by the hydra-node and is used to validate transactions in a Hydra head. This means that clients can submit time bounded transactions to an open head and expect them to be validated using the same slot that would be used on L1. It is important to mention that time only advances on L1 when a block is produced.

Timed transactions in a Hydra Head state channel

This feature will make the Hydra L2 ledger now en-par with UTXO features available on the Cardano L1. A logical next step in this direction could be to make time in the state channel advance every configured slotLength (e.g. every second) using the system clock in between blocks as a form of dead reckoning.

Removing --ledger-genesis #863​

The hydra-node had two command line options to configure the ledger used for validating layer 2 (L2) transactions: --ledger-protocol-parameters and --ledger-genesis. The former option, which is self-explanatory, allows configuration of the Cardano protocol parameters to be used. Often, the same parameters as those on L1 are configured, or similar parameters with reduced fees, for example. On the other hand, the second option required the genesis-shelley.json file previously used to initialize the Shelley era by the cardano-node.

When we started using the current slot in the L2 ledger (see above), we realized that only the start time and slot length are effectively used from that configuration file. Moreover, it would be quite surprising if those were different and slots would be longer or shorter on L2 (unless explicitly configured). We opted to remove the option altogether and have the hydra-node fetch the genesis parameters from the Cardano network. This makes the system easier to configure and more isomorphic to L1.

Improving CI runtime​

The Hydra project embraces Test Driven Development and implements Continuous Integration (CI), checking all these tests pass and more.

The CI could sometimes take as long as an hour or more to run, which has a negative impact on the project's workflow. For instance, all the project's branches have to be fast-forward with master before being merged. So if all the tests on a branch are green and the pull request has been approved but is lagging a bit behind master, it has to be rebased, so the CI has to run again, incurring a one hour or so delay in this case before being able to merge it. The situation becomes worse when several pull requests are ready to be merged.

Analyzing the run from May, 10th we can see that:

• Building and testing takes 19 minutes for the longest
• Generating haddock documentation and running the benchmarks takes 28 minutes for the longest
• Documentation (which will need artifacts generated in previous steps) will take 14 minutes
• In total, this run took 1 hour and 16 minutes.

Let's focus on the build and the testing stage first. We're expecting the plutus-merkle-tree to run fast but it took 8 minutes and 52 seconds , 7m and 11 seconds being spent setting up the build environment. In other words, 81% of the build time is downloading binary dependencies from some nix cache.

Compiling the code requires cabal and other nix dependencies that will trigger massive downloads. Instead, let's, straight on, run the tests with nix and let nix decide what needs to be compiled. Sometimes, most of the code will not have changed and the test binary will already be available in some nix cache to be run without any compilation step.

This has been done in #867. Merging this PR, the build on master took 45 minutes and, specifically, the same plutus-merkle-tree job only took 1 minute and 44 seconds, only 20% of the time observed before.

Then, let's focus on documentation (14 minutes). Most of the time spent by this process is website optimization. It makes sense for the documentation that will actually be published on the website (master or release). But what about all the builds in branches for which the website will never be published? We need to keep this documentation step in branches because it gives us sanity checks, preventing broken links, for instance. But we can definitely remove the optimization step.

This has been done in #880 and it saves 10 minutes from this step.

Our first goal was to reduce CI execution time when pushing on branches and this has been improved. We're now having execution time significantly under 30 minutes where it used to be 45 minutes or even an hour.

We had some issues with compilation output, obfuscated by nix, which have been solved by #889.

Every morning, we rebuild our master branch and we can observe the whole execution time on the following graph (in seconds):

C.I. execution total execution time

Although we observed performance improvements on branches CI execution time, master execution time has not been reduced that much. We may save 10 minutes from the documentation step because optimizing the unstable documentation is maybe not worthwhile. But what's more problematic here is the process variability.

This variability can be explained by nix cache misses. That's something we need to investigate. It's hard to optimize a process with buffers, especially here where in case of a cache miss, recompilation has to happen and takes time. So every cache miss introduces several minutes of overhead which explains why we observe so much variation between two days with this master execution time.

Next steps:

• Inspect why we have an unexpected random cache miss
• Improve haddock generation time (15 minutes)
• Reduce bench time (we probably don't want to run the whole benchmark suite for every single commit, or a smaller one)
• Focus on the changed area (do not compile everything to generate the monthly report)

Community​

Hydrozoa #850​

This month, the project saw a new kind of contribution from the community. @GeorgeFlerovsky has written a research piece about an adaptation of the (Coordinated) Hydra Head protocol into a new flavor - Hydrozoa. The article is currently being examined and discussed on GitHub. Feedback so far has been positive. Of course, one does not simply change the Hydra Head protocol (pun intended), but the ideas contained could drive evolution and be followed up with concrete protocol extensions. Very much like the ideas presented in the original paper.

Hydra for Payments project update​

In this collaborative effort between IOG and ObsidianSystems, we are pushing the frontiers of using Hydra in payment use cases. It will lower the entry barrier for developers and ultimately users to leverage Hydra L2 for sending and receiving ada and Cardano native assets with very low fees and sub-second finality.

The project is in its second phase where the open-source hydra-pay library is being expanded with necessary features while the team pushes for building a dedicated mobile application.

With the recently released hydra-pay version 0.2.0, an end-to-end workflow of the Android App nicknamed "HydraNow" can be realized. This app will act as a layer 2 wallet quite like a Bitcoin Lightning Wallet and drives feature development in both hydra-pay and hydra in the background.

Two instances of HydraNow (in browser) connected via a hydra-pay channel

Spanish translation​

Last, but not least, we would like to thank @Agustinblockchain for their contribution of a Spanish translation #866 of the hydra.family website 🇪🇸 🎉

Conclusion​

The monthly review meeting for May was held on 2023-05-24 via Google Meet with these slides and here is the recording.

The monthly review meeting was lively and we had stakeholders from IOG as well as interested community members join. They were well served by two very nice demos shown by ObsidianSystems and IOG teams.

After having dealt with some issues last month, we are happy to have the first mainnet-capable version released and have already another feature completed this month.

It is great to see that more and more contributors from the community are stepping up to become advocates of the technology. A community is forming and we realize that the project could need some help from someone dedicated to keep track of all the great things happening. What a great problem to have!

This report summarizes the work on Hydra since March 2023. It serves as preparation for the monthly review meeting (see slides and the recording), where the team updates major project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

The project roadmap was only slightly updated this month:

The latest roadmap with features and ideas

Notable roadmap updates​

• There are still many 💭 idea items on the roadmap, however, not on the current and next planned release columns. The process involves clarifying and refining each idea item into a 💬 feature before starting work on it. This includes giving equal consideration to new user 💭 ideas and requests.

• Temporarily postponed the Hydra heads explorer #696 item until there is more clarity. While there aren't any major blockers,the explorer just requires further detailing, especially since we're currently in talks with existing Cardano explorer platforms.

• Converted the aggregated multi-signature #193 from the concrete roadmap into an idea discussion #787. This is based on feedback from the community, and we believe that discussing it there would be more beneficial.

• The main focus for the 0.10.0 release is implementing mainnet compatibility. This is mostly done and only requires some documentation and disclaimer updates. We're about to release 0.10.0, which will be the first mainnet-compatible version.

• Meanwhile, work on the configurable API #380 was completed, which gave rise to new ideas and follow-up feature requests from users. One of them (Add HeadState/Snapshot to Greetings #823) was fairly straightforward and necessary to deliver a consistent, usable increment on the API with the upcoming release.

• Prioritized Support timed transactions #196 higher as yet another use case would benefit from this.

The roadmap without idea items

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

Configurable API​

The API evolved a bit, driven by the issues our users reported #823 #813 #800 #789. Related changes were added to the API server so now our clients can:

• Control the historical messages output. History messages can be displayed upon re/connection or not depending on client needs.
• Snapshot UTXOs can optionally be disabled.
• Transactions can be displayed in CBOR or JSON format.

Our clients can also have a nice insight into the current Hydra node state and head UTXOs that are now displayed as part of a Greetings message.

Next steps on the API level are to further fulfill user needs by grooming and implementing needed changes related to filtering, pagination etc.

Versioned docs and specification​

Over the last couple of months the Hydra specification became an important artifact to use in discussion, review and potential audit of the Hydra Head protocol implementation. The document was now moved from overleaf into the Hydra repository, where it is properly versioned and built on each CI run. Changes can be proposed using our regular pull request worfklow and the final PDF is built and published to the website automatically.

Note that the above link points to the new /unstable version of the documentation, which holds the bleeding edge user manual, specification and API reference (which got a new sidebar) built directly from master. The normal, non-unstable version of the website is always referring to the last released version.

Specification on the Hydra website

Fixed scripts, Plutonomy and custom script contexts​

As we made the specification use a more direct way to represent transactions (instead of the constraint emitting machine formalism), we realized that our scripts are not correctly ensuring script continuity. We identified these 'gaps' as red sections (see above) in the specification and worked on fixing them.

While the actual fix #777 was fairly straightforward and could easily be covered by our mutation-based contract tests, the script size increased and we could not publish all three Hydra scripts in a single publish transaction (which allows for a single --hydra-scripts-tx-id parameter on the hydra-node).

To mitigate this, we looked into the UPLC optimizer Plutonomy. Applying it was fairly simple, our tests did also pass, script sizes and costs also became lower. But, script size does not matter so much as we are using reference scripts and using a (not really maintained?) optimizer which introduces yet another question mark after compilation from plutus-tx to uplc was not our cup of tea right now at least (and we might pull this out of the drawer later).

There is an alternative: decoding ScriptContext involves quite some code, but we don't need everything in all validators. So we introduced a customscript context that only decodes the fields we need.

scripts	@0.9.0	fixes	fixes + plutonomy	fixes + custom ScriptContext
νInitial	4621	4727	3672	4300
νCommit	2422	2513	1816	2068
νHead	8954	9492	7579	9456 (no custom SC)
μHead	4458	4537	3468	4104

As part of this process, we also updated dependencies #826 to the latest cardano-node master. Although it didn't have an impact on script sizes, it's a crucial step towards preparing for upcoming hard-forks.

Rollback bug hunt​

Opening our first head on mainnet failed. We didn't lose any funds, except for some fees, but the head just did not open. Exploring the logs, we figured out that a rollback happened while opening the head and there was a bug.

This is our test pyramid. It already contained some tests about the rollback but we decided to enrich our model-based tests to simulate rollbacks (before that, it used to run on a perfect blockchain). You can find more about our model-based test strategy in Model-Based Testing with QuickCheck.

Testing pyramide

The new property headOpensIfAllPartiesCommit helped prove the issue. At the end of the day, the problem came from a concurrency issue introduced while implementing ADR 18.

In the figure below, the DirectChain processes a new block, updating the chainState stored inside the headState. This also leads to an event being published to some event queue. Later, the HeadLogic (called Node in the picture) will process this event, updating the headState.

At the end of the process, we can see that the new headState points to a previousRecoverableState which contains the same chainState, chainState 1 instead of chainState 0. If a rollback then happens, the headState will be reverted to this previousRecoverableState and the fact that it contains chainState 1 instead of chainState 0 makes some on-chain observations impossible.

Race condition

This explains the issue we had when opening our head:

1. a commit A is observed on-chain;
2. a rollback happens so that the headState forgets about this commit but not the chainState (remember, it's the wrong chainState);
3. the commit is observed again on the chain but ignored by the chainState (because it has already seen it, it just ignores it);
4. the headState will never hear about this commit again and so will never open the head, waiting forever for the missing commit.

We decided to implement the following solution:

• A local chain state is re-introduced in the chain component, not shared with the head logic.
• A copy of the chain state is kept in the head state to keep the benefits of ADR 18 regarding persistence.
• The rollback event is removed from the API until #185.

Possible solution

Rollback management is quite a tricky business. It might be the case that we've tried to be a bit too smart. So we're doing a rollback in our way of handling rollbacks until we focus on this topic again when dealing with this roadmap item: Handle rollbacks II.

Community​

Hydra for Voting​

The project is advancing and a basic vote tallying scenario in the Catalyst use case was demonstrated in the review meeting. The project is driving the API discussions as it is not using any Haskell tooling, but an application in Java with Aiken as the validator scripting language. Besides the catalyst use case, other scenarios like the ballot voting for the summit are also explored now.

Hydra for Auctions​

A new demo was recorded in the wake of an upcoming Twitter space discussing auctions and NFT marketplaces with the community. The feature set currently includes starting the auction on L1, bidding on L1 or (and this is the novel thing!) transferring the auction from L1 to L2, such that it can be bid on L2.

Kupo x Hydra​

In a good old pairing session between IOG and CF engineers, the integration of Kupo with Hydra was explored. This seems to be promising and work started here kupo#117. This will make it possible to run kupo natively connected to a hydra-node, very much it would run with cardano-node or ogmios. Kupo is a lightweight indexer of chain data like unspent transaction outputs and allows its clients to query information on-demand. 🐹

CBIA meetings​

Hydra Team is now a regular participant to Cardano Blockchain Infrastructure Alliance meetings which happen more or less on a monthly basis. Hydra was featured during the meetup that happened while the team was on a workshop in Feldkirch and through this participation we hope to contribute to growth of the Cardano eco-system and position Hydra as a key infrastructure for builders.

Twitter space on scaling Cardano​

This month we took part in a Twitter space about scaling Cardano and how Hydra can contribute to this. Thanks for conducting this @thepizzaknight_ 🙏

Conclusion​

The monthly review meeting for April was held on 2023-04-26 via Google Meet with these slides and here is the recording.

Although it has been a busy month we could not cut a release, unfortunately.

We've experienced several setbacks due to the commits vs. rollbacks bug and oversized script sizes, which have slowed down our progress. Additionally, the back and forth on the API, which at times required creative and unconventional solutions, has also been time-intensive. However, we view this feedback as incredibly valuable, as it will ultimately make hydra-node more user-friendly and capable, albeit through a step-by-step process.

Associated projects in the greater Hydra community are moving ahead nicely due to the collaborative approach and tight loops of interaction between the individual teams.

All things considered, the project can be considered on track. We are very close to cutting our first mainnet-compatible release and the rising number of user requests and interested developers are good indicators that Hydra adoption is increasing.

This report summarizes the work on Hydra since February 2023. It serves as preparation for the monthly review meeting (slides/recording) , where the team updates major project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

The project saw one release this month and several items on our roadmap were updated.

Released version 0.9.0​

• This release brought in on-chain and off-chain changes that are now also fully reflected in our specification.

• We managed to decrease the costs of our plutus scripts by using a new error code framework and made also the head script a reference script.

• The mutation test suite is improved. Now, if tests fail they show the correct reason.

• The contestation deadline is now pushed out by each contesting party, so the contestation period parameter can be chosen irrespective of number of parties.

• Added a tutorial contributed by @perturbing.

• Full release notes and list of delivered features

The latest roadmap, with many items marked as idea.

Notable roadmap updates​

• Now that 0.9.0 is released, the focus is on mainnet compatiblity. The next planned version 0.10.0 will be the first hydra-node to be mainnet compatible.

• Also prioritized API configurability #380 higher and planned it into 0.10.0 as more users were requesting this.

• We aim for several more 0.x.0 versions to incorporate user-requested features before reaching a 1.0.0, which will be a fully maintained release where features can be considered stable and won't be removed without appropriate deprecation cycles.

• Marked multiple features as 💭 idea, meaning they are up for discussion. Many have been on the roadmap for a long time without user demand or input, while other ideas are incubating as github idea discussion. We are planning to convert these idea issues to GitHub discussions as these provide better dicussion tools and allow to gauge interest in topics. In turn, by cleaning up we provide more room for the most popular ideas to be planned onto the roadmap in a more timely manner.

• In short, if you want to see some feature added, show your support on the corresponding idea discussion.

• The 💬 feature items remaining are the currently identifed "must-haves" or too vague to remove 🔴 red items.

The roadmap without idea items.

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

• Making Hydra mainnet compatible. Besides making it technically possible to run on mainnet, this is about safeguarding our users and preventing them from shooting themselves in the foot with a mainnet gun. That is why we limited the amount of ada you can commit to a head on mainnet. Our smoke tests should be running on mainnet also so we made sure to return the leftover funds back to our faucet. There was also work on our CI that enables running the tests on mainnet using a dedicated github runner.

• Improving the Hydra UX. We noticed a possible pitfall when restarting the hydra-node using different parameters than in the persisted state. Now, the node would prevent this kind of misconfiguration.

• Optimize the on-chain scripts. Reduced the cost of opening/closing a Head by reducing size of scripts via error codes #748 and also having the head script as a reference only #701).

  The mutation test also got improved by making all cases expect the corresponding error codes #705 and new golden test suite ensures the script hashes don't change accidentally.

  Furthermore, we addressed a problem discovered in our property tests #724 by preventing committing outputs with reference scripts to a Head #766. This is still a documented known issue, but a workaround is available.

Community​

• Team workshop in Austria. Meetup of core contributors in Feldkirch, Austria for retrospective, grooming and discussions on future plans for Hydra. Part of the agenda was also a presentation of Hydra to members of the Cardano Foundation and a meeting with builders from the Cardano community.

  The highlight, though was the common effort of going the last mile to open a head on mainnet for the monthly review meeting! 🎉

  JUST IN: We have a Hydra Head live on the #Cardano Mainnet 🚀 pic.twitter.com/6kDKq7T7no

  — Emmanuel 𓂀 🍕 𓅓🇬🇭🦄🟣⚡️ (@thepizzaknight_) March 22, 2023

  We demonstrated our good old hydraw application on that Hydra head and we saw thousands of pixels painted by hundreds of users (no detailed metrics).

• Next step in Hydra for Payments. Also announced in the monthly meeting was the next phase on the Hydra for Payments project. The scope of this will be to extend hydra-pay and build a mobile payment channels app that makes direct use of it - working title: HydraNow.

Conclusion​

The monthly review meeting for March was conducted on 2023-03-27 via Google Meet - slides/recording.

This month was very important for the project and culminated in the first demonstration of a Hydra Head on mainnet! The demo was well received and we received positive feedback in the meeting, on following twitter announcements and on the published recording on youtube. Inspired by this, we saw multiple people offer to help and collaborate on communicating, educating and ultimately spreading the love. The invitation via the new discord category worked well - we had about 40 community members in the call - and we will continue with this workflow. Make sure to follow the Hydra #announcements on the IOG Technical discord server.

While being mainnet compatible is a major milestone for the project, there are still many known issues, shortcomings and requested features. The roadmap changes this month should make it clear that we are serious about the latter - Hydra will only reach 1.0.0 if it is used by an application on mainnet. Hence, we will focus on adding features required for payments, voting, auctions, ... and eventually, your use case.

Hydra Head is ready to be used on mainnet. Are you ready to use it?

This report summarizes the work on Hydra since January 2023. It serves as preparation for the monthly review meeting, where the teams update major project stakeholders on recent developments to gather their feedback on proposed plans.

Roadmap​

Looking at the roadmap, there are a few things that changed this month:

• In general, there will be several items in flux as we align the technical roadmap to support our high-level objectives for the year.

  • New items for example: create a use case-centric roadmap #710 and open specification with change process #693.

• Also note that we have not had a release in a while and we want to have smaller faster releasable items again.

• Aligning implementation with HeadV1 specification #452 was more work than anticipated. There was quite some back and forth with the writing of the specification and discussions with researchers leading to scope creep.
  Most importantly, we made on-chain behavior consistent with specification.

  • Follow-ups for some off-chain protocol logic:
    • Authenticate network messages #727
    • ReqSn only sends transaction ids #728

• Version 0.9.0 will be the first release to be developed against a full technical specification. We expect a release in early March.

• Version 0.10.0 will be the first mainnet compatible release. For this, we created and groomed the main feature item Mainnet compatibility #713:

  • Not only making it technically compatible but also adding guard rails to limit risk (“not shoot yourself in the foot .. with a real gun now”) and documenting known issues.

• For a fully maintained version 1.0.0 we still need to address some things, but we aim for an increased rate of planned releases in the next months as we march toward it.

The latest roadmap with multiple new items and starting to reflect Hydra objectives of 2023.

Development​

Issues and pull requests closed since last report

This month, the team worked on the following:

• Aligning specification and implementation. This major feature item was the topic we worked on over the last couple of months. This time we could finish the major, must-have parts and can continue now on other features. Notable things from this month were:

  Pushing out contestation deadlines: The original paper defined that all participants need to be contesting within the contestation period (after close, before the deadline). In discussions with researchers last year, we identified better semantics: the contestation deadline gets "pushed out" on each contest. This has the benefit that the period needs to only accommodate the minimum time required to get a single contest transaction into the blockchain. Before it was depending on the size of the Hydra head - each participant can contest once.

  Align off-chain protocol logic: While most concerns addressed were about the on-chain scripts of the Hydra protocol, this is about the layer two protocol logic. That is, which messages are exchanged between Hydra nodes and how a node reacts to them. Here we realized that the implementation was slightly different from the formalism used in the paper and specification. This makes it non-obvious whether the behavior is exactly as specified or whether the specification can even be implemented into a working system.

  To clarify this, we put in some effort to align the implementation with the specification and the other way around. This allowed us to reason through our behavior tests whether the protocol as specified is sound as well. Below you see an excerpt of the Haskell code against the LaTeX pseudocode formalism:

  

• Profiling Hydra transactions. We will want to maximize the capability of the Hydra Head protocol. This includes writing the on-chain scripts efficiently. Besides having transaction cost benchmarks, we worked on a profiling workflow for Hydra protocol transactions and published a tutorial on the Hydra website. This allows any contributor to Hydra to profile the scripts used in these transactions and identify code that uses a lot of the memory or CPU budget in flame graphs like this:

  [Interactive]Head not closable/limiting script size discussion + acceptance property

Community​

• Audit guidelines. Hydra is an open-source project and, as such, should be freely used, reviewed and audited. The goal of audits and peer reviews is to increase confidence in the security of a system. However, conducting an audit is also very expensive and not in the scope of our current budget. All the while, the worth of an audit is not the same to all users, some will not even trust an audit they have not paid for themselves, while others do not care at all and would blindly trust the word of the maintainers.

  While the Hydra project continues to seek for funding to eventually pay for an audit of the protocol specification and the hydra-node implementation, it is a prerequisite that the necessary artifacts for such an audit exist. As we have been creating them in recent months, we also worked on guidelines for auditors, now published on our website. These guidelines should serve as a starting point and scope description of what would be most crucial to audit.

• New tutorial. This month we received a great contribution from @perturbing, who created an end-to-end tutorial on using a hydra-node and opening a Hydra head on the testnet. We gladly adopted this tutorial and aim to keep it up to date with upcoming versions of the hydra-node. It will provide a great starting point for new users and makes the project more approachable. Thanks! 💚

• Hydra for Payments demo. As we are about to kick off the next phase of the payments use case with Obsidian Systems, they demonstrated the use of hydra-pay in a new backoffice-like view, listing multiple payment channels and providing a great teaser of using payment channels for pay-per-use APIs or cashback use cases (micro-transactions).

  

• Hydra for Voting. The project with Catalyst and Cardano Foundation on voting with Hydra (or rather tallying with Hydra) showed some progress. They have contributed Merkle Trees to Aiken, which will be useful to prove the correct tallying of votes in the L2. They started work on a hydra-java-client and provided valuable feedback on the hydra-node API over the last month.

• Hydra on Demeter. The folks from @txpipe have been working on integrating Hydra into their platform-as-a-service offering, Demeter. This is a great way to get started with running a hydra-node as a service and we plan on integrating one instance into our long-running hydraw demo. It's still not fully released, but the teams are working together to make this integration great. Thanks for providing this service! 💚

• New discord category. The Hydra project will be using Discord a bit more in the coming weeks and months. For that, a new category on the IOG Technical Discord server has been created. Look out for news and, most importantly, invites for the next month's review meeting in the new Hydra #announcements channel!

  

Conclusion​

The monthly review meeting for February was conducted on 2023-02-27 via Google Meet - slides / recording.

This month's installment featured two demos, one about the Hydra Payment channels, and another about the progress on the Hydra for Auctions project. It was great to showcase not only core development progress on the specification and protocol implementation in the hydra-node, but also to see contributions from the Community on tutorials and tooling (Demeter integration).

An exciting, but also challenging next step will be to aim for mainnet compatibility and at the same time deliver quick releases to enable these lighthouse projects and the increasing number of use cases explored by the builders of Cardano.

This report summarizes the work on Hydra since December 2022. It serves as preparation for the monthly review meeting, where the teams update major project stakeholders on recent developments to gather their feedback on the proposed plan forward.

Roadmap​

Looking at the roadmap, there are just a few things to report this month:

• Regroomed Create updated specification for coordinated head protocol #448 to 💛, as good progress was made on the specification

• Marked Align implementation with HeadV1 protocol specification #452 as 🔴, as the team experienced scope creep and has not progressed as fast as expected.

  • The plan is to break this initiative into smaller tasks to address and explain implementation challenges.

The latest roadmap with minor changes, which needs a reflection of the latest objectives.

Development​

Issues closed since last report

This month, the team worked on the following:

• First write-up of Hydra spec online. The team has been aggregating the latest definitions of on- and off-chain semantics into a new technical specification. The latest version can still be found on Overleaf, before its integration into the core Hydra repository. Should you have feedback, please send it our way through one of the communication channels.

• Closed more gaps in the Head Contract. The team addressed two more gaps in the on-chain scripts based on discussions on the specification.

  • All validators are authenticated now and ensure contract continuity by checking for the state tokens in the output (or getting burned).

  • Abort now fully reimburses all committed UTXOs, even if two parties would have committed the exact same output (quite a theoretical attack vector).

  This work is captured in #452, but this work item is taking quite a bit longer than expected. Also, it's a single feature item, lacking some rationale on certain requirements, so we intend to break this down into smaller pieces over the next days and weeks.

• Mutation test framework improvements. While closing gaps in the contracts, the team realized that some of the mutation tests were correctly tripping validators and making transactions invalid, but not always for the right reason. In fact, some of the mutations were ‘too harsh’ and made the even transaction not even pass phase-1 validation. The team addressed this by introducing a first (naiive) way to assert the right cause of the failure by checking strings in the validator log. #679

• Add headId in API and TUI. The team not only added the unique headId to the appropriate server outputs and the TUI #678, but also added sequence numbers and timestamps to produced outputs #618. This feature was asked for by users and simplifies identifying of heads and integration with the hydra-node.

• Hydra explorer experiment. Some of the team used the holiday season to conduct some experiments on summarizing Hydra Heads observed on a network. This quick hack already demonstrates the value of a Hydra explorer as a tool to measure adoption by open heads on a given network. Naturally, it would reuse code from the hydra-node and challenge the architecture used in it for tracking multiple heads. Also, it will drive discussion about the versioning of hydra-plutus and the possibility to track multiple versions of the Hydra protocol on the chain (hashes/addresses change in each version).

Hydra explorer first experiment UI

Community​

• The Hydra for Voting project, which got kicked off in December is picking up steam with deep dives into Catalyst voting and Hydra technology. Discussions are currently held in various settings and the team is continuing to develop a picture of what is achievable.

• Hydra for auctions lite paper. IOG is collaborating with MLabs on exploring how Hydra can improve auction use cases. At this stage, the teams discussed multiple approaches on structuring the problem space with various Hydra topologies and specific trade-offs. The full case study was published on Essential Cardano. The teams plan to upstream the findings back into the use case section on the Hydra website from which this project originated.

Themes for 2023​

After kicking off planning in a workshop and reflecting on what Hydra achieved in 2023, the team has progressed in fleshing out relevant themes and objectives for this year to position Hydra as a sustainable open-source project in the age of Voltaire:

• A mainnet mature application

  • Why? The core contributors should be able to use the protocol on mainnet, so it should be maintainable and mature enough to lock some ada in a Hydra Head using a reference DApp (eg, hydraw). By creating and growing this DApp, the team will dogfood features and hence improve additional usability.
  • How to measure? Number of heads on mainnet > 0 and core contributors feel confident to lock at least 100₳ in a Hydra Head on mainnet.
  • Next steps:
    • Publish Hydra Head V1 specification
    • Close gaps in implementation and release 0.9.0
    • Create a request for proposals (RFP) to audit the specification and its implementation

• Increase adoption

  • Why? The team wants to showcase what is possible with Hydra through benchmarks and lighthouse projects, enabling use cases and reducing friction to use Hydra. Also, ensuring interoperability through open standards and reference implementations.
  • How to measure? Number of third-party created heads on any network > 0
  • Next steps:
    • Build a basic Hydra Head explorer to measure adoption
    • Benchmark off-chain performance (of a selected scenario)
    • Demonstrate Hydra for payments to potential users
    • Support Hydra for voting project
    • Hydra for auctions project exploring a delegated voucher auction

• Sustainable open-source-development

  • Why? The team wants to scale Cardano in the long run. Hence this project should not be owned by a single entity, but by the community and the Cardano network itself. It is thus essential to make contributions to the project possible and ultimately it should be easy to become a contributor.
  • How to measure? Number of contributors on GitHub
  • Next steps:
    • Open up monthly reviews
    • Add tutorials to the website
    • Publish and maintain a use-case-centric roadmap and feature map

Conclusion​

This monthly review was the first public one with not only stakeholders from IOG and the CF, but also collaborators of the latest projects. This was the first step to a more open and transparent development process.

The teams showed some classic demos – like the ‘hydraw’ application which has been running on the same Hydra Head for four weeks now – and some new developments on the Hydra explorer and minor improvements to hydra-node API and TUI.

It was very interesting to bring various teams together to exchange ideas and thoughts on the individual roadmaps for the new year. The teams also shared the relevant themes and objectives of the Hydra project for 2023 (as also listed above). They will now make sure to reflect these steps into this concrete roadmap such that it will contain marketable features bringing us closer to these goals.

This report summarizes the work on Hydra since November 2022 and also serves as a preparation & write-up of the monthly review meeting, in which we update major stakeholders of the project on recent developments and gather their feedback on our proposed plan forward each month.

Roadmap​

Looking at our roadmap we can report the following updates:

Notable changes​

• Completed Validate coordinated head protocol against formal model #194
• Detailed a follow-up Validate soundness with Model Based Testing #656
• Focus on specification and gap-closing to support audit
• Scoped down 0.9.0 to be only about script updates and moved Commit from external wallet #215 into next version
• Groomed React to mainchain protocol changes #195
• More users requesting Support timed transactions #196

Latest roadmap with drafted 0.10.0 version.

Development​

Issues closed since last report

This month we have been working on:

• Fixed AcquirePointTooOld errors by changing the way the internal wallet initializes its state #621. See the PR description for details. This did not strictly happen this month, but we have not mentioned it last time.

• Closed first gap in Head Contract: We collected gaps between specification and current implementation in #452 and started now closing them. This first issue, was to prevent DoS / stalling of heads, by bounding the lower and upper transaction validity time of the close transaction. As always, time handling is hard and this required detailed discussion in the form of an ADR. See ADR21 for more details.

• Reduced cost of commits by at least 30% using reference scripts also in the commit transaction. We only used reference scripts in abort so far. This reduces the min ADA cost of commit because the transactions are roughly 5.2KB smaller.

  Before:

  After:

• Use nix flakes & build docker images using nix: Triggered by a migration to Cicero for a nix-based CI, we started using nix flakes for nix build and nix develop instead of the legacy nix-shell in #646. While we ended up sticking with Github Actions as the only CI and a Cachix cache, we extended this work to build our docker images more efficiently using nix in #654.

  Image sizes went down between 20-50%:

  

  While docker CI builds on master went down to 1-10min (from 10-30m+)!

Hydra V1 Specification​

Finally, we bit the bullet and started a write-up of the specification. Over the last couple of months we had worked with marked up versions (see below) of the original paper and a GDoc to discuss and review the protocol as it was implemented.

As we also updated the security properties and proofs, we started aggregating a LaTeX document on overleaf. While it holds a similar background and preliminaries as the original paper, it only specifies a single version of the protocol with no variants, less modularization, as well as explicit assumptions and important notes for implementors.

We aim to be publishing the spec as part of the main hydra repository soon.

Community​

• Hydra for Payments project completed the first phase of this work stream by tagging version 0.1.0. This open source project conducted by Obsidian Systems and sponsored by IOG created a great starting point for developers to build services/APIs of payment channels using Hydra Head. This month we reviewed the improved documentation and are looking forward to see this work applied to eventually build fast & cheap payments for light-wallet users and power payment channel use cases from business-to-business (B2B) in the short-term.

• Hydra for Voting project kicked off this joint project between IOG Research, Catalyst & Hydra Tribes, and the Cardano Foundation. The goal is to assess suitability of Hydra Head to support voting use cases by implementing parts of Catalyst voting system on Hydra Head and benchmark execution at scale. The kick-off meeting took place on December 15.

• Blog post by CF: To round off the year, Matthias wrote a nice retrospective blog post about Hydra at the CF. Besides summarizing our progress made in 2022 it also gives an outlook where Hydra is headed for in 2023.

Conclusion​

Although Hydra is in essence an open source project right now, we would love to make it "even more open source" in the coming months. Our vision of Hydra requires to have the builders of the Cardano ecosystem to be part of this journey. For that matter, we want not only to open up these monthly reports, but the review meeting backing it as well!

This, besides our engagements with the community above, will allow the project to grow beyond IOG & CF contributions. To become the true Cardano open source project we believe will be worthy to serve as the scalability component required by the Cardano network.

Introduction​

This report summarizes the activities of the Hydra team since October 2022 and also serves as a preparation & write-up of the monthly review meeting, in which we update major stakeholders of the project on recent developments and gather their feedback on our proposed plan forward each month.

Roadmap update​

Looking at our roadmap we can report the following releases and updates:

Release 0.8.0​

• Long overdue feature release adding persistence to the hydra-node
• Backup & restore the state of a Hydra Head #187
• Improve user experience following hydraw experiment #518 from being only a discussion to a (to-be-groomed & planned) feature
• Full release notes

Release 0.8.1​

• Follow-up release addressing user feedback on persistence from 0.8.0
• Allow clients to see latest state after restart #580
• Bug fixes of following chain state #599
• Full release notes

Notable changes​

• Impact mapping from workshop (see below) and reviewing the starmap above had us discuss whether the scope for 1.0.0 is correct or not
• Need to be conscious of scope creep vs. getting something audited
• Focus on getting the spec done, 0.9.0 tagged and an audit under way now → minor re-ordering in priorities
• Iterative releases afterwards still - even while getting audited, non-contract changes first
• Promoted https://github.com/input-output-hk/hydra/issues/635 from being only a discussion to a (to-be-groomed & planned) feature

Latest roadmap with 0.8.0 and 0.8.1 already released and slight re-ordering on 0.9.0

Development​

Issues closed since last report: Issues - input-output-hk/hydra

Besides the things in the releases above (see roadmap update), we have been working on:

• Flaky TUI CI tests: We have sunk quite some time again in investigating cryptic CI failures in our TUI end-to-end tests. They have been crashing abruptly without any information and thus hard-to-debug. This investigation was unsuccessful and we realized at some point that the TUI tests are not that important anyways - we have API-level end-to-end tests. So we disabled these tests for now.

• Published hydra-cardano-api to CHaP: With the Cardano Haskell Packages (CHaP) becoming available now and wanting to be a good citizen, we pushed for getting our flavor of the cardano-api also published there. This is non-trivial though, as we are at an integration point even "further up" than the cardano-node #504

• Implemented ADR18: While the first stints on persistence already make it possible to restart hydra-node without needing to close a Head, we have implemented this in a nicer way now only keeping a single state (ground truth) for both L2 and L1 information #541 of the Head protocol and the specification, we kept discovering bigger and bigger issues and hence realized the need for a discussion on transaction validity in context of closing / contesting Heads. #615

Formal verification & specification​

An important part of the project right now is the formalization and audit preparation:

• Internal audit has started with involvement of two persons from IOG IT team, each one addressing different part of Hydra:

  • One auditor is working on the project as a whole, targeting potential vulnerabilities with the off-chain code, the infrastructure, dependencies, etc. This has not lead to any significant issue nor action plan so far.
  • Another auditor is specifically targeting the formal specification and the on-chain code to identify vulnerabilities in the protocol itself.

• Discussions with researchers have led to the development of a joint Coordinated Hydra Head V1 specification defining formally the protocol as it is actually implemented.

  • Work on this document has already allowed us to identify gaps
  • It is the basis on which BCryptic's analyst is working to audit Hydra protocol
  • It allows us to make explicit a lot of implicit assumptions that are in the code but not in the original paper and "Shape" the language used to describe the protocol

• We have drafted an RFP for submission to external auditors https://github.com/input-output-hk/hydra/blob/audit/rfp/security/RFP.md defining the scope and targets of the audit

• We are still unsure about whether our approach is the right one as having a "formal specification" in a manually checked document spanning a dozen pages seems quite brittle.

  • Some progress has been made on "formalizing" properties to be automatically "QuickChecked" using an executable model of the expected behavior of the system but this approach seems more suited for "team-internal consumption", e.g. building confidence within the core committers and contributors

  • Ideally, we would want a proper formalization of the protocol, using an existing theorem proving/model checking framework, through which properties could be asserted. Quite a lot of work has already been done in blockchain space, including some work on Lightning and TLA+ or some other state-machine/temporal logic based language appear promising:

    • https://www.youtube.com/watch?v=wecVT_4QDcU
    • https://github.com/rberenguel/tla_lightning
    • https://www.amazon.com/Practical-TLA-Planning-Driven-Development/dp/1484238281

  • This effort should be done in accordance to the DApps certification process

Product​

Most updates on the product side of things have been addressed in the starmap update above.

• Hydra for Payments lighthouse project: Latest demonstration shown by Obsidian Systems on how to use payment channels in a "light way" from a web frontend. The API evolved and we were able to use the preview testnet now. The project is progressing very nicely and we are optimistic to close it successfully soon with improved documentation and usability.

• NFT auctions lighthouse project: After doing surveys and interviews, MLabs is currently concluding the discovery phase with a light / white paper on how NFT auctions could be implemented using Hydra (today or in the future with more features).

• Drafting Voting on Hydra project: Work has started on building a Proof-of-concept for voting on Hydra targeting Catalyst’s use case but with an eye towards building a generic solution suitable for large scale voting systems based on Hydra Head

  This development should be undertaken jointly with the Cardano Foundation and SundaeSwap who are also interested in building such a system and recently demonstrated their capacity of doing voting via the Cardano Summit voting system.

• TxPipe demonstrated integration with demeter.run: Without our help nor even knowing of this effort, TxPipe has recently shown an early prototype of hydra-nodes instrumented via their demeter.run platform. This has come a bit as a surprise and is exactly why we love open source ❤️ → Video

Team & open source​

Some notable developments this month have been:

• Renamed the repository: from hydra-poc to hydra! This was revealed in the summit presentation (see below) and should indicate that the project is not only a Proof of Concept (POC) anymore, but has become more - as also demonstrated by the various early adopters and demos lately. 🐲

• Inner source (IOG) contribution: The education team has been working on a Hydra Tutorial and we have been involved in reviewing and trying it out. This is a great effort and will help people get started with Hydra. Thanks folks 💚

Cardano Summit & workshop​

This month there was also the Cardano Summit, this time organized by the Cardano Foundation and the content was even voted on by the community! We have been both, invited and nominated as panelist and speakers and this makes us extremely proud! In general, the reception of Hydra seems to be very positive in the community from what we could experience first hand on the summit.

Agenda items we participated in:

• Best of blockchain, best of open source - Open Source panel with IOG, CF & TxPipe

• Cardano Ballot Speaker Winner: Dev Team Introduction to Hydra and reveal of the repository rename 🎉

After attending the summit, we also used the fact that the whole team is in one location for a team workshop. We booked a coworking space and spent 3 days together. Not much coding & hacking this time, but we are happy to have produced these results:

• Timeline / year in review: As we had new team members with us, the request was to give a recap of how the Hydra project evolved over the last two years. The whole timeline can bee seen below, and this also sparked the idea of creating a "Year in review" blog post, which is currently in preparation.

• Retrospective: In-person workshops are the perfect place for doing retrospectives to reflect on what was good, bad and collecting ideas & actions in how to improve our work environment and processes.

  

  Retrospective board from 2022-11-22

• Impact map: Conscious about the fact that the project got defined about one year ago in a first project plan, and inspired by timeline and (short-term) ideas, we also set off to reflect on the Why, How and What of this project.

  After reviewing the project vision, we reached for the tool of our choice to ideate on tangible objectives / deliverables - impact maps! Also, we felt the need to use a different goal this time. One that is reflecting more closely (or, in fact, broadly) what our current mission is. So instead of the % of Cardano transactions are done on L2, we set a new goal: being the number one DApp on Cardano (by all known metrics: TVL, traffic, volume etc..) Fundamentally, both goals illustrate the same idea, but the latter better supports the narrative that Hydra is also just a DApp (not a network upgrade) and needs usage & adoption to reach our vision.

  Below you see the result of our session. We checked back to the old impact map after creating this one, and many things are still relevant / similar on the new sample (it's never complete!) - some of the deliverables we even achieved.

  

  Impact map with new goal

Conclusion

Reflecting on our original plans (about a year ago) and our current roadmap we can say we are behind our anticipated schedule. However, recent developments on the lighthouse projects and community members contributing & using Hydra are convincing us that we are on the right track. The summit paints a similar picture and we feel reinvigorated by the great people we met, discussions we had and relationships we created & solidified on the summit and during our Hydra workshop.