As 2024 comes to a close, we’re excited to share the latest updates from the Livepeer AI Network. It’s been an incredible year, and as a community, we’ve quickly advanced AI on the Livepeer network, enabling several applications to perform thousands of AI jobs on our trusted network of orchestrators 🚀. This final update, building on our previous progress, highlights key new pipelines, significant improvements to existing ones, and critical core network upgrades that will support our scaling efforts in 2025. Additionally, it offers the first preview of the work under the new Livepeer Cascade roadmap, which aims to position Livepeer as a leader in real-time AI.

We extend our heartfelt gratitude to the entire Livepeer community for making this AI journey possible in 2024. To our delegators, thank you for trusting our vision and providing the essential public funding that launched this project. To the orchestrators, your quick scaling efforts and operational excellence have been invaluable. To the founders and builders, we’re inspired by your early adoption and innovation. And finally, to our open-source contributors—your dedication and creativity have been central to bringing this vision to life.

Wishing you a joyful holiday season and a successful year ahead!

With gratitude,
The AI SPE Team 🎄👋

Livepeer Real-Time AI Community Preview Launch

We’re excited to kick off this changelog with one of our most thrilling updates—Livepeer Real-Time AI is now in Community Preview! This milestone was achieved thanks to the foundational work of the Livepeer INC team on core network updates, supported by the AI SPE team and Ryan on real-time AI pipeline development, documentation, and testing, Yondon’s open-source work on ComfyStream, and Livepeer.cloud’s efforts on the Stream Manager app. Together, these teams delivered key upgrades powering the first real-time AI pipelines on the Livepeer network.

These initial pipelines, currently in Community Preview, enable real-time artistic video animation with Stream Diffusion, avatar creation using Liveportrait, video depth map overlays with Depth Anything, and prompt-based selective object animation using Segment Anything 2 and Florence. Creators will soon be able to build custom real-time AI video pipelines using ComfyUI and deploy them on the Livepeer network.

Stay tuned for an exciting community update on how you can try out these new pipelines! To make the holidays even more special, the ecosystem team will also be hosting an interactive game—keep an eye on our socials for updates and be sure to check it out! 👀

Big Code Changes & Scalability Improvements

The most significant update in this release is the merge of the AI-video branch into the master branch of go-livepeer, streamlining development and enabling orchestrators to manage both AI and transcoding jobs using the same software. While running both job types simultaneously isn’t supported yet, this marks a major step toward unifying orchestration processes.
We’ve also introduced the AI remote worker, allowing orchestrators to link multiple worker machines to a single main orchestrator—similar to the remote transcoder setup. This enhancement significantly boosts scalability, enabling GPU pools to join the Livepeer AI network. A special thanks to the JoinHive team (formerly Livepool) and Mike | Xodeapp for their collaboration on this feature.

Additionally, we stabilized and documented the external container feature, enabling orchestrators to scale operations beyond Livepeer’s built-in Docker Manager, supporting more complex setups and multiple containers on a single GPU.

New and Improved Pipelines

We’re excited to announce several new batch AI pipelines that have been successfully integrated into the Livepeer AI network, expanding its core capabilities:

• Text-to-Speech: Generate spoken audio from text prompts.

• Image-to-Text: Perform image classification tasks (contributed by Livepeer INC).

We’ve also made significant improvements to several existing pipelines:

• A2T: Enhanced word-based timestamps and major inference speed optimizations.

• T2I & SAM2: Upgraded with new models to enhance functionality and performance.

Additionally, several new pipelines from our team and the community are under review and will be available soon, including text-to-video, lipsync, frame interpolation, sentiment analysis, LivePortrait (batch), sketch-to-image, object detection, inpainting, and outpainting.

New Gateway and Orchestrator Improvements

We’ve introduced an automatic container pull mechanism, removing the need for orchestrators to manually pull AI containers specified in their configuration. Currently, containers are pulled sequentially at startup and don’t auto-update to the latest version, though we plan to make this process asynchronous in future updates to speed up orchestrator initialization. Additionally, orchestrators can now set pricePerPixel as floats, enabling more effortless pricing of AI services. Metrics tracking has also been enhanced, with new support for streaming data to Apache Kafka, added by Livepeer INC, to improve real-time data collection and analysis.

New AI tester and AI dashboard (By Livepeer.Cloud SPE)

We’re excited about the release of the AI Tester and Performance Leaderboard, developed by the Livepeer.Cloud SPE team! As highlighted in their latest treasury proposal, these tools provide greater visibility by allowing delegators to see which orchestrators are actively performing AI jobs through the Livepeer Explorer, while enabling orchestrators to evaluate and optimize their performance. This promotes enhanced transparency and operational efficiency throughout the network. We’re eager to see how these features will foster innovation and collaboration across the Livepeer ecosystem!

Additional Improvements and Bug Fixes

This changelog highlights the major deliverables for this release. Moving forward, smaller improvements and bug fixes will not be listed here, as the AI software has been merged into the main branch. For detailed updates, please refer to the release notes in the respective Livepeer AI repositories.