As 2024 comes to a close, we’re excited to share the latest updates from the Livepeer AI Network. It’s been an incredible year, and as a community, we’ve quickly advanced AI on the Livepeer network, enabling several applications to perform thousands of AI jobs on our trusted network of orchestrators 🚀. This final update, building on our previous progress, highlights key new pipelines, significant improvements to existing ones, and critical core network upgrades that will support our scaling efforts in 2025. Additionally, it offers the first preview of the work under the new Livepeer Cascade roadmap, which aims to position Livepeer as a leader in real-time AI.

We extend our heartfelt gratitude to the entire Livepeer community for making this AI journey possible in 2024. To our delegators, thank you for trusting our vision and providing the essential public funding that launched this project. To the orchestrators, your quick scaling efforts and operational excellence have been invaluable. To the founders and builders, we’re inspired by your early adoption and innovation. And finally, to our open-source contributors—your dedication and creativity have been central to bringing this vision to life.

Wishing you a joyful holiday season and a successful year ahead!

With gratitude,
The AI SPE Team 🎄👋

Livepeer Real-Time AI Community Preview Launch

We’re excited to kick off this changelog with one of our most thrilling updates—Livepeer Real-Time AI is now in Community Preview! This milestone was achieved thanks to the foundational work of the Livepeer INC team on core network updates, supported by the AI SPE team and Ryan on real-time AI pipeline development, documentation, and testing, Yondon’s open-source work on ComfyStream, and Livepeer.cloud’s efforts on the Stream Manager app. Together, these teams delivered key upgrades powering the first real-time AI pipelines on the Livepeer network.

These initial pipelines, currently in Community Preview, enable real-time artistic video animation with Stream Diffusion, avatar creation using Liveportrait, video depth map overlays with Depth Anything, and prompt-based selective object animation using Segment Anything 2 and Florence. Creators will soon be able to build custom real-time AI video pipelines using ComfyUI and deploy them on the Livepeer network.

Stay tuned for an exciting community update on how you can try out these new pipelines! To make the holidays even more special, the ecosystem team will also be hosting an interactive game—keep an eye on our socials for updates and be sure to check it out! 👀

Big Code Changes & Scalability Improvements

The most significant update in this release is the merge of the AI-video branch into the master branch of go-livepeer, streamlining development and enabling orchestrators to manage both AI and transcoding jobs using the same software. While running both job types simultaneously isn’t supported yet, this marks a major step toward unifying orchestration processes.
We’ve also introduced the AI remote worker, allowing orchestrators to link multiple worker machines to a single main orchestrator—similar to the remote transcoder setup. This enhancement significantly boosts scalability, enabling GPU pools to join the Livepeer AI network. A special thanks to the JoinHive team (formerly Livepool) and Mike | Xodeapp for their collaboration on this feature.

Additionally, we stabilized and documented the external container feature, enabling orchestrators to scale operations beyond Livepeer’s built-in Docker Manager, supporting more complex setups and multiple containers on a single GPU.

New and Improved Pipelines

We’re excited to announce several new batch AI pipelines that have been successfully integrated into the Livepeer AI network, expanding its core capabilities:

• Text-to-Speech: Generate spoken audio from text prompts.

• Image-to-Text: Perform image classification tasks (contributed by Livepeer INC).

We’ve also made significant improvements to several existing pipelines:

• A2T: Enhanced word-based timestamps and major inference speed optimizations.

• T2I & SAM2: Upgraded with new models to enhance functionality and performance.

Additionally, several new pipelines from our team and the community are under review and will be available soon, including text-to-video, lipsync, frame interpolation, sentiment analysis, LivePortrait (batch), sketch-to-image, object detection, inpainting, and outpainting.

New Gateway and Orchestrator Improvements

We’ve introduced an automatic container pull mechanism, removing the need for orchestrators to manually pull AI containers specified in their configuration. Currently, containers are pulled sequentially at startup and don’t auto-update to the latest version, though we plan to make this process asynchronous in future updates to speed up orchestrator initialization. Additionally, orchestrators can now set pricePerPixel as floats, enabling more effortless pricing of AI services. Metrics tracking has also been enhanced, with new support for streaming data to Apache Kafka, added by Livepeer INC, to improve real-time data collection and analysis.

New AI tester and AI dashboard (By Livepeer.Cloud SPE)

We’re excited about the release of the AI Tester and Performance Leaderboard, developed by the Livepeer.Cloud SPE team! As highlighted in their latest treasury proposal, these tools provide greater visibility by allowing delegators to see which orchestrators are actively performing AI jobs through the Livepeer Explorer, while enabling orchestrators to evaluate and optimize their performance. This promotes enhanced transparency and operational efficiency throughout the network. We’re eager to see how these features will foster innovation and collaboration across the Livepeer ecosystem!

Additional Improvements and Bug Fixes

This changelog highlights the major deliverables for this release. Moving forward, smaller improvements and bug fixes will not be listed here, as the AI software has been merged into the main branch. For detailed updates, please refer to the release notes in the respective Livepeer AI repositories.

Since our last update, we have been diligently working toward achieving our Q3 milestones as part of Phase 2 in Livepeer's AI journey. With most of our objectives now completed (a detailed retrospective is on the way), we are in the final stages of delivering the remaining items and reviewing outstanding pull requests. As we complete these final pieces, we are confident that the AI network has reached a strong, stable foundation for builders and end-users alike.

In recognition of this progress, the AI subnet has officially been renamed to the Livepeer AI Network 🎉, marking its transition from alpha to beta. We’re also excited to announce that the LLM SPE (led by the Livepool team) and Livepeer Studio teams have joined forces with us, Livepeer Cloud, and the broader community 🤝. Their expertise in distributed systems and commitment to open-source development, as demonstrated in their work on the transcoding network, will play a pivotal role in further shaping the AI roadmap and driving even greater innovation. Expect to see exciting new developments as we collaborate with our community to advance our mission of democratizing AI and building a globally accessible, open AI video infrastructure 🚀.

Here’s a quick recap of the key accomplishments since the last update:

Segment-Anything-2 Pipeline Release

We successfully launched the new Segment-Anything-2 pipeline and, in the process, enhanced our container workflow to enable seamless, permissionless pipeline integration through custom containers in the future. Since its release, this pipeline has generated over $10K in fees and processed more than 400K requests in just one week, as highlighted on the Livepeer AI Dune Dashboard—showcasing the immense potential of the AI Network 🚀! A big shoutout to the Livepeer Ecosystem team for their crucial role in driving demand by collaborating with startups to make this pipeline a success.

LoRA Integration

In partnership with Marco Stronk, we’ve introduced LoRA support, allowing users to apply fine-tuned models to any Text-to-Image or Image-to-Image requests. This unlocks a new realm of creative possibilities (see banner image). With our automatic LoRA loading mechanism, orchestrators can now seamlessly handle thousands of LoRAs without manual configuration ⚡.

Client SDK Release

We’re thrilled to announce the release of Python, TypeScript, and Golang SDKs, streamlining the developer experience and making it easier than ever to interact with the Livepeer network 🛠️. Developed in collaboration with the Livepeer Studio team, these SDKs are crafted to provide a smooth, modern interface that aligns with industry best practices.

New Gateway improvements

We have introduced a new -maxPricePerCapability flag, allowing gateways to set the maximum price at the capability level, providing more precise control over spending. Additionally, the new -DiscoveryTimeout flag has been implemented to extend the discovery timeout when searching for orchestrators, enabling the use of a larger pool of resources for AI jobs.

Improved Error Handling & Documentation

We’ve completely overhauled error handling on the AI subnet. Instead of vague internal server errors, users now receive clear, descriptive messages that make troubleshooting easier and more transparent. Additionally, we’ve revamped our documentation to include the new SDKs and improved pipeline parameter descriptions, further streamlining the developer experience.

Additional Improvements and Bug Fixes

Model and Security Enhancements

- Added support for Black Forest Labs Flux.

- Added SG161222/Realistic_Vision_V6.0_B1_noVAE T2I support.

- Implemented the -enforceMaxPrice flag, now enabled by default, to prevent gateways from accepting prices above the set maximum.

Bug Fixes

- Fixed a runtime error that occurred when AI orchestrators failed to set a price.

- Corrected duration calculations for A2T with certain MP3 files.

- Ensured T2I/I2I pricing now accounts for the number of images.

Over the past two months, we've made significant strides in the AI subnet that we're thrilled to share. We've kicked off collaborations with 7 startups as design partners and successfully launched our bounty program in partnership with the ecosystem team. Additionally, we have been diligently working towards a mainnet release and have implemented major architectural changes, including the AI remote worker, external container release, and pipeline generalization. Given the scale of these changes, we will be rolling them out over several releases in the coming months, with this one being the first.

Here’s a snapshot of what’s new:

• Critical Fixes and Metrics: Several critical bugs have been addressed, and both AI gateways now have metrics to track their AI operations more effectively.

• Text-to-Audio Pipeline: We’re on track to release our text-to-audio pipeline, also known as Whisper, to the AI subnet. This will support two of our startups and significantly boost network demand on the AI subnet.

• Experimental SDK release: We have implemented experimental SDKs and tested them with our partners. These releases will be published to package repositories like PyPI and npm later this month.

The next release will involve syncing the AI-video branch with the main branch to enable gateways to filter orchestrators by go-livepeer version, helping to prevent potential breaking issues. Stay tuned for more updates as we continue to enhance and optimize the AI subnet ⚡!

We are thrilled to announce that as of this week, a dedicated community member, @interptr,, has joined our team full-time to drive the AI roadmap forward 🚀.

Main Changes

Features

• Add Upscale pipeline - @Livepeer.cloud (colab 🫂)

• Implement new audio-to-text pipeline.

• Provide dApps with NSFW warnings for T2I, I2I, I2V and Upscale pipelines.

Improvements

• Take inference request latency into account in selection algorithm.

• Allow users to use --gateway instead of --broadcaster.

• Rename pricePerBroadcaster flag with pricePerGateway.

• Remove pricePerUnit dependency for AI orchestrators.

• Provide a way to specify the AI Runner container version.

Model/Features

• Add support for Real-viz model.

• Add support fo Stable Diffusion 3.

• Add Pix2Pix support - @Mike | Xodeapp.

Pipeline/Improvements

• Allow multiple prompts support to T2I pipeline.

• Add the num_inference_steps to T2I endpoint - @Mike | Xodeapp.

• Made num_inference_steps configurable in ai-worker's I2I and I2V pipelines - @Jason | everestnode (bounty 🪙).

• Enabled configuration of num_inference_steps on the go-Livepeer side for I2I, I2V, and Upscale pipelines.

Bug Fixes

• Resolve T2I and I2I output truncation with non-empty seed and batch size > 1.

• Process T2I batch sequentially to avoid CUDA memory errors.

• Throw error for empty ai-runner response.

• Fix nil pointer runtime error in I2V.

• Fix huggingface login token not found error.

• Ensure I2I latency score takes number of images into account.

• Fix incorrect latencyScore for ByteDance/SDXL-Lightning model.

• Fix pipeline multipart writers.

Metrics

• Rename census Broadcaster metrics to Gateway.

• Add Orchestrator census metrics.

• Add Gateway census metrics.

Documentation

• Replace --broadcaster flag with --gateway in AI subnet documentation.

• Add num_inference_steps to T2I API reference documentation.

• Replace Broadcaster with Gateway in the livepeer.org docs.

• Reduce documentation showcase image sizes to improve loading times.

• Create documentation for the text-to-audio pipeline.

• Improve pipeline optimization documentation.

• Lots of small documentation fixes.

More improvements & fixes

• Create Ruby client SDK.

• Create Javascript client SDK.

• Create Typescript client SDK.

• Create python client SDK.

• Create Golang client SDK.

• Add AI apps and tools to Livepeer Awesome list.

• Create Livepeer AI dune Dashboard.

• Improve CI github actions to improve docker and binary releases.

In recent months, we have been diligently working to realize the vision set forth by the Livepeer AI Video SPE proposal. We are delighted to announce that the subnet is now stable, with 18 orchestrators completing approximately 14,000 inference jobs across three different inference pipelines on the network.

To avoid overwhelming you with individual updates, this first changelog includes a link to the AI SPE Retrospective forum post, where you can find detailed information about our achievements during this initial phase. As we move into the second phase of the SPE roadmap, we plan to release regular changelogs to this channel to keep the community updated on our progress 🗒️.

Best,

Brad | ad-astra-video,

John | eliteencoder,

Rick | transcode.eth

A heartfelt thanks to all the community contributors who collaborated with us in this first phase 💚. Special thanks to: @Papa Bear | Solar Farm, @Mike | Xodeapp, @Brad | ad-astra-video, @Ben | authority-null, @Marco | stronk.dev, @Speedybird | speedybird.xyz