The addition of a Frame Interpolation pipeline would provide users with a good way to increase the smoothness of images or interpolate between two images. A good model that can be used to do this is [FILM](https://github.com/google-research/frame-interpolation) by google. This pipeline was already implemented in [Yondon's Cog branch](https://github.com/livepeer/ai-worker/blob/old-cog/jobs/containers/svd-xt-film/pipelines/film.py) and can easily be added.

It appears that the old JSON string format Yondon was using is no longer functioning. The parameter description needs to be updated to indicate that it now also accepts JSON files. Additionally, it would be beneficial to modify the code to support both formats, as providing escaped JSON strings is useful for development. [https://github.com/livepeer/go-livepeer/blob/87c24a2b37c79700dacd2c9480ffe39707414132/cmd/livepeer/livepeer.go#L155](https://github.com/livepeer/go-livepeer/blob/87c24a2b37c79700dacd2c9480ffe39707414132/cmd/livepeer/livepeer.go#L155). Logic that was implemented by Yondon is found \[here\]([https://github.com/livepeer/go-livepeer/blob/bac48c5e321c01fd936300d53ed9d9d5642869e3/core/ai.go#L72](https://github.com/livepeer/go-livepeer/blob/bac48c5e321c01fd936300d53ed9d9d5642869e3/core/ai.go#L72)). It was implemented in [https://github.com/livepeer/go-livepeer/commit/5e73dcbad4d5d69a87d2714aa7eb7eab208973be](https://github.com/livepeer/go-livepeer/commit/5e73dcbad4d5d69a87d2714aa7eb7eab208973be). ### What does happen Users have reported a `nil` error is thrown in [https://discordapp.com/channels/@me/1240228650582806588/1266341662548951040](https://discordapp.com/channels/@me/1240228650582806588/1266341662548951040) when the json string format is used. Reason the default pricing is not used when using the Json string format.

During today's watercooler chat, Titan raised a question about the Orchestrator's capability to select the GPU for loading models. Specifically, if they add a 4090 and a 3090 to their machine, they would like to load the SVD1.1 model on the 4090 and the ByteDance/SDXL-Lightning on the 3090. This might be worth implementing before we start the Subnet alpha tests.

We have observed an issue where the pipeline encounters problems that aren't always effectively communicated to the user. One of these issues occurs when a model is not pre-downloaded on the orchestrator. In such cases, the runner attempts to download the model, which can take a long time. The orchestrator is not informed that this is happening, and if the download takes longer than 5 minutes, the runner will timeout. **Steps to Reproduce:** 1. Clone and build the `go-livepeer@ai-video` branch from [GitHub](https://github.com/livepeer/go-livepeer/tree/ai-video) (detailed instructions available [here](https://www.notion.so/livepeer/go-livepeer-ai-video-local-development-3c56ea0abd5e4c8394abc4c00de8d094?pvs=4)). 2. Add the specified `aiModels.json` configuration to your `~/.lpData` directory: ```json [ { "pipeline": "image-to-video", "model_id": "stabilityai/stable-video-diffusion-img2vid-xt-1-1", "price_per_unit": 3390842 } ] ``` 3. Neglect to download the `stabilityai/stable-video-diffusion-img2vid-xt-1-1` model into the `~/.lpData/models` folder. 4. Launch a local broadcaster and orchestrator. 5. Request an image-to-video job. **Result:** The container crashes, leaving the user uninformed about the progress only to receive a `service unavailable error` after a very long timeout. Additionally the orchestrator has to check the container logs to find out why crashes the container which is also not ideal. **Reason:** The failure is due to the inability to access the token-gated model `stabilityai/stable-video-diffusion-img2vid-xt-1-1` from the URL `https://huggingface.co/api/models/stabilityai/stable-video-diffusion-img2vid-xt-1-1`. This model is gated and requires authentication for access, which isn't provided in this scenario. This error occurs when the Orchestrator forgot the download the given model but said it was available. The critical error message states: ``` Cannot load model stabilityai/stable-video-diffusion-img2vid-xt-1-1: model is not cached locally and an error occurred while trying to fetch metadata from the Hub. Please check out the root cause in the stacktrace above. ``` **Proposed Solution:** To enhance user experience and ensure clear communication, it's essential to implement more robust error handling mechanisms. Specifically, when the pipeline encounters such an error, it should catch the exception and provide a user-friendly error message explaining the issue and suggesting potential fixes, such as verifying model availability or ensuring proper authentication credentials are provided.

There are conditions where the ai-runner returns a solid black image. Some logs have been seen in the ai-runner docker image, but no examples available to post at the moment. Working on reproduction steps.

Currently, to load new AI models, users are required to restart their node, which can be inconvenient. We are developing a CLI tool to allow model switching without a restart. However, relying solely on a CLI may not offer the most user-friendly experience. It would be more intuitive and efficient if orchestrators had access to a local UI that allows them to manage and control their operations seamlessly through a web interface.

What if there is an error and then the container is stopped by the worker? Currently if it is during setup Os will not know an error occurred but will have to wait for the container timeout to occur. # Solution We should implement some kind of communication protocol to signal to the O that an error occurred in the AI runner container.