Strategies for Self-Hosted Apps?

[CyberMonk]

Hi folks,

I’ve finally had some time to start experimenting with Tidbyt app development, and while it’s a lot of fun, I’m a little disappointed with the story for local/personal app hosting (i.e., there doesn’t seem to be one currently). I thought I’d solicit any advice/suggestions around how folks are handling serving their own apps to their personal Tidbyts.

Context: I’ve written a custom app that combines a clock, calendar, weather (using the met.no API), and some other custom status indications from Home Assistant into a single view (see attached image, with HA info removed). Due to the specific-to-me nature of the app, this isn’t something I’m interested in publishing to the Community, but I’d still like it to serve as my primary (and at this point, only) Tidbyt app for my personal device.

If I just wanted to re-render the app every few seconds and push, I’ve already got a solution for this working on my local machine, but I’d prefer to leverage the pixel serve command in order to cache my API calls with the included cache helper–I don’t want to spam the weather API. However, from what I can tell, curl calls issued to a running pixlet server don’t trigger a new .webp view generation as the server “view” in browser is actually a little Javascript app (which curl cannot execute).

Is there no way around this other than building out a separate service to handle rate-limiting my calls to the met.no weather API, and pulling in that data via a local HTTP call? Is there something I’m missing around forcing a local pixlet server to re-generate the .webp view for device push? Any and all help would be appreciated. Thanks!

[Tavis_Gustafson]

I don’t think that’s what the serve function is for at all. If you want a new webp then why not just call pixlet render ?

Are you sure the cache functionality is actually working with your local pixlet install ? I was under the assumption that the cache module only worked when the .star file if run within the pixlet server environment.

[CyberMonk]

To clarify: I realize this is at least not a documented use of the serve function, but I also saw some (older) references to triggering re-renders while it was running rather than using the pixlet render call. Specifically, I was hoping that there was some way to leverage the included cache helpers in a locally hosted app (as I assume the Tidbyt devs are leveraging this when hosting Community apps, given they require reasonable caching). Based on the lack of replies here and some follow-up in the Discord, it sounds like this isn’t currently a thing.

(Also, I may not have been clear, but I never had caching working in a locally served app that I was actually pushing to my Tidbyt—only within the pixlet serve browser-based environment.)

I’ll follow up later when I’ve put something together, but my plan is to build a simple Node.js app that will receive incoming requests from my .star app when rendered and handle the caching itself–either grabbing new data from any APIs I call, or serving up cached data if we’re under whatever rate limit I establish. I’ll then plan to script the render and push of my app to my Tidbyt on some smaller interval (which is simple enough with a bash scripted while loop, and something I already have a proof of concept working for).

[CyberMonk]

If anyone’s interested, here’s a very quick and dirty (and not at all robust) Node.js script I threw together tonight to serve up rate-limited API data every time it’s called (I’m using the OpenWeatherMap API). You’ll have to fill in some of the gaps around the referenced Sequelize includes where I define models and whatnot, but you could just as easily cache data in-memory and eschew the MySQL store stuff I’m using.

const axios = require('axios')
const express = require('express')
const app = express()
const port = 4036

//set up DB models
const {
	Weather,
	Sequelize,
	sequelize
} = require('./sequelize');

const Op = Sequelize.Op;

app.get('/', async (req, res) => {
	try {
		const existingWeather = await Weather.findOne({
			where: {
				createdAt: {
					[Op.gte]: Sequelize.literal("NOW() - INTERVAL 12 MINUTE")
				}
			},
			raw: true
		})

		//no matching record found
		if (existingWeather === null) {
			await Weather.destroy({
				truncate: true
			})

			axios
				.get('https://api.openweathermap.org/data/2.5/weather?lat={lat}&lon={lon}&units=imperial&appid={API_KEY}')
				.then(apiRes => {
					(async () => {
						const newWeather = await Weather.create({ weather_response: apiRes.data })
						res.send(apiRes.data)
					})()
				})
		} else {
			res.send(existingWeather.weather_response);
		}

	} catch (error) {
		console.log(error)
	}
})

app.listen(port, () => {
	console.log(`Weather server listening on port ${port}`)	
})

I’m running this Node app, and then using a bash script to call pixlet render and then pixlet push for my Pixlet app every few seconds. The Pixlet .star app script is what handles calling the Node app and parsing the received weather data.

[CyberMonk]

A little update: my old bash script to push updates was extremely simple and unaware of any changes it was pushing, plus I believe it was causing me some timeout issues with the API.

I’ve written a new node script that will re-render your pixlet .star file, grab the image as a base64-encoded string, and then issue a push to the API to update your device. I’m also comparing each new render with the previous one, and if nothing has changed, it won’t bother making a new push to the API. Feel free to borrow this if you’re hosting your own app! (Note you’ll need to run this with a modern version of node, and the pixlet binary needs to exist in the same directory you run this script from.)

Important note: I have this running every 2 seconds because the endpoint my pixlet script uses to grab new data is also self-hosted and implements some rate-limiting so as not to overload any services I’m calling. If your pixlet script is calling a third-party API, make sure to adjust the interval so that you don’t get yourself rate-limited with any external services.

/*
* App to render pixlet, grab base64 encoded string of .webp, and push to Tidbyt API
*/

import axios from 'axios'
import fs from 'fs'
import * as child from 'child_process'

const apiToken = 'your-api-token-here'
const deviceId = 'your-device-id-here'
const installationID = 'your-installation-id-here'

const axiosConfig = {
	headers: { Authorization: `Bearer ${apiToken}` }
}

let previousHash = '';

const pushInterval = setInterval(() => {
	let renderPixlet = child.spawn('./pixlet', ['render', 'your-app.star'])
	let base64webp = fs.readFileSync('./your-app.webp', 'base64')

	if (base64webp !== previousHash) {
		previousHash = base64webp
		
		axios
			.post(
				'https://api.tidbyt.com/v0/devices/'+deviceId+'/push',
				{
					"image": base64webp,
					"installationID": installationID,
					"background": true
				},
				axiosConfig
			)
			.then((response) => {
				/* No need to do anything */
			})
			.catch((error) => {
				console.log(error)
			})
	} else {
		/*console.log('nochange')*/
	}
}, 2000)

[CyberMonk]

P.S. Here’s an updated screenshot of the app as it currently stands. Time, date, weather conditions, outdoor and indoor temps (respectively), and color-coded air quality.

[bbotv2]

Neat implementation! I am just starting to review my options for internal temp sensors as well. I have 8 sensors in the home and want a quick visual of temp velocity by region of house.

[dannielum]

Nice work there CyberMonk! I am actually planning to work on something similar. I see you are doing it with more like a push approach instead of pull approach. Is there any reason why you chose to do it this way? Normally I would think the client device should pull the data from the API. I am trying to learn if there is a limitation with Tidbyt that makes you chose this approach? Thanks.

[CyberMonk]

The Tidbyt device is pretty “dumb”, by which I mean all it does it show an animated image (GIF or WEBP format). So any updates or information you want to display on the Tidbyt you first need to put together on another machine, then generate the animated image using that information, and finally push it to the Tidbyt. (Official and community Tidbyt apps have a server component hosted by the Tidbyt folks that coordinates pushes.)

My push script above is actually just using the pixlet app provided by the Tidbyt folks, and my script there is what calls my bespoke API to pull together the data I use for my view. And because I’m pulling in data that is, in part, behind rate-limited APIs, I have a flow that looks like this:

• Provide self-hosted API endpoint that can be called on demand but implements rate limiting for relevant services, that returns all necessary data for Tidbyt view
• Call aforementioned API in Tidbyt .star file, which is rendered by pixlet
• Routinely re-render .star file and see if output has changed; when it has, push an update to the Tidbyt via the official API

Hopefully that answers your question!

[dannielum]

Thanks for your explanation. So Tidbyt could only display static contents compiled from the star files. If I understand correctly, in order to display dynamic contents, we need a server to consistently regenerate the webp files and push to the Tidbyt devices. If I publish my Tidbyt app to the community, I don’t need to worry about setting up a self hosted server to consistently regenerating the webp file right? This way I could still do it with the pull approach right?

[CyberMonk]

As long as you don’t really think of the Tidbyt as ever “pulling” content, I think you’re right. But, truly, it never does. The Tidbyt receives an image (possibly animated), it displays that content, and it awaits a future push to start displaying a new image. That’s the extent of the logic, as far as I understand it, on the device. (If I’m wrong, please chime in Tidbyt folks!)

If your concern is where to host your own app, making it a community app would theoretically handle the hosting side of things for you, but keep in mind that there’s a process and a timeline for publishing community apps, and there are other caveats there as well. E.g., the app I’ve featured in this thread I will never publish because it just wouldn’t make sense–not everyone has the same sensor layout I do to support this design. If your own app is similar, you still may want or need to host it on your own and pursue a frequent push strategy like I am.

[dannielum]

Thanks CyberMonk. I am having hard time understanding how the community app gets updated content from the Tidbyt hosted server if there is no pull model. Do you mean the server is consistently doing the pull from the external API for new content and then regenerate the webp and push to Tidbyt device? If not, does it mean we still need some service to consistently ping the Tidbyt server to update the app content?

[yourfavorite]

https://tidbyt.dev/docs/build/build-for-tidbyt#push-to-a-tidbyt

Check here. You push an image to the device via api key.

[CyberMonk]

Every time you use pixlet to “render” an image, it’s actually running a .star script, which is written in Starlark (kind of a limited form of Python) that actually contains the logic to produce that image. This includes any API calls, so every time the image is re-rendered, any API calls will be re-issued (though the Tidbyt folks also offer some caching functionality for community apps). For a community hosted app, apps get re-rendered periodically and pushed to devices that are subscribed to them, or something along those lines.

You should explore the online documentation as it discusses a lot of these topics: Tidbyt | Dev

[dannielum]

Thanks again for your explanation. I will read around and play with it. I wish Tidbyt client app could be more dynamic.

[humanapp]

@CyberMonk thank you for posting your solution. I had a similar use case where I wanted to make an app that updated frequently, but probably not of much use to the broader community. I was able to use your technique with success! I did hit one wrinkle where my Tidbyt API key wasn’t working in API calls. I worked around it by reverse engineering the calls pixlet makes and was able to mimic them (good times!), but it’s the same as yours otherwise. I wonder if anyone else is having trouble with the API key, or if I’m just doing something wrong.

Anyway, here’s my project, in case it can help others: GitHub - humanapp/ferry-tidbyt: Display WSDOT ferry status in relation to the Kingston (WA) terminal on a Tidbyt device.

It’s a Tidbyt displaying Washington State Ferry status in relation to the Kingston (WA) terminal in real time. It doesn’t look very nice yet, that’s coming next