Clock Skew

Overview

A few folks have identified that their Tidbyt occasionally shows the incorrect time - anywhere from a few seconds to 30 minutes in extreme cases. A clock that tells the wrong time is useless, which makes this class of bug really painful to both the Tidbyt team and our users. This post breaks down the issue and describes mitigations we’re putting in place to resolve this class of bug.

:camera: A photo of my desk, where I keep a Casio F-91W next to my Tidbyt as a last resort monitor.

The Issue :broken_heart:

When we first started shipping Tidbyts, we used a time synchronization system that many computers use today to ensure an accurate time. As more users received their Tidbyts, we started to see numerous customer reports about Tidbyts being unable to connect.

Upon investigation, we discovered many Internet Service Providers and home networking equipment blocked or interfered with the time synchronization system we had in place. As a fix, we rolled out a less accurate time keeping system to ensure every customer who purchased a Tidbyt could connect their Tidbyt to our cloud.

The Fix :wrench:

giphy

:camera: via GIPHY

We’re actively rolling out changes to our platform that will ensure Tidbyts are accurate within a few seconds while maintaining our improved connectivity. If you’re experiencing significant time skew, please feel free to reach out to [email protected] and we’ll be more than happy to investigate the issues.

Nerd Zone :nerd_face:

For our tech savvy users, we wanted to provide a bit more detail. When we first started shipping devices, time synchronization via NTP was a required step to connect to our servers. We had a ton of issues with NTP across the board. This article provides a pretty good overview for the types of issues we were experiencing, with the most relatable being the graph showing NTP success rate for Comcast :weary:.

We switched to HTTP Date, which allowed for a much higher success rate for connections while still providing an accurate time. The issue is, after a successful connection, we relied on the real-time clock of the Tidbyt which we later discovered can skew by several minutes if left unchecked.

Our solution moving forward will be to use NTP with an HTTP Date fallback, and continuously keep the time in sync via either mechanism.

Great writeup! Question: as someone with a custom app that currently keeps the time within ~5 seconds of accurate by brute force (AKA continuous API pushes from my own server), is this method something app developers will be able to leverage somehow? Cheers.

@CyberMonk The initial fix to use an alternate, better time synchronization will automatically improve all clock apps, including those developed by the community.

We’re exploring other features for further improvements — for example, allowing apps to report when they are “stale” and need to re-rendered. This is still a work in progress, but anything we roll out for our own apps does get rolled out to the broader community as well (once we’re done testing it).