According to git history I started work on "Activity Title" in August 2024. It's advertised on my Projects page. The user I'm targeting is me. This is a hobby project that I made available to the public and it has maybe 20 users.

It's been hosted on Fly.io and, for over a year, it's been a low-maintenance project that "just worked". The monthly cost was below $5 so it was free to me.

Earlier this week I tried logging in to activitytitle.com and was greeted with this error on Firefox

Secure Connection Failed

An error occurred during a connection to activitytitle.com. PR_END_OF_FILE_ERROR

Error code: PR_END_OF_FILE_ERROR

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

The domain was fine so the security angle was a red-herring. The fly domain, https://strava-deck.fly.dev/, gave the same error. Something is busted!

The fly machine was down so I tried redeploying it.

> fly deploy
> ....
> fly logs

[PR04] could not find a good candidate within 40 attempts at load balancing

I checked the fly.toml and Dockerfile -- but nothing had changed lately.

After a night of debugging I identified that the issue is the volume.

[mounts]
  source = "deck_data"
  destination = "/data"

If I comment this out the machine runs and the website returns a different error (progress!). In the logs I see Django errors because there's no database, which is expected (the database is a file in the volume).

Just when I was about the wipe it all and recreate the volume I found this support post from an old-school Google search: https://community.fly.io/t/machine-suspended-and-it-wont-start-again-not-another-post-about-cold-start/24984/4

The trick is forking the volume, creating a copy, and deleting the old volume. The new fork volume attaches to a machine like magic and we're running just like before with all data restored.

Conclusion

What did I learn from this ordeal? Fly.io is a little janky. And my website is janky too. But, hey, they're free so don't complain.