OctoCam

Before we get anything started here, this is in fact a photo of the aurora that OctoCam took on the morning of May 11, 2024.

OctoCam saw the aurora in Worcester for nearly six and a half hours during the strong solar storms at a 1 second exposure time in night mode (more about this as you read), made very lucky given it had a NE view of the sky in its current setup.

It's honestly incredible that OctoCam captured a very rare solar storm of this magnitude. Like, damn. Anyway back to the article!

For reference, this is what the hardware looked like in November 2021 before the Pi Zero 2 W upgrade. The Pi & camera are meant to nicely tuck onto the mounting plate with 4 screws to hold the PCB.

But y’know, then the Camera Module 3 happened.

This is the OctoCam homepage! When you first visit the website, you’re greeted with the latest image that OctoCam captured, along with a navbar at the top so you can navigate all the various pages on OctoCam. The current page you’re on is highlighted.

With the June 2023 camera upgrades, OctoCam switched to capturing in 16:9 and at a higher resolution, so you’ll see some screenshots with the old sensor’s 4:3 aspect ratio.

You’ll also see in this image the power of night mode, which is a custom system in OctoCam to crank up the exposure at night to get more image detail. This gets talked about more later in the article.

Lastly - the upgrade from the OV5647 to the Camera Module 3 was MASSIVE. Image quality improved by leaps and bounds, especially when it came to nighttime captures (like this one).

The navbar mentioned just a moment ago is also mobile friendly! It’ll collapse all the links into a sidenav that can be triggered from the navbar.

Here’s a demo of the latest image automatically refreshing, doing so every 20 seconds (as that’s how often OctoCam uploads images). Excuse Squarespace absolutely destroying the image quality here, but hopefully you get the idea.

This auto-refresh system is quite a bit more complex than it appears! Depending on what time of day it is, OctoCam is either capturing in normal daytime mode where it takes 1 second to get images on the backend, or in a long-exposure mode at night (night mode) where it takes ~3-4 seconds to get images on the backend. There’s also a transition mode in-between night mode and normal mode where images take ~1.5 seconds to get on the backend.

The auto-refresh system uses weather data regularly fetched by the client (used for the current conditions box coming up in just a few pictures!), along with known intervals for when night mode starts and stops to figure out precisely when to make latest image requests!

Top that off with a super slick UI that stops auto-refresh when you’re tabbed out (to save bandwidth - each image is about ~400-600 KB), and you’ve got yourself quite a pleasing auto-refresh system from a UX standpoint.

(you don’t want to see the table describing the 5 iterations night mode has been through. it’s a lot.)

Right below the latest image box, there’s a handy box showing you the weather conditions present at OctoCam’s location. These conditions are fetched every 5 minutes on the frontend, then distributed down to clients to save on requests.

The weather box shows the current weather, severe weather alerts, precipitation rate, along with pretty standard weather data to confirm what’s happening on OctoCam. Personally, I like the temperature data and sunrise/sunset times a lot.

The super cool part shown here is the night mode on/off section! As mentioned earlier, OctoCam has a special night mode for night-time capture. There’s pre-defined intervals for when night mode turns off for daytime capture, then turns back on after sunset, and the weather box uses those intervals in junction with the day’s sunrise and sunset times to produce night mode on/off times.

Those manual intervals work great…until it’s really cloudy and OctoCam either exits night mode too early (or enters it too late), resulting in about 15-20 minutes of darkness on the daily timelapse. Using the power of AI (if statements…), the Pi-side automatically fetches the current conditions just around sunrise/sunset, and determines if it wants to get out of night mode late (for high cloud cover sunrises), or go into night mode early (for high cloud cover sunsets). This data is then reported to the frontend, which is relayed to the client, then you’ll see (early) or (late) next to the night mode on/off timings.

It’s an incredibly cool system that’s taken a lot of fine tuning, but it’s awesome!

As with everything on the client page, the weather automatically refreshes as well at the appropriate time. Since the weather data is only a dozen or so KB and doesn’t require a lot of special processing, the oh so slick UI is quickly hidden.

Below the weather box, you’ll find a quick blurb about OctoCam, then the meat and potatoes of OctoCam, the timelapses!

The timelapse library on the homepage shows the last 30 days of full-day timelapses, in addition to a partial day timelapse. Partial day timelapses are generated every 3 hours (at 3 AM, 6 AM, etc) and show the daily timelapse progress up to the point of rendering. It’s a cool way of seeing the day so far without needing to wait for the full-day timelapse to come out. The 3-hour interval was chosen as a nice compromise of semi-frequent refreshes without having to hammer the backend too often with render requests (and subsequently drive up my electric bill).

In the library, each timelapse shows a quick weather summary, and the duration/FPS of the timelapse.

The weather summary use to come straight from the Dark Sky API and was pretty good at summarizing what happened during the day. But now that Dark Sky has bit the dust, a weather summary is generated from WeatherKit using my internal WeatherKitProvider library and it gets…close enough to what Dark Sky would generate.

Similar to the image auto-refresh, the timelapse library also auto-refreshes! A quick disclaimer, this is a manually triggered refresh after I generated a timelapse for this demo. In short, the timelapse library auto-refreshes at pre-defined timings when a new timelapse should be available.

But just like before, implementing this wasn’t so simple! While timelapses do get generated at regular 3-hour (10800 second) intervals, there’s a couple key things to consider.

First - there’s an encoding/upload delay between a request getting made and the timelapse showing up here. Usually this delay maxes out at 4 minutes. But that’s not too hard to implement - just auto refresh at 12:04 AM, 3:04 AM, etc.

You then quickly realize that it’s a bit mathematically tricky to mathematically get the next one of these 3 hour intervals! Let me explain.

To get the next refresh time for current conditions (for example), what I do is take the current UNIX time, divide it by 300 (the refresh interval in seconds), then multiply it back by 300 to get a floored number, before adding 5 seconds to account for server delay. I do this flooring because of the delay. For example, if it’s 3:05:01 PM, this flooring gets me 3:05:00 PM + 5s = 3:05:05 PM so we can auto-refresh correctly at 3:05:05 PM. And if this floored + corrected time is below the current time (e.g. it’s 3:07:30 PM), then I add another 300 seconds to the next refresh time (to get 3:10:05 PM).

The problem is when you go over 3600 seconds, this flooring/multiplying starts to occur as it applies to UTC+0! For instance, dividing by 10800, then multiplying by 10800 for a 3 hour interval gets you the nearest 12 AM/3 AM/6 AM in UTC+0…but not in Eastern Time which is where myself (and the backend server) live.

To complicate things, Eastern Time shifts twice a year! Our UTC offset goes from 14,400 seconds to 18,800 seconds, then back to 14,400 seconds.

Ultimately, what you have to do is get the current UNIX time, then get the current UTC offset in America/New_York (which you CAN do in JavaScript, but it’s a hacky mess using toLocaleString and breaks twice a year but there’s no better way that doesn’t add luxon/moment), then offset the UNIX timestamp by the UTC offset (so now you’re working with the UTC timestamp like it’s in Eastern Time), do all the flooring and multiplying, then add back the offset to get the proper UNIX time, and then you add the delay. Easy peasy right?

And that’s a quick behind the scenes of some of the crazy auto-refresh systems in OctoCam. Good UX doesn’t come cheap!

Anyway, where were we…

Ah yes, the timelapse library. Clicking on a timelapse expands the timelapse showing the video element, and some weather details about the timelapse.

The timelapse thumbnail is blank because HTML5’s video tag does thumbnail rendering in the most uninnovative way ever: just fetch the whole video! Ask me how I know, this happened wayyyy back when the OctoCam web stack was being made and my client was pushing 200 Mbps of video data to prefetch videos for a thumbnail.

Anyway, below the video is the weather for the day of the timelapse. You get the condition summary, high and low temperature, and oodles of weather data because not-so-secretly I am a weather nerd. The night mode timings are also included, and you can see the early night mode feature in action here! A log of when OctoCam does this early/late night mode stuff is logged for the library.

Below that is the download button. No magic there, it just downloads.

Now you might be wondering, where do timelapses go that are older than 30 days? The answer is the archive!

Early on in OctoCam’s life, I was pretty cautious about the privacy implications for OctoCam, and had a very strict purging policy. But a month or two in, I realized that no two days OctoCam captures is unique! And it’s cool to have a day you remember and then watch the OctoCam timelapse for that day (like the day I missed class registration by 4 hours on my birthday only to be woken up by thunderstorms, this is a true story).

As such, I stopped deleting daily timelapses starting in September 2021 (with backfill to March 2021 as I found saved timelapses in my downloads folder on my laptop).

On the homepage of the archive, you’ll see the current count of videos in the archive (as of March 2023 it’s up ~960 timelapses), a cool on this day one year ago feature (more about this later), and then the actual archive library with pagination.

A super cool feature in the archive is the on this day one year ago box which shows the timelapse OctoCam captured from a year ago (if available). Given how OctoCam is now in my apartment 24/7, this feature should hopefully improve loads as 2023 rolls on.

Of course implementing this feature wasn’t super easy due to our good ol friend time. Let me explain.

For a bit of context, OctoCam timelapse videos literally embed information into the filename itself, in the format of <UNIX timestamp of the end of the timelapse>_<seconds the timelapse covers>.mp4. For instance, a full day timelapse covering November 17, 2022 is named 1668747600_86400.mp4.

My first idea with this feature was, oh cool, let me just do the whole flooring and multiplying thing at 86400 seconds, subtract 31536000 seconds (365 days), and then if a timelapse with that timestamp and _86400.mp4 exist, we’ve got ourselves a winner!

And then reality hit. Reality always hits. Not only do we have to contend with daylight savings time, leap year is also a thing.

Thankfully, Python makes it relatively easy with third-party libraries to find the UNIX timestamp for midnight of a certain day (this is how full-day timelapse generation avoids issues with DST), and figure out the UNIX timestamp that happened a year ago that factors in leap years and time changes.

Hopefully this feature won’t break on February 29, 2024…

And below that is the actual meat and potatoes, the archived timelapses.

Fun fact, pagination had to be implemented in September 2022 due to the page just getting too big to render! At that point, there were about 400 timelapses in the archive, and while it took ages for the page to server-side render, it did, you know, get them all in one go.

That is until WebKit (oh WebKit…) ended that party and my iPhone started refusing to render the archive page, so I had to add pagination.

Anyways, the archive library shows all the same detail as the normal timelapse library, with weather conditions for every timelapse. When clicking on a timelapse, the UI is exactly the same as on the frontend with detailed weather conditions and a download button.

The archive use to go deep. So deep that you could uncover videos of OctoCam on the 80 degree FoV, and even a timelapse from MotionEye OS.

But since OctoCam has been running for 2+ years now, these videos started to eat away at my Web VPS’ storage, only compounded by the June 2023 camera upgrades massively increasing video size.

As such, I developed a solution called long-term storage. Basically, videos older than a year are stored on my apartment server where there’s a lot more storage. I figured out a way to reverse proxy the videos out of my apartment’s network and over to the end user (at the expense of awful loading times due to…cable upload speeds being cable upload speeds), but you can in fact access every single timelapse OctoCam has ever made.

Which is honestly quite an impressive feat.

Moving on, let’s take a look at the OctoCam status page! This page was mostly made for me to keep an eye on OctoCam’s operations, but there’s some cool things going on here.

First and foremost is the OctoCam image delay metrics. As mentioned earlier, depending on the time of day images will come down to the backend with a differing amount of delay.

The image delay graph was actually born out of a time when my university network was acting up for a few days, producing persistent packet drops, which then made OctoCam have the tendancy to successfully upload images about 1/2 of the time. Of course I was wondering what on earth was going on and made a graph to further understand things.

(these packet drops also borked an entire timelapse because the request to get sunrise/sunset data for night mode intervals failed! and OctoCam stayed in night mode all day long…protections are now in place to avoid this.)

Nowadays, it’s a chill graph and I can monitor for any network anomalies that may occur. There’s also some 5/15/60 minute delay metrics inspired by load average metrics on Linux.

This box also auto-refreshes like the many other boxes on the website. No advanced system here, just refreshing every 60 seconds with about 15 seconds of delay to account for night mode.

Below that, you’ll find the image library status summary, which does exactly what it says. It summarizes the status of the images OctoCam has been capturing over the last 30 days!

And yes, OctoCam really does capture that many images per month! It astounds me how reliable this stack is to regularly achieve over 99.9% uptime (with the remaining 0.1% usually happening as a result of hardware maintenance).

Below that, you’ll find the details of OctoCam’s missing images in the last 30 days. Even though OctoCam is very reliable (and usually has image uptime in excess of 99.9% almost all the time), sometimes there are hiccups that aren’t a result of hardware maintenance.

This is where this section comes in handy, as it shows when images weren’t captured for each day. At a high-level overview, you can see each day, how many images were captured versus expected, and a daily uptime percent. Status icons indicate an overall status for the day (green = 100%, yellow >= 99%, red < 99%)

Now you may think this is easy to calculate, because OctoCam images so handily have the UNIX timestamp they were captured at as their filename. Just see if they exist, right? Sort of.

See, Python isn’t exactly the fastest language (even Python 3.11 won’t help here), and looping through 130,000 image names 30 times to calculate each day status isn’t fast. This is what I did for a while, and it led to the status page taking 10-15 seconds to load. Instead, multithreading is now used so that each day’s status is calculated by its own worker, and that worker checks if every one of 4320 expected images exist (or not). As a result, the status page loads much faster.

For a day when an outage occurs, clicking on it will immediately give you details to the statement “Bono, my images are gone!”.

The details show exactly when an outage starts, gets resolved, and how long (in terms of the timelapse) each outage lasted for.

To try and automatically resolve any camera-related issues resulting in image downtime, OctoCam has built-in systems to detect when these issues happen and automatically issue reboots.

In short, OctoCam from time to time experiences MMAL errors, which occurs when the camera is being used by another process and can’t capture an image, or the camera connection wasn’t closed properly.

Since the OctoCam script gets started every time an image is captured (to the tune of ~130,000 times a month), this does happen occasionally, meaning newer capture scripts can’t use the camera connection.

If the capture script detects one of these issues, it automatically sends an issue to my GitLab server, which then pings my private service monitoring Slack. This image shows an example of the auto-generated issue.

If the capture script detects that three of these issues occurred in the last 60 seconds (usually persistent MMAL issues that result in long periods of image capture fails manifest this way), then OctoCam sends another issue report to GitLab saying it’s gonna reboot, reboots, then comes back online to hopefully no more camera errors. There’s bootloop protection in place to make sure OctoCam can’t immediately bootloop after starting up.

Additionally, the backend server is always checking to ensure that new images are being uploaded. If the backend goes 5 minutes without getting a new image - another issue is posted. This is usually to catch hardware issues that require intervention and avoid huge swaths of downtime. These alerts are pretty rare - most of the time it’s due to the Pi’s Wi-Fi connection being a bit finicky (and having to reboot it & the AP), or at least once, the Pi going down so hard that MMAL issues were only getting reported en masse after many hours of the system being stuck.

In short, these notifications are to keep me aware of OctoCam’s operational status, and to intervene when necessary. It’s also what maintains the 99.9+% uptime OctoCam (usually) has, barring any larger issues that require extensive downtime.

The next page up is the changelog page, which has the big changelog of all the things changed in OctoCam. As you can see, there’s been quite a lot of changes.

And last but not least, the admin panel. Now, the admin panel got a lot more use in ye olden days when a lot of OctoCam’s content wasn’t public. As I’ve been making OctoCam’s content public, the admin panel doesn’t get used a lot.

However, there are two use cases (plus some cool technical talk about how timelapses are generated) that the admin panel has.

If you’ve made it this far in the article, take a moment to get some water!

The two functions mentioned earlier are manually generating timelapses, and to manage timelapses on the website.

The saved tab is a relic from when OctoCam was first introduced, and at this point only includes timelapses I manually generate for myself. The tab highlighted is the public page tab, and this is where I can manage timelapses on the public page. This is useful if one of the automatic timelapses had something wrong with it, and I needed to delete it before manually generating a new one.

For each timelapse, there’s buttons to download and delete the timelapse - delete being the key function here. As mentioned before, sometimes the automatic timelapses don’t come out right and I need to delete them.

You’ll notice that weather data is not included on the admin panel. It’s mostly because it’s not really needed, and rendering that data would only increase page load times.

Clicking on that generate timelapse button throws you into this cool modal and with that, it’s time to bore you about OctoCam’s timelapse making abilities!

OctoCam has three distinct timelapse generation modes - timeframe from now, time ranged, and stepped. We’ll talk about all three of them in subsequent images.

Timeframe from now is the simpliest mode - you plug in a number of hours to go out from now and OctoCam generates a timelapse doing exactly that.

For each timelapse generated, you can tweak the FPS, resolution (1600x1200 is the resolution images are captured at), and the video bitrate (defaults to 6 Mbps). Defaults are pre-filled, and usually I only change the FPS setting for faster/slower timelapses.

At the bottom of each timelapse generation form is a box you can tick to generate the timelapse to the public page. It comes in handy for when I need to manually generate timelapses for the public page if an automatic process broke down, or for showing something cool on the public page.

The next and more frequently used mode is time range. In this mode, you plug in a start time and end time (UNIX timestamps on the backend), and make a timelapse using images from this range.

This is how daily timelapses are made, except it’s an automatic call using the midnight timestamps for the current day and the previous day (to account for daylight savings time as mentioned earlier).

The same FPS/resolution/bitrate settings are all present here.

As an exclusive to the time range format, there’s an extra checkbox called “Mark as a public partial timelapse”.

In short - partial timelapses are normal timelapses with partial_ added to the start of the filename, allowing them to get specially marked in the timelapse library with a special icon.

Most importantly, when partial timelapses are auto-generated every 3 hours, any timelapse with the partial_ prefix in the filename gets deleted so the next partial timelapse can replace it.

The final timelapse mode is stepped timelapse. While OctoCam timelapses are a good length for one day (2 minutes and 24 seconds to be exact), scale that up to a longer time frame and you’ve just found yourself a feature film…

(it’s no joke, but the OctoCam archive has a link to a 8 hour, 21 minute video of basically 8 months of timelapses from my sophomore year dorm!)

To solve this problem, the stepped timelapse mode was made to get really long form timelapses capturing weeks or months while maintaining reasonable video lengths. It achieves this by skipping over a large majority of images in defined configurable steps, be it 1 minute, 10 minutes, 60 minutes, etc.

This mode is used at the start of every month, when a stepped timelapse with 10 minute intervals is made to cover the last month in 2 minutes and 29 seconds. It’s super cool! Special formatting is also applied in the library to show a different icon, and metadata about the step length.

You made it to the end of the article! Your reward is this detailed flowchart of how the OctoCam stack works (minus the long-term video archive storage)

The OctoCam stack was originally made almost three years ago - since then I’ve built a lot of features on top of it and a full refactor has been largely out of question since I simply don’t have the time to do it. As such, you sorta get this mess of a flowchart here.

OctoCam is a three-server setup due to my Web VPS not having the power nor storage to properly encode video. As such, I had to offload those duties to a server in my apartment. The only catch? My apartment internet is asymmetrical with slow upload speeds - not great for serving videos to clients (but I am forced to deal with it for the long-term archive so, you win some and lose some).

Thankfully, the videos OctoCam encodes are relatively small to the point that my web server can store them. Given my web server has plenty of upstream bandwidth, this best-of-both-worlds combination led to how the stack came together.

—

With that, that’s OctoCam! It has been a project many years in the making and I hope it can continue to stay that way. This project has gone above and beyond all my wildest imaginations. Building a project that can operate with extremely little manual intervention (and just work!) has been an absolute joy.

At this point, you may want to check out the OctoCam (v0) project in experiments to see a super early version of the OctoCam stack!