Snow Day API (2019-)

This project is describing the backend API for the Snow Day Predictor that I run. If you want to instead see project details about the frontend for the predictor, please click here.

APIs are a really cool thing. You fetch a URL, you get something back, or something happens on a server somewhere. Despite documentation nightmares and having to keep up with an API that could change in an instant, they’re still really useful to loads of developers.

Before the Snow Day API, I had always been a consumer of APIs. Until, that is, I decided to make my own API. A bit of backstory.

In January 2019, I decided to ask about 3 people whether I should push out SMS predictions for my Snow Day Predictor (I know, sample size is very representative). Out of the 3 responses, 3 of them said yes, so know I had to figure out how to do push SMS predictions. After some light research and realizing what I had got myself into, I came to the conclusion that I would have to build my own API to run this SMS Service.

For this brand new API, I’d have to design it to do three key tasks:

  • Handle signing up new users for the service
  • Send out message blasts
  • Deal with inbound messages from users (to confirm registration, unsubscribe, etc)

So, off I went, with the power of Python, Flask, and Twilio. A framework and a service I had never used before.

Building the API at first was, to put it mildly, very tricky. I had to learn about authenticating API keys, reading this weird thing called form data, how to make Twilio call the API when a new message came in then format the response in their specific markup language (don’t ask), register users with confirmation, and handle potential errors. It’s just a small learning curve…right?

7 hours later, the Snow Day API was born! I couldn’t believe my excitement, I had made my own API!

After that first magical night happened, I continued to make the API more like an actual API. First up on the list was lots of error catching – and oh boy did I add lots of that. In the code to register a number to the service, there’s about 15 different responses the API can generate depending on what happens. This is sort of important – as this status was being directly reported to an end user when they were signing up, so it’s important to let them know that the SMS Service doesn’t work on a landline. Or that they blocked the number. Why exactly did I implement this? Not sure.

Something that I’ve questioned implementing – automatically kicking people off of the SMS Service when too many delivery blasts failed to deliver. The way it works is simple – for every message sent in a blast (in my case), Twilio will call a URL with updates on the message delivery status, and keep calling it as the message traverses the carrier network (for instance, the status goes from sent to delivered).

This sounded like a great idea when I implemented it – Twilio still charges for messages that fail to deliver. And, technically, this should work if somebody’s number starts returning repeated delivery errors. The problem is that this rarely happens in real life, and this system ends up catching erroneous events on the carrier side if anything. It’s never actually automatically unsubscribed a user, and that’s a good thing, because this is something that you should not do! Punishing users for something out of their control is never, ever, ever a good idea.

 

Another API feature that started from the get-go was a system of internal status codes to replace the HTTP error codes. I thought to myself that HTTP error codes don’t have enough specificity (which of course they don’t, they’re designed that way), so I would replace it with 2-3 digit status codes instead. At first, the status codes had some organization, usually the first digit representing a category of errors. Of course, as the API grew and grew, the system got more and more disorganized, and I also realized that always sending back an HTTP 200 with every response is a bad idea, so now relevant HTTP codes are sent back.

 

 

The biggest milestone with this entire API was adding the logic & methods to handle a basic (but fully-fledged) dashboard that relied on the Snow Day API for the backend. 80% of the logic code is for session management (starting new sessions, ending new sessions, making sure sessions aren’t expired), but there are methods to validate one-time passwords (sent via SMS), update user settings, along with fetching user settings.

(If you want more information about how the Dashboard works with the API, visit the Snow Day Dashboard page).

 

For this project, I did need a database to store all the session information & numbers to be subscribed, and for that I decided to use DynamoDB. For my use case, it costs nothing, I don’t manage it, and it’s not SQL. Perfectly happy with it.

 

Another thing I wanted to accomplish with this project was to follow with the semantic versioning scheme as much as possible. Previous projects of mine had followed semver (Semantic Versioning) to a lesser extent, but this was going to be my first project where I really had to stick with it, and boy did I do that. The API is still running on v1 because the entire API is backwards compatible to this day. A pretty good accomplishment, I’d say.

 

Nowadays, a large part of APIs is making sure that your response times are as low as possible. That’s why I’m happy to say that the Snow Day API has about a 1000ms response time! Of course, this isn’t terribly great, most APIs will respond back within 200ms. But, I decided to cut myself some slack here, this is my first API after all. In reality, there are a lot of spots where I could dramatically improve the API’s performance (something about not having a bajillion library imports and the API being one file). But, I digress. Maybe in a future version I’ll work on optimization.

 

One last thing that was a bit of a pain to deal with when developing the Snow Day API was API documentation. Even though this is a single-use project, I wanted to have detailed documentation of every API method & error code for when I’ll eventually release the source code. In the beginning, documentation was done in Markdown, which while it did get the job done, it wasn’t the prettiest thing ever. But, as the API grew, so did the Markdown docs file…so I had to find a solution.

After doing some digging in car rides, I settled on making an OpenAPI spec file using APICurio Studio. It’s actually quite a nice platform for documenting APIs, albeit with a few minor quirks. Documenting all (now) 17 methods took about 10-15 hours, but was pretty seamless. While most APIs use Swagger to display documentation, I decided on using Redoc instead, as it looked more professional.

On the other hand, the API spec when encoded in JSON is bigger than the actual API source code. If you needed one more example why Python is a damn efficient language, well, here you go.

 

 

To say the least, I’ve had (and I’m still having) a lot of fun developing the Snow Day API. Will I become a full time API developer? Absolutely not. Did I learn a lot from developing an API? Absolutely.

Of course, this being my first API, there are a lot of flaws with how I designed and built this API. This includes:

  • Testing everything in production because it’s a pain in the ass to change the URL twilio calls for inbound messages. I could absolutely get a second number and configure that to use a dev server, but you know, I’m lazy.
  • No CSRF (Cross-Origin Request Forgery) for the dashboard form data. Since the dashboard runs on a totally different domain and with different software (plain Apache), really the only method for adding this protection isn’t available. I plan to eventually move the dashboard over to a subdomain running on Flask, which will not only add the CSRF protection, but also eliminate the need for many API methods.
  • API keys are for every method (but I do plan to add better permission management down the line)
  • DynamoDB calls are scan and not query (an easy fix down the line, though)
  • Twilio keys & AWS keys are hardcoded in
  • As talked about earlier, the API is one large file (instead of separate files for each method), which makes things super inefficient
  • Having an option to close down the API (I quickly learned that no, you cannot close an API)
  • Every method is prefixed with /api/…instead of /api/v1/. Whoops!

Despite having a good amount of flaws, the rest of the API is actually built pretty well. Some highlights include:

  • There’s loads of error catching (and even more to come), so it’s nearly impossible to get an Internal Server Error (500) that doesn’t have a proper response attached
  • The API returns as little data as possible to the end-user for API security.
  • Session management is optimized to also return as little data as possible. When a registered number starts a new session, the API generates a random, meaningless session ID and returns it to the user. To get any data, the API has to be called, and can only retrieve data tied to that number. It’s a really smart system that works better than 90% of APIs on the market. Security!
  • Every method has a pretty sensible name (the register number method is not /api/heyGetMeSignedUpForTheSMSServiceThankYou/, it’s /api/registerNumber)
  • Status code & API information is encoded into the headers for every request
  • Responses are standard for every API method (except inboundSMS where TwiML-compatible XML is required, don’t ask)

 

And that’s the story of how I built my first API. Do I think I’ll ever be an API developer in the long-run? Absolutely not. But, making this API helped me gain a new understanding of a different Python development type, and how different it is to make & debug an API versus a standalone program.

On the bright side, though, now I can make fully-integrated systems! It’s an extremely satisfying thing to see two systems work in harmony (front-end and back-end), and having the skills to make basic APIs opens up tons of opportunities for future projects.

 

Now I finally know how to “run” Python code in a browser.

(didn’t get the joke? I hear a project about QA may give you a hint)