Snow Day API (2019-)

This project is describing the backend API for the Snow Day Predictor that I run. If you want to instead see project details about the frontend for the predictor, please click here.

APIs are a really cool thing. I’ve used them in plenty of my projects, and despite the documentation nightmares and having to keep up with an API you don’t have any control over, they’re really, really useful in the modern world.

And then I built my own API. A bit of backstory.

 

After getting some feedback that I should push out SMS predictions for my Snow Day Predictor, I had to figure out how to, well, push out those SMS messages. After doing some research, I figured out that I’d basically have to build my own API to handle key functions of the SMS service:

  • Signing up via. the website
  • Dealing with inbound messages
  • Sending out message blasts

So, off I went, with the power of Python, Flask, and Twilio – a framework and service I had never used before.

Building the API was, well, tricky. I had to learn all about adding & authenticating API keys, handling errors, this weird thing known as form data, and how to get Twilio to call the API when a new message came in (and respond back to the person if need be). But, after about 7 hours of straight coding, I eventually made an API in one night that handled all of this functionality.

Since that first night, I’ve continued to make lots of improvements to make the API more professional, per say. When a new user signs up, there’s actually a quite extensive error catching system so a user actually knows why registration failed (registration confirmation), having a system to catch when a new message blast failed to deliver to a number (and unsubscribe the user after 3 errors), and enough error catching code to probably blow your socks off.

 

One interesting API feature that started from the get-go was a not-so-organized, awful system of internal error codes. While HTTP has it’s own error codes (404, etc) that do a good job, for better specificity, I decided to add some internal error codes. In the first few versions I was completely happy with how the system worked, but as the API grew, the error code system got extremely disorganized, since there isn’t 100+ error codes across the API. Additionally, for the first few versions of the API, a 200 OK HTTP code was always being sent back. I learned that this is bad, so now relevant HTTP codes are sent back.

In coming versions of the API, I’ll be making a Status Codes “v2” system, which will have a lot more logical organization. But hey, let’s look on the bright side, an English message is always sent back!

 

The biggest milestone with this entire API was adding the logic & methods to handle a basic (but fully-fledged) dashboard that relied on the Snow Day API for the backend. 80% of the logic code is for session management (starting new sessions, ending new sessions, making sure sessions aren’t expired), but there are methods to validate one-time passwords (sent via SMS), update user settings, along with fetching user settings.

(If you want more information about how the Dashboard works with the API, visit the Snow Day Dashboard page).

 

For this project, I did need a database to store all the session information & numbers to be subscribed, and for that I decided to use DynamoDB. For my use case, it costs nothing, I don’t manage it, and it’s not SQL. Perfectly happy with it.

 

Another thing I wanted to accomplish with this project was to follow with the semantic versioning scheme as much as possible. Previous projects of mine had followed semver (Semantic Versioning) to a lesser extent, but this was going to be my first project where I really had to stick with it, and boy did I do that. The API is still running on v1 because the entire API is backwards compatible to this day. A pretty good accomplishment, I’d say.

 

Nowadays, a large part of APIs is making sure that your response times are as low as possible. That’s why I’m happy to say that the Snow Day API has about a 1000ms response time! Of course, this isn’t terribly great, most APIs will respond back within 200ms. But, I decided to cut myself some slack here, this is my first API after all. In reality, there are a lot of spots where I could dramatically improve the API’s performance (something about not having a bajillion library imports and the API being one file). But, I digress. Maybe in a future version I’ll work on optimization.

 

One last thing that was a bit of a pain to deal with when developing the Snow Day API was API documentation. Even though this is a single-use project, I wanted to have detailed documentation of every API method & error code for when I’ll eventually release the source code. In the beginning, documentation was done in Markdown, which while it did get the job done, it wasn’t the prettiest thing ever. But, as the API grew, so did the Markdown docs file…so I had to find a solution.

After doing some digging in car rides, I settled on making an OpenAPI spec file using APICurio Studio. It’s actually quite a nice platform for documenting APIs, albeit with a few minor quirks. Documenting all (now) 17 methods took about 10-15 hours, but was pretty seamless. While most APIs use Swagger to display documentation, I decided on using Redoc instead, as it looked more professional.

On the other hand, the API spec when encoded in JSON is bigger than the actual API source code. If you needed one more example why Python is a damn efficient language, well, here you go.

 

 

To say the least, I’ve had (and I’m still having) a lot of fun developing the Snow Day API. Will I become a full time API developer? Absolutely not. Did I learn a lot from developing an API? Absolutely.

Of course, this being my first API, there are a lot of flaws with how I designed and built this API. This includes:

  • Testing everything in production because it’s a pain in the ass to change the URL twilio calls for inbound messages. I could absolutely get a second number and configure that to use a dev server, but you know, I’m lazy.
  • No CSRF (Cross-Origin Request Forgery) for the dashboard form data. Since the dashboard runs on a totally different domain and with different software (plain Apache), really the only method for adding this protection isn’t available. I plan to eventually move the dashboard over to a subdomain running on Flask, which will not only add the CSRF protection, but also eliminate the need for many API methods.
  • API keys are for every method (but I do plan to add better permission management down the line)
  • DynamoDB calls are scan and not query (an easy fix down the line, though)
  • Twilio keys & AWS keys are hardcoded in
  • As talked about earlier, the API is one large file (instead of separate files for each method), which makes things super inefficient
  • Having an option to close down the API (I quickly learned that no, you cannot close an API)
  • Every method is prefixed with /api/…instead of /api/v1/. Whoops!

Despite having a good amount of flaws, the rest of the API is actually built pretty well. Some highlights include:

  • There’s loads of error catching (and even more to come), so it’s nearly impossible to get an Internal Server Error (500) that doesn’t have a proper response attached
  • The API returns as little data as possible to the end-user for API security.
  • Session management is optimized to also return as little data as possible. When a registered number starts a new session, the API generates a random, meaningless session ID and returns it to the user. To get any data, the API has to be called, and can only retrieve data tied to that number. It’s a really smart system that works better than 90% of APIs on the market. Security!
  • Every method has a pretty sensible name (the register number method is not /api/heyGetMeSignedUpForTheSMSServiceThankYou/, it’s /api/registerNumber)
  • Status code & API information is encoded into the headers for every request
  • Responses are standard for every API method (except inboundSMS where TwiML-compatible XML is required, don’t ask)

 

And that’s the story of how I built my first API. Do I think I’ll ever be an API developer in the long-run? Absolutely not. But, making this API helped me gain a new understanding of a different Python development type, and how different it is to make & debug an API versus a standalone program.

On the bright side, though, now I can make fully-integrated systems! It’s an extremely satisfying thing to see two systems work in harmony (front-end and back-end), and having the skills to make basic APIs opens up tons of opportunities for future projects.

 

Now I finally know how to “run” Python code in a browser.

(didn’t get the joke? I hear a project about QA may give you a hint)