The CRUD Crucible: Becoming a Software Engineer

“So how did you transition into this and get to the level that you’re at? I’ve been doing this for 23 years and I’m not sure what advice to give people getting into this other than ‘Make stuff for a really long time.'” That was the question I got from a fellow engineer I’m working on a new project with.

And my answer is: “I tried and failed to figure out how make a small Software as a Service application several times over the course of a couple years.” There’s a lot hidden in that sentence, so I’m going to unpack that here.

The High Level Goal

So here’s the basics. To do business on the internet, you need a strategy to solve a few problems:

  • Get people to sign up for something in a secure way
  • Create a piece of software that makes some part of their life easier
  • Integrate it with a payment mechanism, preferably a subscription mechanism

Accomplishing these goals implies some common technical challenges. Not only is it hard to get past most of these challenges, but the best solutions to the problems change with time.

The Parts

There’s 3 technical parts to this problem:

  • The server
  • The frontend interface
  • The payment integration

The first of these is what I find the most interesting, and it is, in my opinion, what makes me a software engineer.

The other parts are important, too. I’ll cover them–just with not quite as much enthusiasm.

Servers and Where They Live

So what’s a server? A server is a program on a computer that listens for requests sent to it and sends back answers. On the internet, everything you send is a request, and everything that you get back is an answer from a server. These answers are also called server responses.

A server program can be anywhere, but it’s most useful when that computer is in a publicly accessible “neighborhood” of computers with server programs running on them. Making a group of computers publicly accessible means giving it a static IP address.

A static IP address is a number associated with a network. It looks like or something like that. Not super easy to remember, which is why we hide them behind domain names.

A question you might now ask is, “Can I set up a publicly accessible server on my computer?” And the answer is yes, but the static IP address part, and the fact that you turn your computer off sometimes makes it impractical.

Having a static IP address for any network means having a special relationship with your Internet Service Provider and paying fees to reserve that IP address, because there’s a limited number of them.

So that’s where a server lives and that’s what it takes to get to it over the internet.

There’s another detail to cover. On an given computer, there’s often many programs and servers running, so if I send a request, how do I know I’m reaching the right program? And the answer is ports. If a static IP address points to a neighborhood, then a port is a house number for a program, NOT a computer. The computer is a like a “block” of houses in the neighborhood, but the houses are like the servers. The most common ports on the internet are 80 and 443.

Now on to the actual server part.

The Server Part

So we have this program called a server. How does this help and why do we care?

Well the thing that generates a lot of money on the internet is creating, reading, updating, and deleting data records. We often use a convenient acronym (CRUD) to describe this series of actions.

On its own, just creating and modifying data isn’t very valuable. But creating useful relationships between data records, and kicking off automatic chains of events when people do things with that data, IS very valuable.

That’s pretty abstract. Let’s use a concrete example.

Let’s say you owned a sports bar, and you have different types of customers on different nights. One night you have people into baseball, the next day football, and sometimes there’s boxing events. Let’s say that there’s also 4 other bars in your area. If left to randomness, and you don’t have a communication strategy with your customers, then on any given night, if 100 people go out in your area, you’ll get 20 of them. (There’s 5 bars and you’re just 1 random choice out of those 5).

But let’s say you decided that you were going to collect data about who these people were, what they were into, and then had a way to proactively make use of this information.

These days, people aren’t too guarded about their cell phone numbers, so SMS marketing is getting more common, and getting people to give you their phone numbers is getting easier.

So let’s say that you had an iPad on your bar where people could enter phone numbers based on the vague promise of future deals or offers. You might get some data. Now you have a way to contact these people. Unfortunately you don’t know who they are, and your offer will be kind of watered down. Also you don’t currently have any way of contacting these people other that texting them one by one or mass texting them.

In which case, they might just gang up on you in a group text and troll you for marketing to all of them at once and exposing their phone numbers to everyone in the group. Not cool.

So what could a server do for us here?

Well first of all, our server could talk to a database where we store those phone numbers. Second, we could add a bit of context. Let’s say we don’t know who the people entering phone numbers are, but we do know they entered their phone number on baseball night, football night, or boxing night depending on when they were there. So we might program that schedule into our server and now we have records that tie phone numbers to specific sports.

The next thing we might want is an integration with a service that automatically sends out text messages to sports specific customers. In this case we might write a program that knows when the next baseball game or boxing match is, and sends out a text message with a specific offer to those customers the day of a game.

All of the sudden, you’ve differentiated yourself from the other bars in your area, and chances are you’ll do a lot better than random at corralling those 100 people into your bar on Tuesday.

So that’s the value. We want to write a program that collects data, adds some context, and then automatically performs a useful action.

The frontend interface

So there’s another piece of this puzzle. We actually need to put boxes, buttons, and pictures on a screen that people can interact with. This is simultaneously a design problem, a technical problem, and a human problem.

Humans weren’t meant to interact with squares on a screen. It’s weird that we do it at all, but it ends up being so useful to us, that we deal with it. And engineers and designers invest lots of time and energy into make this into as intuitive and painless of a process as possible for everyone. Albeit with mixed results.

As the internet has evolved, how this part is executed has changed a lot. People expect a lot from applications. A little bit of animation, tasteful design, and thoughtless workflow.

The workhorse of this space are the modern JavaScript frameworks. React is the 800lb gorilla, and then the other notable choices are Angular and Vue. These are ways to write components, style them, and connect them together to make a useful webpage or application.

Often, when a user sends a request to a server, like a browser call to, an HTML page with one of these applications on it is sent as a response. The front end parts: pages, images, scripts, browser apps are all sent as responses from servers. Apps are not servers.

So a webpage, or the app on it aren’t the server and have nothing to do with it, they’re just a response. I find this to be a common point of confusion.

The payment part

Lastly, once you have a useful thing and a way to interact with it, you need to make money and pay for it.

Luckily Stripe and a host of payment processors have abstracted this problem nicely. It’s just up to you to integrate it into your system.

Payment is what creates the need for enforceable security mechanisms and privacy.

Figuring out how to conduct payments and then use payment status to make decisions is the core reason for learning how to implement security mechanisms like API Keys, OAuth2, and JSON Web Tokens.

API Keys

You’re going to need these to connect external services securely. For instance, how do you know you’re accessing your own data and not someone else’s inside of Stripe? And how does Stripe know that it’s talking to your server that suddenly appeared out of nowhere? The answer is API keys. Long strings of letters and numbers that are closely guarded that both parties have knowledge of. They get injected into the headers of requests and provide a secure connection.

JSON Web Tokens

One of the most common modern security mechanisms on the internet. It’s a bit of data about a user, a service, or a session, combined together into an object, and then cryptographically signed with a signature that lives in the memory of a server somewhere. The idea that no one knows the secret used to “sign” or scramble the data is the only guarantee that the person sending the request is who they say they are. These keys are also used to build in expiration mechanisms, which can be pretty important.


No one likes to keep track of passwords. That’s why single sign on is wildly popular. Also, how do I know some service isn’t doing weird things with my information? And what guarantees do I have that they’re storing that data securely?

OAuth and related strategies are mechanisms to authenticate users into a service, like the server you create, by involving a third party who we deem as an identity provider — often Google or Facebook.

Wrap Up

So the reason for this article was to talk about the parts of the problem that I believe is important to solve on the way to becoming a seasoned software engineer who builds distributed systems.

The particulars of implementing these things are more straightforward. They are difficult, and have tons of details, and there’s a near infinite number of ways to make a webpage, an interface, or a server. For that reason, it’s important to keep an eye on the overall theme of the skill you’re trying to cultivate: a general strategy and tools for putting buy buttons on the internet for things that provide a useful data service/automation.

Next time, I’ll focus on particular languages and frameworks. I might even do a video for this article. We’ll see.

As a quick preview, I’ll be talking about Node.js, Vue.js, Kubernetes, Stripe, WordPress, and WooCommerce. In general, these are the suite of tools I use to solve problems.

Let's get started

Just let us know the basics and we'll send you a Slack invite to discuss with the whole team.

What are we going to be talking about?

%d bloggers like this: