Skip to main content

Why DSA and Big-O Still Matter

The Importance of DSA/Big-O​

The Importance of DSA/Big-O​

DS&A (Data Structures and Algorithms) and Big-O analysis are often reduced to "interview prep" topics — but that mindset undersells their real-world value. DS&A & Big-O are important not just because they can land you that high six-figure job, but because they are fundamental to the performance limits of our application.

The Business Value of Performance​

Performance isn’t just a nerd-flex. It costs money:

  • Serverless function pricing is often time-based. Shave 500ms off your code, and you might be shaving thousands of dollars off your cloud bill at scale.
  • Even on a local machine, performance translates to developer velocity — less waiting, more shipping.

Well, that’s just money, right? Who cares. It’s AWS’s money, or your employer’s budget — it’s abstract. No one’s losing sleep over a cloud bill. But what if it’s not just about money? What if it’s your own time you're wasting? What if the code is running in a hospital, or a car, or a rocket? What if performance delays a diagnosis — or a critical reaction?

Beyond Money: When Performance Impacts Reality​

Sometimes performance isn't just about cost — it's about outcomes. There are real-world domains where time and space complexity directly impact the final result:

  • Bioinformatics: Genome sequencing pipelines often involve aligning, sorting, or scanning through massive datasets. An inefficient sort or duplicate-removal step can delay discovery timelines or invalidate real-time analysis windows.
  • Safety-Critical Systems: Think aerospace fault detection or self-driving vehicle decision loops — every millisecond matters. A poorly chosen algorithm could make the difference between reacting in time or not.

These aren’t hypothetical. They're cases where better time or space complexity can directly reduce error, save money, or in some cases, save lives. You don’t always need the optimal solution. But you do need to understand the tradeoffs.

My History: When I Paid the Price of N²​

I was building a scraper for LinkedIn data, and at some point I had to merge/associate items from one part of a messy JSON blob with another — but the structure wasn’t clean. Think deeply nested .included[] arrays with polymorphic object types, like:

// Not the exact code, just something I found in an old repo to represent how messy some of these JSON reponses from LinkedIn were.
const companyInfoMain = await data.included
  .filter((i) => i["$type"] === "com.linkedin.voyager.organization.Company")
  .filter(
    (i) => i["*followingInfo"].split(":")[6].toString() === cID.toString()
  )
  .map((i) => {
    ...
  });

I had two lists of objects, and I needed to compare each item in one list against every item in another — not just for an ID match, but for a match buried somewhere inside nested fields. So I wrote it in the way that made sense to me — loop inside loop.

I knew it was O(n²) — but the JSON was ugly, and the logic was tricky. I needed to get it right first.

Then I tested it on 20,000 items. It took ~90 seconds (which made sense for 400M comparisons (20k * 20k))

Once I verified correctness, I went back and realized: I could build an intermediate structure (I think it was a hash map or a filtered index of just the relevant objects) — and instead of comparing every item to 20K others, I could just look up what I needed.

Refactor result: ~20K–400K comparisons (worst-case), runtime dropped to under 1 second.

I didn’t need to know Big-O to build it, but I absolutely needed Big-O to scale it.

It’s the best example I have of:

  • ✅ Optimize after you understand the data
  • ✅ Use brute force when prototyping
  • ✅ Know that algorithmic complexity becomes cost — in time, money, or energy

My Favorite Resources​

PK_ToDo: Affiliate links

PK_TODOs​

  • Add cost estimates for slow serverless execution (use AWS/GCP price lists)
  • Create local demo showing O(n²) vs O(n) merge pattern
  • Consider connecting this to reliability/SLOs (e.g., if this fails to finish, downstream jobs fail)
  • Show refactor stages (from brute-force to optimized)
  • Add affiliate links for DS&A books I actually like