Posted on ::

Ten years ago, I thought I was smart as f**k.

I could talk about database indexing strategies, distributed systems, scaling patterns. I'd read the papers, watched all the talks, followed the right people on Twitter. I was ready to architect the shit out of any system you threw at me. In theory.

People didn't take me seriously and I was so confused. Like, didn't they hear me explain CQRS at lunch? Did they miss my Slack message about event sourcing? I had opinions on everything. Strong ones.

Now I get it. I was knowledge-rich and experience-poor. And in software engineering, that distinction matters more than we admit.

The tech industry has a bad habit of conflating knowledge with experience. We treat someone who's read all the Kubernetes docs the same as someone who's maintained production K8s clusters at scale. We interview for knowledge (algorithm puzzles, system design on whiteboards) but we should hire for experience (has this person actually done the thing?).

Neither juniors nor seniors are honest about this difference. Juniors overestimate what knowledge alone can do. Seniors forget how much of their capability comes from PTSD.

What they are in my humble opinion

Knowledge is what you learn from books, talks, documentation, courses. Understanding how a hash map works. Knowing the theory of database indexing. Being able to explain CAP theorem at a dinner party.

Experience is actually building the thing. Maintaining APIs at 10,000 requests per second. Debugging a production incident at 3am when your DB index with tens of billions of documents is thrashing. Shipping code and living with the consequences.

Here's the insight: knowledge is useless until applied.

Learning for learning's sake is a trap. I'm not against understanding theory, but theory without application is just a trivia trick. Kinda like a dog that knows how to play dead but won't come to you when you call it off-leash. You can watch 100 YouTube videos about distributed systems. You can memorize every CAP theorem tradeoff. But until you've debugged a split-brain scenario in production, you don't really know distributed systems.

The one-way street

Experience creates knowledge. Sometimes explicitly (you learn a specific technique), sometimes through suffering (you learn what NOT to do after a 3 day outage).

But knowledge does NOT create experience. And this is where people get confused.

If you have to build a super scalable database and you can pick between:

  • Someone who knows a ton of database theory, has read all the papers, can explain B-trees and LSM-trees in detail, but has never built a database
  • Someone who has built several databases already

I'm picking the latter every single time. Not even close.

Why? Because the person with experience has knowledge PLUS all the things that can't be taught:

  • The edge cases that only show up at scale (never in your laptop, always in prod, always on Friday at 5pm)
  • The tradeoffs that only matter in production (turns out "eventual consistency" means "eventually someone will be pissed")
  • The unknown unknowns that will reveal themselves to you in the most spectacular fashion 7 months into the project that should've been delivered 6 months ago because you scoped it like shit thinking "how hard could it be" and now the database is on fire the API is returning 503s your manager is asking for a post-mortem you haven't even written the mortem yet because the patient is still dying and you're seriously considering just deleting your Slack and moving to a cabin in the woods my brother you are NOT recovering from this one

The experienced person might not remember every academic paper, but they know what actually breaks in the real world. That's worth infinitely more.

How humans actually learn

Here's the thing: there's quite a bit of evidence that humans learn by doing, by practicing, by writing things in their own words. Understanding is more powerful than just memorizing. And doing is the best way to deeply understand a topic.

This isn't just about retention. It's about developing intuition that can't be taught from a book.

Let me give you a concrete example from my recent experience. I've been getting into low-level programming lately. I watched probably 10 YouTube videos about how I/O multiplexing (event loops) works. I understood the concepts: epoll, kqueue, file descriptors, non-blocking I/O. I could explain it to someone.

It was a complete waste of time compared to actually building an in-memory key-value store in Zig and Rust with 100% manual event loops using Linux and macOS syscalls.

After watching the videos, I had surface-level familiarity that would've been forgotten in a week. After building it, I have deep understanding of:

  • Why event loops work this way
  • What the actual tradeoffs are
  • Where things break down
  • How to debug when things go wrong
  • The feel of working with these systems

The videos gave me knowledge. Building gave me experience, which created deeper knowledge. One way street.

This pattern holds everywhere. You can read about how to write concurrent code, or you can debug a race condition for 6 hours and never make that mistake again. You can study memory management, or you can chase down a memory leak in production. The latter teaches you in a way the former never will.

What experience actually teaches you

There's a concept called "unknown unknowns" - you don't know what you don't know. Book knowledge can't teach you this. Only experience reveals it. It is like legendary pokemons, they are not in the pokedex

When I started taking on genuinely big, complicated projects at massive scale at work, everything changed. Building APIs that handle tens of thousands of requests per second. Managing database indices with tens of billions of documents. These experiences taught me things that no documentation, no course, no amount of studying could have prepared me for.

And here's the thing nobody tells you: books teach you clean examples with perfect abstractions and well-named variables. Production is a 7-year-old Django monolith with a God class called Utils.py that's 3000 lines long, database migrations that were written by someone who quit in 2019 and left no documentation, a caching layer that sometimes works and sometimes just returns null for reasons nobody understands, and a comment that says "TODO: fix this properly" from 2021. You will work with this. You will ship features in this. The books did not prepare you for this.

Practical takeaways

So what do you actually do with this? Here's my advice:

For juniors (and I wish someone had told me this):

Say yes to everything, especially the shitty projects. Doesn't matter how boring or unglamorous. You will learn. Even if the project is a disaster, you'll learn what NOT to do, which is just as valuable.

Stop over-indexing on reading and courses. Build things. Ship things. Break things in production (in a non-catastrophic way, ideally). The suffering is part of the process.

Learn with purpose. Use knowledge as instrumentation toward a goal. That goal doesn't have to be concrete like "build an app" - it can be softer like "be a better engineer."

Want to understand low-level memory management? Don't watch 20 videos. Build a toy project in C or Zig. Struggle with it. Debug segfaults for hours. You'll be a better engineer in your high-level abstracted day job because you'll understand what's happening underneath.

For everyone:

Be patient. There is no cheat code for experience. You can't speedrun it. You can't fake it (for more than a few interviews). Time and struggle are part of the equation.

Be humble about what you haven't done. Reading about scaling Kubernetes pods is not the same as maintaining production systems at 10,000 RPS. I'm saying this from experience - I used to confuse these things.

You need the right environment. You can't get experience with high-scale systems if you don't work somewhere with those challenges. This isn't your fault, but be mindful of it. Don't claim experience you don't have.

Be open to taking weird, complicated projects at your day job. The messy ones nobody wants. The legacy systems. The fire drills. These are where you learn the most.

My reflection

I think about my younger self often. Why didn't people take me seriously even though I felt so knowledgeable?

Because I was knowledge-rich and experience-poor. I could talk about systems I'd never built. I could explain patterns I'd never used in production. I had consumed a lot of information but hadn't earned any scars. It was shallow knowledge.

Ten years later, I don't consider myself as knowledgeable as I falsely thought I was at the beginning of my career. But I'm infinitely more capable. The difference is experience.

I've built things at scale. I've been paged just to find myself looking at the screen as if I was looking at abstract art. I've made mistakes that cost me years of life. I've debugged problems that don't exist in textbooks. I've shipped code (and by code I mean bugs) that millions of people used.

That's not knowledge. That's experience. And you can't get it any other way.


Knowledge is valuable. But experience is invaluable. Stop reading this and go build things, you twat.

Table of Contents