Category: Software

AI Code Binge: The Next 50k Lines

Posted on by Eric F. Savage

I’ve continued to work on the project I discussed in the last post, and it’s now well into the “real project” range. At the moment it’s about 150k lines of code, after 574 commits, 298 PRs, 300 issues. Many of those last two have come recently as I’ve adapted my workflow, which is what I’ll discuss in this post.

I took a few days off to work on other things, and when I came back to the project I took stock and didn’t really feel like the simple chat->code->test loop was cutting it as the system got more complicated and capable. At 150+ pages the design and architecture docs were too unwieldy for both me and Claude to reason about with scattershot ideas and feedback. In a real team this would be where you want to spread the work out so people aren’t stepping on each other’s digital toes, but this is a different shape of that problem. I could fire off random bugs and ideas and have it build things, but you either do that in series, which is slow, or in parallel, which causes lots of merge conflicts and duplicated efforts.

Themed Versions

Side Note: Experienced devs and managers will start notice a theme here, that we already know how to manage this type of stuff, it’s the same things we’ve used to manage complex projects for decades. It’s just not totally obvious up front or even in the middle how we can apply that, and where things can be or should be different.

The first thing I did was to tag what I had as 0.1.0. Then I brain dumped everything I had queued up from big ideas to small ideas, and with Claude started to cluster these into themes. Then we mapped those themes to versions in a roadmap, with practical criteria that would define the goal of each version.

Then we drilled down into the next version, addressing specific details, refining the scope, etc. Ideas kept coming from all over, but if they don’t fit in this bucket, they simply get parked on the roadmap. With a tight scope, Claude breaks it down into tasks, which it can do very well. These tasks are pretty well specified, with background, acceptance/test criteria, related tasks, and medium level implementation details (e.g. table names, but not file names).

Review Time

Things are getting tricky enough now that I don’t want to do the “commit then review” approach, I want to try reviews prior to merging. I start handing the issues off to agents and having them send PRs. I also have them review each other’s work, and while they’re mostly writing good if not excellent code, it is finding enough things that I definitely validate that the reviews are worthwhile.

So now I’ve got some agents writing, some reviewing, some addressing feedback, and others trying to merge good PRs. I didn’t expect this to work, but I had to see where it was going to fail, which it quickly did. There were endless merge conflicts, agents deleting stacked branches, conflicting details and formatting. Gemini struggled with large conflicts, it would try to fix them, and would eventually succeed, but it took forever. Codex would realize what it just stepped into and often take a more surgical approach where it would directly re-apply the changes to a clean main. Claude did OK, but much slower than Codex. Claude tends to repeat obvious mistakes so I had to add some guardrails to it’s MEMORY.md, which I haven’t yet had to do with the others.

Oops: Formatting and Linting

I quickly realized I had never set up any strict linting and formatting. I implemented this, which invalidated a dozen or so pending PRs that were in merge hell, so I just closed those out. I did a full formatting/linting pass with pretty much every option turned on, then re-did those PRs. This would be utterly demoralizing for human coders, but the bots don’t care and it all took a couple of hours to get back on track.

More Merging > Less Merging?

The formatting helped with the conflicts but the agents were still butting heads frequently. I slogged through wrapping up that version, which took a couple of nights, and then took a new approach for the next one. This time I had the issues more strictly organized using GitHub subissues. I also had started to realize that Codex was consistently doing everything way faster than the others, and at similar or possibly even higher quality for the smaller scoped tasks. Also, Codex on the $20 plan seems to get more coding work done than the Claude $125 plan, which I frequently exhaust even using mostly Sonnet for coding tasks.

I told Codex to send me a PR for all open issues, of which there were about 48. It cranked for a while and finished the job. Then I had Claude review all of the PRs, leaving feedback as comments. Then Codex addressed all of the comments. For a human coder, this would be kind of a bonkers approach, but it worked well. Finally, I had Codex merge the PRs in batches, in whatever order it deemed appropriate. After each merge it would run the quicker tests, and after each batch it ran the full tests: unit, integration, E2E, which had already passed on push so they weren’t going to be far off. Halfway through I pulled and built the app myself and tested the new stuff, then it finished. Overall this approach handled more work in a lot less time. I don’t know how big this scales, but 48 issues is a pretty good sized chunk of work in terms of planning and effort on my part so I’m not sure I need to go too far beyond that.

Phases

I tried that version for two versions and it worked well but I made a tweak on the most recent one. This version lent itself well to phases so instead of a giant batch of 40 issues it became about 6 batches of 5-8 issues/PRs. The throughput is a bit lower, but this handled drift between phases better, so feedback on the second issue that affects the 30th issue doesn’t cause headaches because it’s already merged. I think the optimal batch size can vary here depending on the focus of the version, but it feels like the best approach I’ve tried so far.

Syncing Up

I forgot to do this with a couple of versions, but figured the design had probably drifted a bit from the implementation since some of the decisions were only the roadmap or the issues. I asked Claude chat to check a few key docs to confirm this, which it did. Then I asked Claude Code (Opus) to review everything in detail, it fired up a bunch of subagents, and … immediately ate my entire 5 hour Max quota in about 20 minutes of sifting through code. It eventually finished in the next window, and did a great job, but boy is it a monster for tokens for that type of work. I tried it again on the next version and it didn’t consume the whole quota so I think it’s something to do either in chunks, or frequently.

Random Bugs, No Backlog

If I come across an actual bug while I’m using it or testing it, I’ll describe it to my planning Claude session and it will file the bug on my behalf, unless it’s a symptom of a larger change, in which case it goes on the roadmap. I do not have a backlog of issues, if anything is an issue, it gets picked up and fixed. This is an anti-pattern for human developers, but it’s a lot easier to just tell the agents to fix all issues, rather than deal with tags and milestones and versions. When we plan the next version, these changes get incorporated there.

Overall this feels like a pretty sustainable and productive method. It’s not as exciting or tiring as the initial burst, but it is very elastic. I can spend 10 minutes and kick off a chunk of work, or I can spend 3 hours and keep things moving while designing the upcoming work. This project is far from done so we’ll see how we do with a progessively larger and more complex environment.

AI Code Binge, February 2026 Style

Posted on by Eric F. Savage

I recently took a week off work to recharge and ended up going on a bit of a binge planning and building out a new system. It gave me a chance to explore some non-Gemini/non-Google stuff with more energy than I’m normally able to in my spare time, and I figured I’d share some thoughts:

Models

No real surprises here, but Opus 4.6 is really fantastic at planning and reviewing. Sonnet 4.6 doesn’t seem to produce any worse code than Opus, but it does make more mistakes when it comes to decisions. Codex 5.3 is by far the fastest, and also the most focused and direct. Gemini is faster than Claude, tends to take a more meandering/thorough route than Codex. Opus feels like the best partner of the batch in terms of design, but they all have useful aspects. Where I’ve settled is to iterate in Opus and periodically run it by ChatGPT and Gemini for feedback, which has been fruitful. I’ve done most of the early coding work with Claude because Claude Code is just a little ahead of the other CLI tools, but the models are all good enough for pretty much anything.

Workflow

This was a greenfield project and is now about 100k lines so it went through a few phases pretty quickly over ~30 hours. I spent a lot of time in a chat session just planning it out before building anything, so I started the build with a 60+ page design doc and a similarly sized architecture doc that I’d iterated on over probably ~10 hours. Claude came up with a pretty good phased approach, so I had it go through this for a few steps. The first couple of phases I kept a tight leash but after a while I went to YOLO, once I’d had enough patterns established.

As I iterated, I would use the Claude chat, which was now managing the docs in GitHub in a branch. This made it much easier to review via PRs that resulted from the decisions to make sure there weren’t any side effects or lossiness. The chat will create GitHub issues based on changes. Then I go to claude code/codex/gemini and tell it to fix a specific issue or just fix them all. Claude takes 10-20 minutes to handle most things, up to 40 for bigger batches or bigger changes. Sometimes it does them in parallel, sometimes not, I don’t think it’s really dialed in yet on where to split things up, but it errs on the side of serial so it almost never conflicts with itself.

Code

I don’t review the code closely but I do read it and it all looks really good. There aren’t many examples of the issues we’ve come to expect from these things. No significant cases of overcommenting, creating multiple version of the same thing, naively structured files/classes. I think this is a combination of:

  1. The models getting better
  2. Starting from scratch, no legacy decisions to consider or tech debt or “this is how we used to do it”.
  3. Having a thorough (though not formal in any sense) design and architecture spec with derived artifacts like roadmaps. Major changes are tracked in ADRs so it’s only tried to undo that once.

Context window and compacting are challenges at this point for design, less so for coding, as the fairly rigorous design approach yields tighter iteration loops, scopes, and smaller blast radii for changes.

Biology

I’m not tooting my own horn here, as this is much more “this is what these things can do if you let them”, but what I’ve built in a week, both in terms of capabilities and polish and raw metrics (200+ pages of design/docs/tutorials, 100k lines of code, 1k+ tests, dozens of E2E tests) is way beyond 10x. I’m a fairly prolific coder when possible and a good big picture thinker but in line with this has been exhilarating and exhausting in a novel way. It’s less like a creative Flow state where time slips away and more like a good video game. “Just one more feature” feels alot like “just one more quest”. I don’t think I could keep this up indefinitely, or it would at least take a while to adapt. A typical session looks like this:

  1. Run through the app, trying previous/new things, typing notes into the design chat.
  2. Iterate a bit there, it updates docs, creates issues.
  3. Have the agent work on the issues.
  4. Repeat, doing step 1 while the previous iteration of step 3 is happening.

The step change is that this is a ~30 minute cycle, not a 2-3 week sprint, and these can be pretty significant or deep changes. It’s literally building things faster than you can design and try them (not even including the self improvement loop). And it’s doing them well, this isn’t a simple project and it’s not making garbage code. It’s novel because it’s more productive than Flow but also less comfortable. I’ve only been spending like 3-4 hours a day on it and my brain and dopamine circuits still haven’t really figured out to react to it yet, so you end up in a contradictory state of doing smart things with your lizard brain. That said, it’s been really fun and I recommend trying it if you can!

Building at the speed of … builds

Posted on by Eric F. Savage

I’ve been thinking about build speed lately, usually while waiting for builds, and I think the thing that’s underappreciated isn’t the raw numbers, it’s that different speeds are qualitatively different experiences. Faster is always better, but it’s far from a linear relationship.

Working on a package that builds in 100ms is basically invisible. You don’t even notice it’s happening. The feedback loop is so tight that it feels like the code is just doing what you told it to do. You’re in conversation with the machine and you are the bottleneck, which is the goal.

At 10 seconds, it’s disruptive, but if the tooling is set up well you can stay in flow. You wait. You’re still there when it finishes. You might even find a bit of rhythm or cadence here and get a little thrill from the anticipation like hitting a long fly ball and seeing if it makes it out.

At a minute, it’s more like someone tapping you on the shoulder to ask a question. Your attention wobbles. You notice you could use a coffee, or you tab over to email to check something “real quick.” Five minutes later you come back and the build failed two minutes ago. Now you’re reloading context.

At 10 minutes, it changes your whole relationship with the work. You start actively avoiding triggering builds. You’re trying to see how far you can get while holding your breath. If it fails at 9:30 you’re genuinely frustrated, and maybe you’ll just go find something else to do for a while.

The reason I think this matters is that people tend to look at build optimization as a spreadsheet exercise: spend 8 hours to save 30 seconds, amortize across however many builds, calculate break-even. Even if the math works out it feels tedious and while the other coders might thank you for a 5% reduction the suits won’t.

I think that exercise misses the point entirely. The less quantifiable stuff pays back almost immediately. You’re more focused. You’re doing better work. You’re just happier. A developer who’s been trained by their feedback loop to flinch isn’t going to produce the same work as one who can iterate freely.

But AI

There’s an argument to me made that AI changes this calculus, that it doesn’t matter anymore because the AI is doing the building in the background and will let you know when it’s done. But I think it actually makes build speed more important, not less.

Since the flow state and focus don’t matter as much with async coding, now the math is actually meaningful and the small wins will compound even further. If you’re coding at 1x speed and building every 10 minutes, and the build takes 2 minutes, you’re spending about 20% of your time waiting on builds. Annoying, but manageable.

Now imagine an AI coding at 10x. It wants to build every minute to verify its work. But the build still takes 2 minutes. Suddenly 66% of the time is build. The AI isn’t going to get frustrated and check its email, but it’s also not doing useful work during that time. And if you’ve got multiple agents running in parallel, that bottleneck adds up and leaves even more open loops to manage.

When you speed up one part of a pipeline, the bottleneck shifts somewhere else. AI sped up the coding. Now the build is often the bottleneck. If anything, that’s an argument for investing more in build speed than we did before, the returns are even higher when you’re trying to iterate faster.

The Fantasy of Always and Never

Posted on by Eric F. Savage

One of the patterns I picked up during my freelancing years was to gently probe any time a client made an absolute statement. “We always do X” or “Y never happens.” These were usually non-technical clients describing their business processes, and I needed to understand them well enough to build software around them.

Honestly, I can’t remember a single one of those statements holding up to any scrutiny. Most fell apart with a single question. “What if Z happens?” “Oh, in that case, we do this other thing.” Almost none of them survived two or three follow-ups. It wasn’t that people were lying or even wrong, they just had a mental model of how things worked that was cleaner than reality.

This matters a lot when you’re building abstractions. When you create a shared component or a unified data model, you’re betting that these things really are the same. That they’ll change together. That the commonality you see today will hold tomorrow. Sandi Metz said “Duplication is far cheaper than the wrong abstraction.” I’d take it further, you need to be really certain an abstraction is right, or it’s probably wrong.

Abstractions and DRY are, in a sense, intentional single points of failure. That’s not to say they’re bad (I build them all the time), but it’s worth keeping in mind. You are hitching a lot to the same post. If you try to abstract physical addresses to cover all addresses everywhere, you’re left with basically zero rules because they’re all broken somewhere. Same for names, medical histories, pretty much any taxonomy that involves humans.

So now when I’m looking at a potential abstraction, I try to pressure-test it. “Users always have an email address” is fragile, there’s probably an exception lurking somewhere. “Users usually have an email address, and we handle it gracefully when they don’t” is something you can build on. If your abstraction can’t survive that kind of flex, it might not be an abstraction at all, just a few things that happen to look similar, until they don’t.

Java 15 though 25 Catchup

Posted on by Eric F. Savage

Continuing my previous post, let’s pick up the Java story where we left off…

Java 15 (Sept 2020)

Sealed Classes

A nice organizational tool. Not very handy for my personal projects but definitely useful across a large org.

Java 16 (March 2021)

Stream.toList()

Not much else in this release besides this minor but more readable improvement.

Java 17 (Sept 2021, LTS)

Nothing really new here just polish and finalizing things. It looks like this might be the dominant stable version in many enterprises and the baseline for open source right now.

Java 18 (March 2022)

Simple Web Server

This might be very useful for my prototypes. It wasn’t hard to do before but if this works with whatever is a good basic HTTP/API framework these days then it’s great that it’s built-in.

@snippet in Javadoc

I do actually comment my personal code fairly well (if I’m submitting it, even to a private repo) so this seems nice.

Java 19 (Sept 2022)

Record Patterns

I’m only reading some examples of this and definitely want to try it but it seems like it could be pretty clean. It’s kind of like Go interfaces, which I’m a fan of (though I wish they had a different name since they’re flipped backwards in some regards to older language’s interfaces).

Virtual Threads

Very interesting. I do tend to do a lot of concurrency in my projects so I’m definitely going to be spending some time with this one.

Java 20 (March 2023)

Scoped Values

ThreadLocals aren’t difficult per se but they are weird and easy to misuse. Streamlining the easiest usages of them seems like a win.

Java 21 (Sept 2023, LTS)

String Templates

This syntax seems like it might be a little too streamlined for readability, especially on review, but that seems solvable with tools so I’ll wait and see if I like these.

Structured Concurrency

I’ve rolled my own versions of this very useful concept many times so it would be great if this could standardize that.

Sequenced Collections

On the one hand this seems like a nice taxonomic update, but it also seems like it could be easily confused with Sorted Collections, but maybe that’s just me.

Java 22 (March 2024)

Statements before super(…) / this(…) in constructors

Ooh, this seems like it’s a bigger change than it appears on the surface. I have vague recollections of some significant class-layout workarounds for this limitation, but I’m also getting a Chesterton’s Fence vibe here in why this limitation existed in the first place.

Stream Gatherers

A few projects in my queue are data/statistics-based so this might come in handy, if third-party libraries don’t already handle this well enough.

Java 23 (Sept 2024)

Implicit classes & instance main

I never really had any issues with this boilerplate because the IDE always wrote it and it never really changed after that, but it’s cool that it got streamlined.

Java 24 (March, 2025)

Key Derivation Function API

Figuring out how to get signed APIs working almost always feels like it’s harder than it should be, so I’m all in favor of standardizing it. I’m not sure what the long-term impact is here because I’m sure the next great crypto approach will have some structural reason you can’t use this…

Java 25 (September 2025?, LTS)

The next LTS, nothing really major on the menu but a number of finalizations which will be nice.

Final Thoughts

I started this mini research project thinking there were going to be more things like lambdas which I thought were going to maybe take the language away from what I always liked about it, but that definitely doesn’t seem to be the case. There are a ton of streamlined features that work well within the same mental model and “spirit” of the language. I’m really looking forward to digging in and using almost all of them.

Java 8 though 14 Catchup

Posted on by Eric F. Savage

After a long hiatus, I’ve been increasingly motivated to do a few tech side projects. Working at Google everything there is Google-specific, or at least Google-flavored, and even if I wanted to use the same stuff I mostly couldn’t. My primary language there is Go, which is fine, but I’m going back to Java, at least for the backend/offline stuff, for now. I’m in the process of picking a stack, and starting with the language.

The last real Java project I did was Java 7, which was already aging at that point but we were going for stability and Java 8 was only a few months old when we started. Java 24 just came out, so I’ve missed 17 versions! I could just jump in but I haven’t really followed the language at all aside from some tinkering over the years and I was probably not really taking advantage of anything new. I think it would be fun to roll forward and read up on each version, seeing the highlights of what each version added or changed.

I’m mostly interested in the language aspects and the core libraries. I’m not that concerned with things like GC versions and improvements. Those are very important but for my hackery it’s unlikely I’ll need to get that far into the internals. Also, I’m going to discuss preview features mostly where they first appeared, not the iterations and finalizations in subsequent releases.

Java 8 (March 2014)

Lambdas

Now I remember why we didn’t upgrade, lambdas seemed a bit daunting and I don’t know if the tooling (Eclipse + Lombok at the time) had really caught up yet. I’m still not a huge fan of lambdas in any language, I think they are nice shortcuts but I’d prefer cleaner-delineated blocks. In JS where I use them the most I almost always define a function separately and then reference that, unless it’s just a line or two.

Type Annotations

I love annotations when used judiciously, @NonNull seems cool but I could see it getting out of control so I’ll have to wait and see how it’s used in the real world.

Optional

I’ve used this a lot in C++ and it definitely fits my be-explicit style.

Streams

A companion to lambdas, these will look strange at first but i use this pattern a lot in JS so it will probably feel right eventually.

Java 9 (Sept 2017)

JDK Modularization

This looks like a big deal, but probably more on the enterprise level than personal projects.

Collection factory methods (List.of, Set.of, Map.of)

Very nice ergonomics, looking forward to that especially for hacky prototype stuff.

Java 10 (March 2018)

Local-variable type inference (var)

Lombok had this but I didn’t use it too much. It’s really dependent on your tooling, as it adds complexity to refactoring which is one of the things I loved most about Java (and hopefully holds up!). I’m curious how this looks in real code.

Unmodifiable collection copies

More ergonomic polish that I’ll likely be using for hacking things up where hardcoding is common.

Java 11 (Sept 2018, LTS)

Looks like this was the first true LTS vesrion? Nothing groundbreaking on a language level, just some ecosystem cleanup.

Java 12 (March 2019)

Improved switch

Kind of cool. Looks like a strictish lambda variant.

Java 13 (Sept 2019)

Text Blocks

Pretty good, basically parity with most other languages at this point.

Java 14 (March 2020)

Records

Interesting. I always preferred a pretty clean “bean” structure and used Lombok for the boilerplate so this probably wouldn’t make my code look that much different but it’s always nice to be able to build on core concepts instead of just conventions.

Pattern Matching for instanceof

Makes sense. I found that needing to introspect/cast (since generics) was almost always a failure of the APIs involved but sometimes they were still unavoidable. I used them when trying to make things more magical and automatic and bury complexity behind a clean interface.

Helpful NullPointerExceptions

OMG how did this take 14 versions to happen.

Thoughts So Far

So that was about 6 years and some pretty decent improvements overall. I don’t want Java to go the way of C++ and just get more and more complex with a never-ending stream of new ways to do things, even if they are probably better. I like my Java to be boring and predictible, very easy to read, very easy to refactor, and very easy to hand off. Not all of these are great for that, notable lambdas and type inference but we’ll see how it goes.

Stack2020: Backend Basics

Posted on by Eric F. Savage
Image by Bethany Drouin from Pixabay

So the side project I mentioned a few posts ago is still just an idea, before I start building it I will need to pick a stack. I’ve been out of the non-big-tech loop for a while (one of the main drivers for this project) so this may take a while, but should be a fun experience.

The last time I picked a fresh stack was late 2013. We were building Mondogoal and needed to get to market very fast on a limited budget. We started in late November 2013 and needed to be in beta by March, and ready to launch (and get a gaming license!) in time for the 2014 World Cup that started in June. With two developers.

Somehow, we made it, and I think that one of the main factors was in how we picked our stack. In short, we kept things as boring as possible. The backend was all stuff I could do in my sleep: Java 7 with MySQL, using Maven and Spring, running on Tomcat in a mix of AWS and Continent 8. The only significant piece that I didn’t know well was Obsidian Scheduler, which is a fairly lightweight job scheduler, and we added Firebase later on. The frontend was backbone with a grunt build, which had been out for a while. This stability and going down well-trod paths let us focus on executing the business and product instead of tinkering and I doubt we would have been able to hit our deadline if we’d opted to explore some of the cool new stuff that was coming out. While the business didn’t succeed long-term, I have no regrets about those choices and can assign no blame to what we built it on.

Luckily, this new project has no deadline! It’s mainly a vehicle of exploration for me. I would definitely like to launch it, but if that takes months (unlikely) or years that’s OK. Let’s recap the “requirements” of this project:

  1. Learn how to build a modern front end.
  2. Give me a reason to explore the Google Cloud offerings (since those products are effectively my customers for my day job).
  3. Get back up to speed on things like analytics.
  4. Scratch an itch I’ve had for a long time (a nerdy economy-based game).
  5. Doesn’t risk any conflict with my day job.
  6. Give me something to blog about!

#2 is going to make some of these decisions easy. As a loyal dogfooder and company man (despite the lack of employee discount…), I will default to Google Cloud where possible. A quick check puts most of the basics on par with Amazon as far as cost goes. If GCP doesn’t offer it, then it’s up for grabs. Let’s start picking!

But wait…what is and isn’t part of The Stack?

I take a broad view of the term, to me The Stack is basically everything in the shop. You could (and I might in a future post) break this down into groups (tech stack, marketing stack, etc.), but to me if it’s a part of making the business run and it’s something you didn’t build yourself that you expect employees to get or be skilled at, then it’s part of The Stack. This includes everything from Java or Python to Salesforce and Gmail.

Domain Hosting

Winner: Google Domains

Cost: $14/year

Status: Up and Running

Initial Thoughts

I’ve got domains hosted at lots of places. GoDaddy, Namecheap, Hover, OpenSRS, I even still have one at Network Solutions. Compared to those, registering on Google was a pretty painless process. The pricing is more transparent than most other places (no first-year rates that go way up later). They also didn’t upsell junk I don’t need too hard. Setting up G Suite was also pretty easy (as you’d hope), I had it all running in like 15 minutes without touching any DNS entries.

To dogfood even deeper, I’m using a .app TLD, which Google owns, and was a few bucks cheaper than the other registrars.

Email Hosting

Winner: G Suite

Cost: $5/user/month

Status: Up and Running

Initial Thoughts

As most of us are, I’m pretty familiar with all of these tools, and I’m pretty sure most surveys would have Gmail at the top of people’s preferred email services. As a bonus, setting this up was super easy since the domain is also with Google.

Compute Servers

Winner: Kubernetes/Compute Engine

Cost: TBD

Status: Exploration

Reasoning

There were two things that came up in virtually every conversation/interview I had during my last job search, React and Kubernetes (AKA K8s). Virtually everyone was using these or trying to move to them. I’ve never touched K8s, so combined with #2 above, I feel like I should play with it.

I have used App Engine, and I assume non-K8s Compute Engine is pretty similar to AWS’s EC2 which I’ve used quite a bit, so I will fall back to those when it makes sense to do so.

Backend Language

Winner: Java

Cost: Free

Status: Defrosting

Reasoning

I am a moderately capable but generally reluctant programming language polyglot. My first employee profile badge at Facebook was “Committed Code in 5 Languages”. But I’ve never bought into “pick the best langauge for the job” and tend to favor the approach of “pick the best language for all the jobs”. This offer does not extend to JavaScript backends.

Java was my primary language from probably 2000 through 2016. Since then I’ve mostly been writing C++. I’ve grown to like the language, it is lightyears better than it was when I started writing Java, but I’ve never worked in it outside of the padded walls of FB/Google’s infrastructure, and to be honest, am not terribly interested in doing so.

While we upgraded our runtimes to Java 8 at Mondogoal after a bit, we never got around to really using any of the features, so I’m effectively only up-to-date through Java 7, and would like to explore the recent additions. There are also some new parts of the Java ecosystem that are worth exploring, like Quarkus and GraalVM.

Also, I just kind of miss working in it.

Runners Up

There are two languages I am interested in tinkering with, once I’ve warmed back up on Java: Kotlin and Rust. They both have had a pretty good reception and have some attractive features. Kotlin as a JVM language should be easy enough to experiment with. If I can find a task that would benefit from Rust I will probably give it a shot.

IDE

Winner: IntelliJ IDEA Ultimate

Cost: $149/$119/$89 for years 1/2/3+

Status: Trial

Reasoning

I initially wrote Java in emacs, then JCreator, then switched to Eclipse c2002 and used it through 2016. I’ve tried IntelliJ a few times over the years but never really got the hang of it or saw a lot of value in it.

However, Google does quite a bit of Java work, and their primary and only fully supported IDE for it is IntelliJ. I’ve also been using CLion (basically IntelliJ for C++) and it’s been OK.

The “Ultimate” edition of IntelliJ includes support for other languages and even React so that’s a strong argument in favor of trying it out. I’m not opposed to ultimately landing on using different tools to work in different languages (e.g. I often used Eclipse for Java and Sublime for JS), but if you can do it all in one, that’s nice.

My Eclipse muscle memory is very strong, so I expect this to be somewhat painful transition, but I will give it as fair a shot as I can manage.

Java Build

Winner: Gradle

Cost: Free

Status: Exploring

Reasoning

There are only two real choices here: Maven and Gradle. And given that Gradle uses Maven-style repositories, they aren’t even that different in many respects.

Maven

I’ve used Maven for many years, since Ant, and like most people had some struggles with it initially. I eventually learned to coexist with it, or at least avoid its sharp edges, and would just copy/paste my pom from one project to next and had minimal issues.

Gradle

Gradle has three main “advantages” over Maven that people seem to crow about.

One is that it’s written in Groovy, and you can therefore script your build and do advanced stuff more easily than writing a Maven plugin. I would put this in the Probably a Bad Idea category, like stored procedures. I bet there are some cases where this is useful, but probably far more where it’s a kludgy workaround to some other problem people don’t want to solve.

The second is that it’s written in Groovy, which is not XML. I always thought XML was nice when used properly, and that config files were one of those proper uses. However, something about it causes a primal aversion in many people and they convince themselves that things that are not XML are inherently better than things that are.

The third is that you can do different build versions more easily, and this one I get, especially in the context of things like Android apps. Given that I might be targetting different JVMs (regular and GraalVM) this might be useful, but probably won’t be.

So I’m not really impressed with Gradle either, but given that there are literally only two choices, I might as well know both. It’s pretty trivial for a small project to switch back or even run both, so this is a pretty low-risk experiment.

Source Control

Winner: Git + Monorepo

Cost: Free (plus hosting)

Status: Up and Running

Reasoning

I think there are only 3 real options these days for version control that don’t fall into the “never heard of it” category for most people.

Git

The dominant force, and the only one that many developers know these days.

Mercurial (hg)

I have grown to prefer Mercurial in a code-review + monorepo environment since starting to use it at Facebook. Implicit branches and easy commit management map very well to the “commits should do one thing” best practices as opposed to the pull request pattern where mainline commits should favor comprehensiveness. For a solo project this isn’t relevant and it’s basically the same thing as Git.

Subversion (svn)

For solo/small teams, SVN is totally fine, it’s basically how you’d be using a DVCS anyways, but if you don’t have a server running already then it’s probably not worth setting one up.

Mono vs. Multi Repo

For large organizations, monorepo is the clear way to go for reasons I can discuss elsewhere. For solo/small teams, it doesn’t really matter, and it *might* be better to split up your repos if you have a *very* clear separation (e.g. front/back end), which is how we did it at Mondogoal, but I would say to start with a monorepo and only split if there is a compelling reason to do so (e.g. regulations, licensing).

I’m going to call this a toss-up between Git and Mercurial and give Git the edge due to the fact that it’s massively more popular and more likely to integrate well with other things like IDEs and deployment tools.

Source Control Host

Winner: Google Cloud Source Repositories

Cost: Free to 5 users & 50GB storage/egress, then $1/user/month and $0.10/GB/month

Status: Exploring

Reasoning

Given that I’ve chosen Git, GitHub is the obvious first choice here, but since Google has a product we’ll invoke requirement #2. This also might integrate better with the other services I’ll be using, though I have barely researched that beyond the marketing.

One of the nice things with Git is that it’s trivial to multi-host, so if I ever find a compelling reason to also use GitHub, I can use both or just switch.

Next Up

There’s a lot left to do here, databases, frontend, and more. Stay tuned!

How to Pick a Software Stack

Posted on by Eric F. Savage
Image by Bethany Drouin from Pixabay

When starting a new company or project, one of the most daunting phases is selecting a stack. There are lots of things that factor into the decision, and anyone who has done this before will attest that you will make some bad choices, and some of those will be nearly impossible to undo, especially if you’re successful. Some questions to consider:

What do you already know/love/hate?

When considering your options for some piece of the stack, it is totally reasonable to award points to things you know and love, and deduct points from things you know and hate. It is, however, not a good idea for this to be the principal factor. You should be able to defend your favorite choice against the others objectively and quantitatively, not solely on the basis of “it feels better” or “I know it already”.

Are you just trying to execute a business/goal or are you trying to explore and learn something new?

One of my two virtues of engineering is curiosity. Throughout your career, you should always be looking for opportunities to explore new ideas, new tools, new methods. If this is some moonlight or passion project, actively try to weight new and unfamiliar things higher. If this is a major venture with lots of other people’s time and money, there is just as much to be gained from leveraging your experience and going for flawless execution of things you’ve already worked out the kinks on.

If you have teammates, what do they know/love/hate?

This is a case where you should set the hierarchy aside and try to get everyone’s opinion. If you’re a senior engineer or architect you can bring experience to the table but don’t discount the novel approaches and backgrounds of your teammates, regardless of their experience. We so often get stuck in our ways, even after only a few years, and avoid novelty and diversity in the name of risk.

Are you going to need to hire/outsource the work?

If your plan involves getting dozens or hundreds of people on board in the near future, you’re going to want to lean to the more popular/vanilla options. Trying to hire 300 people in the next 12 months to work on your Erlang/Neo4J stack running on an Arduino cluster is going to be much harder than getting people to do Java/MySQL on AWS.

Are you building a prototype, an MVP, or a “real” version.

Deciding, and more importantly understanding what you’re building is a topic worth exploring on it’s own, but roughly speaking you’re building one of three things:

The Prototype

You just need to get something “working” to explain your idea or your solution. Unless you’re doing something truly revolutionary (p99.999 confidence spoiler alert: you aren’t), try to prototype your idea/business/product OR your technology, not both at the same time. Also really commit to throwing this away and starting over with a proper plan if there is promise. I’ve seen/had too many prototypes turn into projects that have a pile of technical debt before they even really start.

The MVP

I feel the MVP is overused, and too often is an excuse for a poor quality product or a starved team/budget. You obviously want to iterate, and if you want to call your first release an MVP for buzzword sake, go ahead, but don’t write throwaway code unless you’re committed to throwing it away (see prototype above). I’ve lost count of the number of companies and projects I’ve talked to that are rewriting their Rails or Django app because those seemed like easier paths to an MVP and “now we just need to scale”. That’s like saying “I’m going to get a job, but then I’m going to go and get a completely different job that actually pays my bills.”

If you are truly building an MVP, then your process for stack selection is no different than building the real version, it’s just prioritized more aggressively.

The “Real” Version

The first step here is to figure out, as best as you and your team can, what the successful (often AKA profitable) version of your app/system looks like. Do you need 100 users to break even? Do you need a million? If you need a million, don’t spend a single minute building something that only works for hundreds. If you need 100, don’t spend any time building it for 1 billion (unless you get that for no additional work, E.G. something like Amazon S3).

What features are truly required to hit that goal? Make sure there is a place for them in your architecture even if you don’t plan to build them right away. Do you need support and sales teams to be successful? Make sure you’re putting hooks in for the tools they will need (E.G. CRM, ticketing, admin consoles).

What can you get “for free”?

Everything has a cost, in terms of time, money, people, etc. but those costs are not always the same across companies and projects. Your team’s experience can discount learning curves, your company may already have investments or licenses for things. Never use something just because it’s free, but make sure that is a part of your decision.

If you’re at a larger organization with a number of teams, consider if there is a team supporting certain aspects of what you might use. If there is a logging system that is in place and ready to go and has its own support, it’s should be a hard sell to build or buy or set up a new one just because of a few features. Often that team will even build what you need!

What are your competitors using?

Competitive analysis can be difficult, most software companies don’t go too deep on what they are using for any given situation, but that information can play a large part in your decision. If you find a conference talk where they talk about what they tried and had good or bad results from, you’ve possibly just saved yourself a lot of pain. You will often just get a lot of good ideas (“I never considered using X to do Y”), or pointers to things you’ve never heard of.

You might also find some competitive advantage. If they are using something you know is difficult to change or won’t scale well, make sure yours can beat them in those aspects. By the time they realize you’re about to pass them it might be too late for them to adjust.

What will fail first?

As you’re building your stack, know where the weak points are, and don’t be shy about circulating that information. If you can’t get to your target latency because some component will use 90% of that budget, try to find a replacement early rather than assuming you can plug something else in later. You should be confident that all of your core components can hit your targets, and if you aren’t sure, validate that now. You’ll find that the interconnected nature of systems will often lead you to replacing more than one piece at a time, which only makes it less like you’ll be able to do it, and more likely to fail.

What else?

Picking a stack is harder, and takes more time, than most people realize. I’m sure there are other things to consider and I’ve love to hear your thoughts on this in the comments.

How do you do, fellow kids?

Posted on by Eric F. Savage

I’ve spent the last few years sequestered away in “big tech” which has been an interesting and mostly positive experience, but one of the more unexpected side effects, especially as an extreme generalist, has been the specialization on the technical side of things.

From when I first started coding for money in the mid 90s I was always a true “full stack” engineer, even before that was a term. Give me a bare server and a sketch and I could handle the rest, even some basic UI design. As backends became more complex and interesting I gravitated that way, but was still pretty up-to-date on front-end technology for a nearly 20 year stretch.

The front end world got increasingly difficult to keep up with on a casual basis as single page apps became a thing and powerful frameworks took over. At Mondogoal I was entirely back-end from late 2013 onward, although I was also the product owner and oversaw design and branding. At Facebook I worked on maps and geocoding, writing mostly backend C++ code. At Google I’m deeper yet, working on the networking team.

During my last job search I initially still had “full stack” on my resume, but realized that I didn’t think I met that bar any more, namely because I didn’t really know any of the things most people were using these days, like React or Angular. My skills in that area, (e.g. jQuery), are no longer in demand at places I’d want to work for, supplanted by experience with scalability and distributed systems.

And don’t get me wrong, I have no regrets about this, at all. Back-end experience and knowledge decays much slower than the front-end, so now that I’m often the oldest person in the room, that’s a feature, not a bug.
Back-end work also tends to pay more and offer more paths to leadership roles. But I think I’m still a full-stack developer at heart, and the deeper I go the more wistful I get for being able to make something you can see and touch.

Enter, the Side Project

I haven’t had a software side project in years. With all that’s been going on, especially becoming a parent, my energies have been allocated elsewhere. But there’s been an increasingly more noticeable hole there that I need to fill.

I brainstormed a few ideas and kicked them around until one of them, a browser-based game, settled out that checked off most of the boxes:

  • Learn how to build a modern front end.
  • Give me a reason to explore the Google Cloud offerings (since those products are effectively my customers for my day job).
  • Get back up to speed on things like analytics.
  • Scratch an itch I’ve had for a long time (a nerdy economy-based game).
  • Doesn’t risk any conflict with my day job.
  • Give me something to blog about!

Now what?

I’m going to talk about this in more detail in a future post but after spending a bit poking around the current front end world, I feel like I’m basically starting over. React, Angular, Vue. Yarn, webpack, JSS. The list goes on and on. Even my JavaScript is out of date, with all of the additions over recent years, and TypeScript being a serious option now (it was still kind of a toy last time I tried to use it back in 2012/2013).

This is a bit intimidating, but more so exciting. There’s so much to learn I haven’t even really started to think about the actual project or how feasible it will be to launch. Stay tuned for updates!

Logging Like it’s 2011

Posted on by Eric F. Savage

Earlier this year I revisited how I was logging things in Java, and decided I would try a hybrid approach.  I’m reporting back to say that it’s been successful. There are basically two ways I do it now:

Use java.util.Logging for libraries

It turns out that there’s nothing actually wrong with JUL aside from its limitations in terms of configuration.  It has different names than the Ceki Gülcü loggers (warning() instead of warn(), finest() instead of trace(), etc.) but otherwise works the same.  The configuration side of things I can’t even speak do as I haven’t had to configure it, I never use the actual JUL output.

Use Logback for applications

As suspected, Logback is basically just a new version of Log4J.  It’s got some niceties like MDCInsertingServletFilter that means I don’t have to write that filter myself anymore, and it’s supposedly faster and more stable, so there’s no reason I can see not to use it.  I also like that it has a format for condensing class names without completely losing the package names, so it goes from “com.efsavage.application.servlet.LoginServlet” to “c.e.a.s.LoginServlet” instead of just “LoginServlet”.

Why use two?

I like the fact that my libraries have zero logging dependencies and configuration, so I can always pop them into another app without adding baggage or conflicts or much additional configuration.  I can upgrade the logger in one application without having to deal with version conflicts of my other apps and having to do classpath exclusions and that type of nastiness.

Tip

If you do it this way, and you see your JUL logging showing up twice, you can edit the default logging config in your JDK installation, or if you prefer to leave that untouched as I do, try this (via Java Bits):

java.util.logging.Logger rootLogger = LogManager.getLogManager().getLogger("");
Handler[] handlers = rootLogger.getHandlers();
for (int i = 0; i < handlers.length; i++) {
rootLogger.removeHandler(handlers[i]);
}
SLF4JBridgeHandler.install();

This basically yankts out the default logger and let’s SLF4J do it’s thing alone.  In a webapp you’ll probably throw this in your context listener or startup servlet where you previously did your log4j config.