Stack2020: Backend Basics

Image by Bethany Drouin from Pixabay

So the side project I mentioned a few posts ago is still just an idea, before I start building it I will need to pick a stack. I’ve been out of the non-big-tech loop for a while (one of the main drivers for this project) so this may take a while, but should be a fun experience.

The last time I picked a fresh stack was late 2013. We were building Mondogoal and needed to get to market very fast on a limited budget. We started in late November 2013 and needed to be in beta by March, and ready to launch (and get a gaming license!) in time for the 2014 World Cup that started in June. With two developers.

Somehow, we made it, and I think that one of the main factors was in how we picked our stack. In short, we kept things as boring as possible. The backend was all stuff I could do in my sleep: Java 7 with MySQL, using Maven and Spring, running on Tomcat in a mix of AWS and Continent 8. The only significant piece that I didn’t know well was Obsidian Scheduler, which is a fairly lightweight job scheduler, and we added Firebase later on. The frontend was backbone with a grunt build, which had been out for a while. This stability and going down well-trod paths let us focus on executing the business and product instead of tinkering and I doubt we would have been able to hit our deadline if we’d opted to explore some of the cool new stuff that was coming out. While the business didn’t succeed long-term, I have no regrets about those choices and can assign no blame to what we built it on.

Luckily, this new project has no deadline! It’s mainly a vehicle of exploration for me. I would definitely like to launch it, but if that takes months (unlikely) or years that’s OK. Let’s recap the “requirements” of this project:

  1. Learn how to build a modern front end.
  2. Give me a reason to explore the Google Cloud offerings (since those products are effectively my customers for my day job).
  3. Get back up to speed on things like analytics.
  4. Scratch an itch I’ve had for a long time (a nerdy economy-based game).
  5. Doesn’t risk any conflict with my day job.
  6. Give me something to blog about!

#2 is going to make some of these decisions easy. As a loyal dogfooder and company man (despite the lack of employee discount…), I will default to Google Cloud where possible. A quick check puts most of the basics on par with Amazon as far as cost goes. If GCP doesn’t offer it, then it’s up for grabs. Let’s start picking!

But wait…what is and isn’t part of The Stack?

I take a broad view of the term, to me The Stack is basically everything in the shop. You could (and I might in a future post) break this down into groups (tech stack, marketing stack, etc.), but to me if it’s a part of making the business run and it’s something you didn’t build yourself that you expect employees to get or be skilled at, then it’s part of The Stack. This includes everything from Java or Python to Salesforce and Gmail.

Domain Hosting

Winner: Google Domains

Cost: $14/year

Status: Up and Running

Initial Thoughts

I’ve got domains hosted at lots of places. GoDaddy, Namecheap, Hover, OpenSRS, I even still have one at Network Solutions. Compared to those, registering on Google was a pretty painless process. The pricing is more transparent than most other places (no first-year rates that go way up later). They also didn’t upsell junk I don’t need too hard. Setting up G Suite was also pretty easy (as you’d hope), I had it all running in like 15 minutes without touching any DNS entries.

To dogfood even deeper, I’m using a .app TLD, which Google owns, and was a few bucks cheaper than the other registrars.

Email Hosting

Winner: G Suite

Cost: $5/user/month

Status: Up and Running

Initial Thoughts

As most of us are, I’m pretty familiar with all of these tools, and I’m pretty sure most surveys would have Gmail at the top of people’s preferred email services. As a bonus, setting this up was super easy since the domain is also with Google.

Compute Servers

Winner: Kubernetes/Compute Engine

Cost: TBD

Status: Exploration


There were two things that came up in virtually every conversation/interview I had during my last job search, React and Kubernetes (AKA K8s). Virtually everyone was using these or trying to move to them. I’ve never touched K8s, so combined with #2 above, I feel like I should play with it.

I have used App Engine, and I assume non-K8s Compute Engine is pretty similar to AWS’s EC2 which I’ve used quite a bit, so I will fall back to those when it makes sense to do so.

Backend Language

Winner: Java

Cost: Free

Status: Defrosting


I am a moderately capable but generally reluctant programming language polyglot. My first employee profile badge at Facebook was “Committed Code in 5 Languages”. But I’ve never bought into “pick the best langauge for the job” and tend to favor the approach of “pick the best language for all the jobs”. This offer does not extend to JavaScript backends.

Java was my primary language from probably 2000 through 2016. Since then I’ve mostly been writing C++. I’ve grown to like the language, it is lightyears better than it was when I started writing Java, but I’ve never worked in it outside of the padded walls of FB/Google’s infrastructure, and to be honest, am not terribly interested in doing so.

While we upgraded our runtimes to Java 8 at Mondogoal after a bit, we never got around to really using any of the features, so I’m effectively only up-to-date through Java 7, and would like to explore the recent additions. There are also some new parts of the Java ecosystem that are worth exploring, like Quarkus and GraalVM.

Also, I just kind of miss working in it.

Runners Up

There are two languages I am interested in tinkering with, once I’ve warmed back up on Java: Kotlin and Rust. They both have had a pretty good reception and have some attractive features. Kotlin as a JVM language should be easy enough to experiment with. If I can find a task that would benefit from Rust I will probably give it a shot.


Winner: IntelliJ IDEA Ultimate

Cost: $149/$119/$89 for years 1/2/3+

Status: Trial


I initially wrote Java in emacs, then JCreator, then switched to Eclipse c2002 and used it through 2016. I’ve tried IntelliJ a few times over the years but never really got the hang of it or saw a lot of value in it.

However, Google does quite a bit of Java work, and their primary and only fully supported IDE for it is IntelliJ. I’ve also been using CLion (basically IntelliJ for C++) and it’s been OK.

The “Ultimate” edition of IntelliJ includes support for other languages and even React so that’s a strong argument in favor of trying it out. I’m not opposed to ultimately landing on using different tools to work in different languages (e.g. I often used Eclipse for Java and Sublime for JS), but if you can do it all in one, that’s nice.

My Eclipse muscle memory is very strong, so I expect this to be somewhat painful transition, but I will give it as fair a shot as I can manage.

Java Build

Winner: Gradle

Cost: Free

Status: Exploring


There are only two real choices here: Maven and Gradle. And given that Gradle uses Maven-style repositories, they aren’t even that different in many respects.


I’ve used Maven for many years, since Ant, and like most people had some struggles with it initially. I eventually learned to coexist with it, or at least avoid its sharp edges, and would just copy/paste my pom from one project to next and had minimal issues.


Gradle has three main “advantages” over Maven that people seem to crow about.

One is that it’s written in Groovy, and you can therefore script your build and do advanced stuff more easily than writing a Maven plugin. I would put this in the Probably a Bad Idea category, like stored procedures. I bet there are some cases where this is useful, but probably far more where it’s a kludgy workaround to some other problem people don’t want to solve.

The second is that it’s written in Groovy, which is not XML. I always thought XML was nice when used properly, and that config files were one of those proper uses. However, something about it causes a primal aversion in many people and they convince themselves that things that are not XML are inherently better than things that are.

The third is that you can do different build versions more easily, and this one I get, especially in the context of things like Android apps. Given that I might be targetting different JVMs (regular and GraalVM) this might be useful, but probably won’t be.

So I’m not really impressed with Gradle either, but given that there are literally only two choices, I might as well know both. It’s pretty trivial for a small project to switch back or even run both, so this is a pretty low-risk experiment.

Source Control

Winner: Git + Monorepo

Cost: Free (plus hosting)

Status: Up and Running


I think there are only 3 real options these days for version control that don’t fall into the “never heard of it” category for most people.


The dominant force, and the only one that many developers know these days.

Mercurial (hg)

I have grown to prefer Mercurial in a code-review + monorepo environment since starting to use it at Facebook. Implicit branches and easy commit management map very well to the “commits should do one thing” best practices as opposed to the pull request pattern where mainline commits should favor comprehensiveness. For a solo project this isn’t relevant and it’s basically the same thing as Git.

Subversion (svn)

For solo/small teams, SVN is totally fine, it’s basically how you’d be using a DVCS anyways, but if you don’t have a server running already then it’s probably not worth setting one up.

Mono vs. Multi Repo

For large organizations, monorepo is the clear way to go for reasons I can discuss elsewhere. For solo/small teams, it doesn’t really matter, and it *might* be better to split up your repos if you have a *very* clear separation (e.g. front/back end), which is how we did it at Mondogoal, but I would say to start with a monorepo and only split if there is a compelling reason to do so (e.g. regulations, licensing).

I’m going to call this a toss-up between Git and Mercurial and give Git the edge due to the fact that it’s massively more popular and more likely to integrate well with other things like IDEs and deployment tools.

Source Control Host

Winner: Google Cloud Source Repositories

Cost: Free to 5 users & 50GB storage/egress, then $1/user/month and $0.10/GB/month

Status: Exploring


Given that I’ve chosen Git, GitHub is the obvious first choice here, but since Google has a product we’ll invoke requirement #2. This also might integrate better with the other services I’ll be using, though I have barely researched that beyond the marketing.

One of the nice things with Git is that it’s trivial to multi-host, so if I ever find a compelling reason to also use GitHub, I can use both or just switch.

Next Up

There’s a lot left to do here, databases, frontend, and more. Stay tuned!

Workstation Setup 2011

A new workstation means it’s time to install lots of stuff, and we’re still a long way from DropSteam.  Here’s my log from fresh Windows 7 install in a new VM image to a functional development environment:

First, I hit Ninite and install:

  • All Browsers (I use Chrome as default)
  • Adobe Air
  • Adobe flash
  • VLC Player
  • IrfanView
  • Inkscape
  • Paint.NET
  • Foxit Reader
  • PDFCreator
  • CutePDF (yes, you need both PDF printers, as it’s fairly common for one of them to have a problem with a particular job)
  • CCleaner (tweak settings before running so you don’t nuke more than you want to, like browser history)
  • 7-Zip
  • Notepad++
  • WinSCP
  • JDK

Then I grab the ZIP of all of the Putty programs.  I put installer-less programs like this in C:\bin

Cloudberry Freeware for Amazon S3 buckets.

Download JavaDoc and install in JDK folder.

Download Eclipse (3.4, not impressed with 4.X so far) and then:

  • Set text font to 8pt Lucida Console
  • Most companies and many open source projects are still using SVN so I install the Subclipse plugin for Eclipse.
  • I’m not a huge fan of m2eclipse but I find that doing eclipse:eclipse from the command line costs you too much integration, so I use it.
  • Turn on all compiler warnings except:
    • Non-Externalized Strings – Enable as-needed
    • serialVersionUID – Not useful for most projects
    • Method can potentially be static – False positives on unit tests
  • Turn on line numbers
  • Install CheckStyle.
  • Install FindBugs.

Maven 3 seems a little rough around the edges so I still use Maven 2.X

Install Cygwin and add git, svn, curl, and ssh packages.

Install MySQL Community Edition.  During the installer I:

  • Change the charset to utf8
  • Fix the windows service name to something like MYSQL5
  • Add to windows path
  • Add a password

JRebel.  You’re using this, right?  If not, slap yourself and then go get it.  Pay for the license out of your own pocket if you need to.

Lombok.  I have finally used this on a real project and can say it’s ready for prime-time.  It does not work with IntelliJ IDEA but I haven’t really seen any reasons to use IntelliJ that outweigh the benefits of Lombok.

Photoshop Elements because while IrfanView is great for viewing and Paint.NET is great for simple edits, you will at some point need a more powerful editor.  Also most designers work in Photoshop so this let’s you open those files directly.

Photoshop Elements+ is basically a $12 unlock of some of Elements’ crippled features.  For me it’s worth it for tracking alone.

LastPass is useful even if you don’t store anything sensitive in it, it’s great for testing webapps with multiple users.

I use Git for my own work so we’ll need that. Don’t forget to set your name!

I also make some Windows tweaks:

  • Set desktop background to black.
  • Check “Show hidden files, folder and drives”.
  • Uncheck “Hide extensions for known file types”.
  • Set %JAVA_HOME to JDK directory.
  • Add Maven’s bin directory to %PATH
  • Add C:\bin to %PATH

I will obviously add more over time, but this is the stuff I know I will need.  What’s great is that almost all of it is free, and it can all be downloaded (except the original Windows install), so no disks required like the old days

You might think this is an incomplete list, where is my email client, my MS/Open office, my music player?  I don’t use those unless I have to.  Keep in mind that this is a VM so some of this software is installed on the Host OS, while the rest of it I prefer to use web-based solutions (Meebo, Google docs, webmail) so there’s no issues of having to keep updating settings.

To Git or not to Git

You might notice from my past few posts that I’m basically going through my entire stack of tools and re-evaluating everything. This time it’s version control.

A little history

I’ve mostly used SVN, and before that, CVS. I’ve tinkered with some of the more heavyweight ones like ClearCase, TrueChange, and Visual SourceSafe as part of consulting gigs, but only enough to know that they were skills unto themselves, and ones I didn’t especially want to let into my brain. My personal repo up till now has been SVN after finally switching over from CVS a few years ago.

Why SVN?

The short answer is, because it’s easy. The longer answer is that it’s easy to set up, it’s fairly hard to break, and it has a decent Eclipse plugin. You might notice that I didn’t mention anything about branches, or rollbacks, or speed, or centralized vs. distributed. Those things don’t really matter to me if the first three requirements aren’t satisfied.

Branches are the devil

I don’t hate branches because they were a legendary nightmare in CVS. I don’t hate branches because svn merge rarely works. I hate branches because of the mental cost they inflict on a team.

Having a team work in multiple branches is, as far as I’ve ever seen it, a sign that your team is too big or your project is too monolithic or your effective management and oversight capabilities are lacking.

There are cases where branches don’t impose such a cost, however. If there are no plans to ever merge a branch back to trunk, they are simply an experimental offshoot where a few snippets might be pulled into trunk/master, that’s fine.

If everyone is switching to a new branch, that’s also fine. At one point we had to rollback to code from over a month prior, and weren’t sure if we were going to get back to trunk or not. Everyone switched to the new branch, and luckily everyone was able to switch back to trunk a few months later. It took days to merge back, and that wasn’t SVN’s fault, that was just a lot of human time that was required to pull two very different versions together safely and not leave landmines all over the place.

Not on my resume

I don’t put any VCS software on my resume, because, while it’s absolutely important to use one, I don’t think it’s that important which one I know or use (it is a good interview question, though). They aren’t really that hard to learn, and since everyone uses them slightly differently there’s no avoiding some ramp-up time. If an employer ever has me and another candidate so close that our VCS experience is the deciding factor, please, take the other person.

Dirty little secret

I don’t actually have the command line version of svn or cvs installed on any of my workstations. Nor do I have standalone GUIs or shell integration like Tortoise. I know the command line, and use it on servers, but I do all my actual development with Eclipse’s integrated client. I’ve actually even used Eclipse to manage svn projects that were Flash or C. I just find the command line so restricting and linear for what really is a very non-linear task. The eclipse subversion plugin took years to go from bad to passable, and it’s still not as good as the CVS version, which is one reason I never grew too attached to SVN.

Why? I need to see everything that’s changed, that’s dirty. I diff every file individually before I commit. I often find changes I didn’t really need to make, and nuke them. Sometimes I find that I actually didn’t account for a situation the old code did. Many times just looking at my code in this way makes me think just different enough that I come up with a better way of doing it. I simply don’t have this visibility amidst the >>> and <<< of a command-line diff, because it’s not my native development environment.

Enter git

So, even though I’d be happy to continue using SVN, I need to see what all this git hubbub is about. It’s been around for over 5 years now, so clearly it’s not a fad. It’s also been vouched for by enough voices I respect that it deserved a shot. There is also Mercurial and Bazaar, but I haven’t seen nearly the same level of buy-in from trusted people for those.

My sideline view is that git is favored by the python people and the “ruby taliban”. Mercurial seems to be favored by the Microsoft and enterprise crowd, and bazaar is somewhere out back playing with package maintainers. Java is still mostly in SVN land, probably because it’s more mature, more corporate, and slower moving. 5 years isn’t a long time in Java years these days, so I’d bet that a high percentage of projects people are still working on are from when git was just Linus stomping his feet. The Spring/JBoss people seem to have gone the Mercurial route, while Eclipse is going git.

Git also has github, which is used by some people I know, while I don’t know anyone personally who is using Mercurial’s version, bitbucket (or even using Mercurial for that matter). So I ultimately went with what Eclipse and my friends were using over the other interests, and started with git. From what I understand the differences are slight in the early stages anyways, this was really more a matter of trying DVCS vs. VCS.

First steps

I started off with Github’s helpful handholding, which included installing msysgit. I imported my projects and used it in earnest for a few days. Once I was confident in my ability to actually get stuff up there, I dug a little deeper.

I read the Pro Git book, which I need to call special attention to because it’s really, really, good. It’s short, concise, has diagrams where you need diagrams, and ranks high in terms of how computer books should be written. If you don’t know anything about git, spend a night reading through this, and you will know plenty.

What about the secret?

So yeah, I was using the command line. Lean in closer so I can tell you why…BECAUSE THE GUIS ARE TERRIBLE. They’re not just ugly, they’re interface poisoning at a master level, think what an even more complicated Bugzilla would look like. And yes I say “they”, because there is more than one, and they’re all basically someone jamming the command line output into various GUI toolkits. Part of this is git’s design, but I know that someone will figure this out.

Git’s design allows for a huge number of permutations of workflow, which means there are a number of extra steps when comparing to something like subversion. On the command line, this doesn’t seem to hurt that much (in comparison). But GUIs don’t deal with situations like this very well. They can either be helpful and guide you down a path, or play dumb and wait for you to hold it’s hand. All of the Git guis I’ve seen so far do the latter.

Am I being a stick in the mud and saying that something as marvelous as git should be constrained to the simplicity of dumb old svn? Actually, yes. I should be able to edit some files, see a list of those files, diff them against the “real” version of the file (as in the one everyone else sees) and commit, with message. Then go home. I don’t care about SHA-1 hashes because I don’t remember them, I only need them when you need to tell me that two things are different. I don’t care about branches other than knowing which I’m in (we’ll get to this next). I don’t want to be bothered with any of this fancy information if nothing is broken (or going to break if I continue).

This isn’t actually a problem of git. This is a problem of people being indecisive when it comes to UIs. If you do everything, you fail. If you do nothing, you fail. If you do any subset of everything, you fail for some people. That’s OK, don’t worry, you can optimize as you go. Your first priority should not be exposing the power of git, it should be letting me put my code on the server so I can go home. Let me drop into all this fancy stuff with local branches and pushing tags and rebasing and such on a case-by-case basis, when I need it, and when I’m good at it.

What about the branches?

Git does branches right. I could go into more detail, but Pro Git does it so well I won’t bother, so just go read that. What git does not do very well, and I’m not sure if anything can do as long as humans are involved, is mitigate the mental cost of a distributed branch. It does make merging much more feasible, and allows for a larger number of cases where branches are not going to cost much, but they still need to be used with discipline and restraint.

The new idea that git adds is the local branch. I haven’t had a chance to use this much yet, but this is the feature that may ultimately win me over. I can look back and say “when have I ever needed a local branch?” and the answer would be “a few times, but not often”. But if I look back and ask “when would I have benefited from a local branch” and answer would be “hmm, I don’t know, but probably more often than I needed one”.

The example of the hotfix scenario (where you need to fix/test something from last week’s release and trunk/master isn’t ready) isn’t very compelling to me as an SVN user. It’s easy to make an SVN branch for something like this. I svn copy and switch it if my local copy is clean. If not, I can check out the project again, or if its a big one, I copy it over and svn switch it. Not as easy as git, but then again, I don’t generally have to make alot of hotfixes either.

The issue scenario (work one one issue per branch, merge when/if complete) is more compelling. I’d like to say that it isn’t and that I try to start and finish one issue at a time, but obviously that doesn’t happen enough. I like the fact that it’s so cheap to make a branch that I might as well just do it all the time. If I didn’t end up needing it, no harm done. If I did end up needing it, because some other issue suddenly got more important and the one I’m working needs to chill, then I’m glad it’s there.

The real win here is that nobody has to know about my branch, which means they never have to wonder what’s in it, or if its up to date. This means there is no cost to my team because I have a branch that is 2 weeks out of date. There is cost to me, but no more than having multiple versions of the project checked out, or a set of patch files sitting there waiting for me to get back to it.

One more thing

The fact that every developer has a full copy of the repository on their computer, basically for free? That’s really nice. Sure, you do backups and the chances of your computer being the last one on Earth with a copy of the repo is slim, but the peace of mind is undeniable.

The Verdict

The only reason this is really a decision at all is that git is harder to use for normal stuff. The command line can be scripted, so if someone got the GUI to the point where you start with an SVN-style workflow and deviate as needed, there really would be no argument to using SVN from what I can see.

My decision is that git is the winner here, despite that massive failing because I have faith that a combination of two things will happen. I will learn the GUIs better because it’s worth my time to do so, and someone will eventually figure out how to make it smarter and simpler.

I could not fault someone for sticking with SVN, even for a new project, because you can always import it later, but I will be starting new projects in git.