Skip to content

Software Development at Airtime

In our case, tickets start in an Open state and are assigned by the engineering manager to a developer. That developer moves that ticket to In Development at which point it belongs to that developer.

When that developer has implemented the ticket, they move it to Branch Testing and the ticket is now owned by QA. QA can either fail the Branch Testing and send the ticket back to the developer or indicate Branch Testing Passed at which point the developer owns merging the feature branch.

Once merged QA once again owns performing integration testing and moving the ticket from On Staging to Ready for Release.

Note that in the diagram above Ready for Release transitions to Resolved once a new version of the app has been released or the service deployed.


As soon as an organization grows to more than a few people, communication and interruption become a real problem and slow down your organization’s progress.

The secret to combatting this is to make sure tickets are in the right state and up to date. This allows any team member to asynchronously make the right choice as to what to do without needing to wait on or interrupt anyone else.

Minimal Complexity

The easier it is for a developer to have mental model of your development process the better since it will feel like a tool rather than a constant puzzle to be solved. Even if you are able to automate much of the bookkeeping process it is good to understand what is happening and why.

Save your focus and concentration for developing software that matters, not for trying to remember how your software development process works.

The worst situation to be in is having a manual complex system since that will surely burn out your team.

Similar to our software development process goals above we also have specific goals we wanted to achieve with how we used Git.

  1. Stable Development Branch. An up-to-date, stable branch as a starting point for features so that each developer is not distracted by bugs and build issues that aren’t theirs.
  2. Feature branches. So that engineers can share in progress work with the product team and QA that are only merged once everyone is satisfied with the changes.
  3. Release branches. Allows us to know what changes went into a release and also allow for an easy starting point to make hot fix releases with targeted changes.

These goals could be achieved with the much-referenced “A Successful Git Branching Model” (diagram above) but for us it violated our high level goal of minimal complexity.

Instead what we opted for is a simpler model in which we have a develop branch that has the latest features where other developers can start branches for their feature work from, and where periodic release branches can be branched from. The two models are largely similar except we do not use master and our hotfixes are done directly on the release branches.

Airtime’s Simplified Git Branching Model

Previously I stated that you should focus on what matters and this means also knowing when to automate so you can use the saved time to build product instead of manually distributing a test build for example.

It can be fun to write scripts and tools but if they are only used infrequently or only by a small subset of people of your organization then it is probably a net loss.

Use your head or a convenient table.

Here’s a few DevOps projects we thought were worth it at Airtime.

Automating Builds

At Airtime anytime code is pushed to a feature branch, develop, or a release branch on GitHub a build is kicked off.

If you have more than one developer on a team you should always automate the building of branches that are part of your developer flow.

Here’s a couple of the reasons:

Minimizing downtime. Per our branching model above, every developer starts a new feature from the develop branch. If develop does not build then no developers can start a new feature without fixing it first. This means multiple developers may spend time fixing the same build failure. Takeaway: Always build develop every time it is updated.

Checking all targets and tests. In our experience developers rarely check that all targets/configurations build and tests pass when pushing code to a feature branch. And frankly it’s probably better that they don’t since rather than eat up the CPU cycles of their development machine they can let a build server check.

The best thing about automating your builds is that once you set it up, it is easy to add it everywhere. Given how cheap build services are these days and the benefit it provides, there is little reason not to do it.

Distributing Builds

We also thought it made sense to make it as easy as possible to distribute builds. This means that it would not only be easy for developers but for anyone to send themselves a build they needed.

Specifically we wanted to make it easy for

  1. Product managers to see in progress work
  2. QA to be able to get a build for a feature branch they are testing,
  3. Broader Airtime team to get an early preview of what is about to be released
  4. For developers to upload builds to the App and Play store (mobile teams only)

For the first three goals, we built a system where if someone comments qabuild or stagebuild on a pull request, a build is triggered on Jenkins (needed for keyword detection) which starts a CircleCI job which uploads the build to Fabric and distributes it to the QA team or everyone respectively.

We also build develop once a day if there have been any changes that day and distribute the build to the entire company so everyone can see the latest and greatest changes. They also feel confident that if they see a bug that it is probably still a bug and should report it.

The fourth goal is currently only relevant for the iOS team. Any time a build is pushed to a release branch it is automatically uploaded to the AppStore. We do not release automatically but it simplifies the process of getting release builds uploaded.

Integrating Development Tools

At Airtime we decided to integrate our development tools as much as possible to reduce time wasted checking multiple UIs and keeping them in sync. This has become much easier over the years so there is little reason not to do it.

There are some suites of tools such as GitLab that come integrated out of the box. Chances are though that your team is using at least some tools that require some level of integration.

Here are a few examples of what we’ve done:

Slack and Everything.

At this point most organizations that run Slack have integrated it with their development flow and there are plenty of articles about it. We try to use it as our information center for everything.

We use Slack integrations for notification of build failures on CircleCI and Jenkins, backend service deployment status from AWS, creating JIRA tickets in the flow of a discussion, service interruption alerts from 3rd party services we rely on, notifications on ticket flow on jira, and the list goes on.


In order to automate the bookkeeping tasks that should be done in Jira when working on a ticket we created a tool called Atflow (short for Airtime Flow) based on hubflow and go-jira.

Atflow allows us to start development work on a ticket from the command line and the tool takes care of naming the branch and updating the ticket state in JIRA and making sure it is in the current sprint.

Starting feature development in atflow

By ensuring that you are naming your branches and pull requests correctly you get the little magic in Jira like this on every ticket.

Product Development

In the interest of brevity and staying on topic I did not go into much detail about roadmapping, feedback, and task breakdown.

Suffice to say that we have product managers that take input from a variety of sources, create a feature roadmap, collaborate with engineers and designers on the best way to break up the roadmap into high level feature epics which engineers can then break down into individual tickets.

In the future we will have an article from the product team on how we do product development.

Process Variations Between Teams

Note that building an app like Airtime requires multiple engineering teams and each team has slight variations in the process necessitated by the challenges they face but large in part the process is the same across teams.

A few examples of major differences I did not mentioned above:

  • Our backend team relies on test driven development (TDD) rather than manual QA
  • Our media team has a combination of automated and manual QA test suites for validation of new media server and client library releases.

Manual QA Versus Automation

For app testing we rely on manual QA as opposed to automated UI testing since our product spec changes frequently as we adapt the product to what users want. We felt it was more actually more effort to maintain the tests until we got to a certain level of product market fit.

Good luck developing and improving your own software development process for your organization.

If you like what you see and what we are working on then come join us!

Check out our open roles!

Originally published at on July 2, 2019.

Published inUncategorized

Be First to Comment

Leave a Reply

%d bloggers like this: