O’Reilly Software Architecture Conference 2018 – London, United Kingdom

About this course

so we started on migration early in 2017 but even having good reasons to do the migration it was still a major challenge you do not want to change horses midstream

you’ve got enough other stuff going on we had a lot of services live already at this point in production and under active development when I say under active development this is a diagram

that’s attempting to work out all the functional stuff we wanted to deliver last year we had five teams working on it there were a lot of interdependencies it was extremely complicated and then we

had a migration team trying to move everything underneath it so you can imagine that was quite difficult and we had to run things in parallel for quite a long time because when you’re born

from scratch you often want to build a platform with a few services in it but actually we wanted to know whether we could run our complicated routing and failover logic on kubernetes which meant

having to migrate quite a lot of things this took a while we ran in parallel for a long time in fact we did well over 2000 code releases while running at least part of the stack in parallel

that’s about 8 about a year and even if it takes 10 minutes extra to deploy to your two parallel stacks that’s 47 working days of extra deployment time if you’ve got two thousand at scale these

things start to add up so when we started doing this we needed to make some decisions about how we were going to run stuff in parallel how to make changes for one stack for the other how

to keep things in sync between the two stacks so the first decision was about where do we commit code so we started off by putting all of our changes for kubernetes onto separate branch which

kept the work the migration team were doing separate from the work everyone else was doing but that doesn’t work well if those services are also getting active code changes for other reasons

because eventually you have to merge the code and the feature team doesn’t want to have to merge it because it’s it’s effectively paying a tax on all their development and the migration team has

no idea what this new feature stuff is anyway so ultimately all code had to go onto the master and any difference in behavior between the old and the new stack had to be done through

configuration or code path changes based on environment cement we had a single docker image for a particular the code and we expected people to deploy it to both of the stacks so the

next thing after that was about deployment how are we going to do the deployment if you have completely separate deployment mechanisms you will end up with inconsistencies because

someone forgot to do both deployments either accidentally or deliberately cuz they’re focused on what they want to achieve with 150 services you can’t see it a glance whether that’s happening but

running a single deployment pipeline is risky because you really don’t want to block an emergency deployment for a live issue because your new and experimental stack had a problem so what we actually

did was we had two pipelines that were kicked off by the same single action which was attacking a release in github the last step step of both deployments was manual someone had to click about

and say I want this to go live and we just wrote a script that we ran manually we didn’t get around to automating it that just said what are the versions of the services in both environments and

then one of our developers around every morning they would tell people where they hadn’t actually finished their deployment the approaches we chose meant we were deploying functional changes

into both of our stacks and we were deploying things to production when it was really only a change for running keeping at ease and we didn’t want to double the amount of testing that we did

it was testing Mike reservist is quite can take up quite a lot of time anyway so basically what we did was prioritized the live stack that’s what we cared about and we relied on automation to

help us reduce the risk so we already had a service that would monitor when content was published so someone publishes an article it would monitor to make sure that article made it through

to all of the necessary data stores and api’s so we just ran that in both stacks and then we added a small service in our livestock that just forwarded all the real live production publishers to our

other stack so we’d know if there was any difference between the two was it failing to publishing one of the two stacks well there’s probably something wrong with our configuration or our

routing