Tuesday, March 20, 2007

When Offshoring Your Development Team Means Buying a Boat

Play word association with most developers and the response to 'offshore' will most likely be a city in India. Starting my career in the oil and gas game meant offshore development has an entirely different connotation. Let me share my first experience as an offshore developer.

Offshore -- meaning 'out at sea'

Lesson 1: The sole developer of untested, mission-critical software, should expect to be going wherever the aforementioned software goes.
Sure you chose the oil and gas industry, but you joined as 'the lynch pin of a new technical department'. Sick of relying on external resources for their software, you were hired to write some for them.

By the time you realise their office isn't exactly in Portland, you're sitting in a 10' sea container floating in the middle of the Indian Ocean pressing a button every 108 minutes.

What three things would you bring to a desert island?

You're going 150km (95mi) offshore for 3-6 weeks where there's no broadcast TV, no radio, no phones, and absolutely no internet. There is a satellite phone, so if you're ready to pony up $1 / second you can reach out and touch somebody.

If the boat's orientation is right (and the stars are aligned) there might be a nightly email uplink when you can send / receive today's emails on a public, unsecured PC running Outlook Express. Nothing larger than 50kb and absolutely no attachments.

What to bring? The following are verboten : alcohol, women, drugs, explosives. The challenge? Your total allowance is 10Kg (22lbs) which includes a hardhat and steel-capped boots.
Note: If it's a decent sized job they'll do your laundry nightly, so a few clothes go a long way; just remember they wash them at around 150C, so don't bring any delicates. Or nylon. Or dark colours.
3 pair coveralls (pre-washed and worn to avoid that n00b look), hard hat, steel tipped safety boots, sandals, underwear, socks, t-shirts, and toiletries (the latter will last longer than you expect. Don't think about why).

Then the essentials:
  • Notebook computer, iPod, PSP.
  • Three weeks supply of candy.
  • Large collection of TV episodes and movies.
You mobilise from Karratha, a remote but booming resources town, and no-one but fellow but oil & gas workers are flying there at 6am on a Monday morning. Most smoke, and some look like they've prepared for 6 alcohol free weeks by drinking for the last 24 solid hours.

Three hours on the plane, then straight for the heliport. You catch a chopper out -- with any luck this won't be opportunity to put your HUET to the test -- and so far things are pretty sweet - even if the music piped in through your headset is less 'Flight of the Valkyries' and more 'Best of Tracy Chapman'.

No - that's not a fire exit. Questions? No, there is no fire exit.

You're in a part of the world where there are two seasons, hot / wet and hot / dry. Today it is definitely hot, and the humidity must be 150%. Perfect beach weather, but you're changing in to full length coveralls, boots, and a hardhat ready for your vessel safety induction.

Years of StarTrek means when you're told to 'report to the bridge' you have a brief Wesley Crusher moment. You're shown your muster station and which alarm means to head there -- and which one means you're better off going straight over the side.

You also get assigned a room about the size of a smallish walk-in wardrobe that you'll share with 7 others, each group of 4 on opposing shifts. You learn that your communal death trap room adjoins the engine room and there's only one entrance / exit -- up the stairs. If the stairs are burning you have no chance to survive make your time.

You glance sidelong at the DVD case when you hear the bridge crew talking about the scourge of piracy, but relax when you realise they're talking about actual pirates... until you realise they're talking about actual pirates.

Your induction moves on to the mess (where posters like this do little to quell concerns), the galley, the smoking room, and the rec room. More often than not, these are all the same room. The rec room has the TV and DVD player and houses the ship's library - filled with all the classics: Playboy, FHM, Barely Legal, People. The DVD selection is more mundane but all have foreign covers and a distinct homemade feel.

Next, before anyone can even think about heading to bed comes the 24 hour living nightmare known as 'mobilisation'.

Just like a LAN party, but with more water and fewer pizzas

You're building Windows boxes, installing software, and fixing bugs. Then you're stringing CATV and 440V mains leads around the back deck. Telemetry from the ROV has to be fed to survey on the bridge, then back down to you in your inspection shack, so you've got the RS232 breakout box in full effect. Four live video feeds need to be recorded onto four digital video recorder PCs, a frame grabber on the online inspection PC, the bridge, and also backup VCRs.

There's cables everywhere. Access to everything is awkward, and there're always three people trying to occupy the same piece of space, and work with the same piece of hardware. All this is happening on the back deck of a support vessel as it ploughs out to the field at full steam in 2m seas and 24knot winds, with green water coming over the back deck.

They paid for a qualified inspection engineer. Son I am disappoint.

Hourly cost of support vessel and crew: $20,000. Hourly cost of ROV crew and vehicle hire: $10,000. Look on your party chief's face when you tell him the software's crashed and will take 30mins to bring everything back up so everyone can get back to work: Priceless.

Lesson 2: Mission critical means if the software goes down, the job stops. When the job stops the company responsible starts footing the bill. Feel the burn.

You're a programmer, you know this and so does your employer. But the client paid for an Inspection Engineer and expects to get what he paid for.This would be less of a problem if your employer didn't insist on you doing both jobs, or if inspection shifts were less the 12 hours long, or didn't require your full attention.

Now, rather than being the cutting-edge developer on the front lines you're a barely adequate inspection engineer who seems more interested in the software he's using than the job at hand.

Don't try this at home

The job runs 24/7, no weekends, no downtime. You can simulate the daily offshore experience at home:
  • Put an uncomfortable plastic chair in your bedroom closet where you've hung 6 LCDs.
  • Have 3 video cameras recording the bottom of an aquarium, each at a slightly different angle but looking at the same empty sand.
  • Park a big-rig next to the adjoining wall (and leave it idling).
  • Turn on a humidifier at least 110% relative humidity. Every 30mins alternate between a heater at 45C with an aircon at 15C.
  • Invite five complete strangers into your closet.
  • Get in, sit down and watch the screens. You need to give a running voice-over that describes what you're seeing, as well as logging the action in software. Don't worry if nothing's happening, just say 'The seabed continues to be flat and featureless. No sign of debris.'
  • Have someone lock the door and don't let anyone out for at least 12 hours.
  • This is important, have your friends rock the wardrobe by about 30 degrees side-to-side continuously. Randomly have them drop it.
  • Every 2 to 4 hours have something on the camera change slightly for no longer than 3 seconds. If you miss it, you lose.

At 7am the sun's up and you've done the handover to your replacement. After downing a cholesterol loaded deep-fried heart attack, you should be heading to bed. Instead you're front line support for the day-shift, fixing bugs, and implementing features. Around midday you have one last 3 course meal before wedging yourself in to bed so you don't plummet from the top bunk when the boat rolls 30degrees.

You get up at 5pm and head to the mess for breakfast; a quick look at the menu and you opt for the low GI staying power of pork spare ribs and mashed potatoes with gravy. Breakfast of champions. The ROV crew is on midday to midnight shifts, so there's another full dinner put on at 11:30am and another at 12:30pm. Play your cards right and you can get 5hrs sleep a night and still squeeze in 4 full 3-course dinner meals every. Single. Day.

Do I know you from somewhere...?

Three weeks in you realise that this three week job is not likely to finish in the next 24 hours. You email your loved ones explaining that rather than being home by the weekend, you'll be out here for another month.

Time passes.

At 5 weeks the homesickness has replaced seasickness. You're swearing like a sailor and you've not shaved since day three. It's been a month since you've seen your girlfriend (or any other three dimensional woman). There's so much porn being watched that people remark on being genuinely surprised they've never met any of the actors in real life. You've fixed everyone's notebook at least once, have challenged and defeated the entire crew in a series of all-in Quake3 Arena death matches. It's time to go home.


Last night everyone worked 24 hours to get the demobilisation finished, those on night shift (like you) are heading for 36 hours straight. Everyone is barely recognisable having washed, shaved, and put on their 'good' clothes for the first time in over a month. The flight back to Perth is a blur of G&Ts. For the first few days home you're walking with a swagger and when attending to nature you still lean forward and support yourself against the wall (to counter the non-existent swell).

You sit back and reflect. Your bank account certainly enjoyed your trip, and you will never in your life know your code as well as you do now. Was it worth it? No. Will you be going again..? I was back offshore within a month and did almost a dozen stints over 4 years. I've worked with people who went on one job, got choppered off after 3 weeks and quit the second they hit dry land. Your mileage may vary.

[Based on a much earlier submission to Everything2]


  1. I'm curious to how much you got paid for that hell?

  2. I was a permie at the time so I got an 'offshore bonus'. It effectively doubled my pay while I was offshore. Plus I got a day's paid leave for each weekend I was offshore. So for a 6 week stint you ended up with an extra week's holiday and double pay for the whole stint.

    When the weather got rough you'd huddle in your bunk and focus on how you were going to spend the extra cash...

  3. Anonymous5:46 pm GMT

    Where do I sign up? Seriously...

  4. Anonymous6:19 pm GMT

    Out of all the things you went through and which you described beautifully, only one thing I dread the most : "Fixing everybody's notebook at least once !"

    Congratulation on making it.

  5. Anonymous7:47 pm GMT

    what is the approximate salary?

    Do they take foreigners?

  6. The beauty of the oil & gas industry is that 'foreign' is relative. I worked in Perth (Australia) in a company full of Brits. Now I'm in England and keep bumping into Aussies heading to the North Sea...

    In truth there's not enough companies in the industry that send their developers offshore -- it's an exception rather than the rule.

  7. Anonymous2:15 am GMT

    Yes, can you please answer the question people asked multiple times, how much is the pay? Thanks.

  8. its probably about a thousand per day .. i got a friend who is smart as i but handles very large engines rather than very large chunks o code .... not a bad life but it is different

  9. [Meta: Moved the photos to a different host due to bandwidth issues.]

  10. Anonymous10:12 pm BST

    Hi Reto,
    I would like to say, reading your blog has brought a smile to my face. I have been stuck in central Guatemala for the last 9 weeks on a 2 week job commission a power station. My conditions are not half as bad as you had; at least I’m on dry land, but the living condition….man! Ever sleep in a hayshed (hotel); you got your usual friends, ants, mice, cockroaches and lizards. The toilet (outhouse) is a different experience; one of the locals got bitten on the ass by a rattle snake a couple of weeks ago, so every time I go in, I make as much noise a possible and waiting for the rattle. I don’t speak Spanish because my company said “it will not be necessary to learn you will only be there 2 week”, so I speak in broken English so the guys here can understand me. I just wanted to say thank you for sharing you have made my day. Cheers buddy.

  11. Good grief this made me laugh. I was in tears. Yes, it really it like that.

    If I had to pick one quote out it would be
    "Breakfast of champions"!

    I'm a geek but not a fully beaded c munching hacker; took an interest in Seismic Processing but HR blocked to the transfer :/ On the lookout for links.

    I guessed you worked all over.

    here's my spam mail - jago25_98 at hotmail.com

    It's good to talk.

  12. Anonymous8:55 pm GMT

    Thanks for writing this. My son said it would help me understand his job better and it does. It was entertaining and informative.

  13. love your write up. i was a qc on on a seismic boat, back then though we did 8 weeks straight so don't come crying round here with your 3-6 weeks!


  14. Great read, thanks for sharing!! :)

  15. Yeah, well written. I am not sure how you got over the fear of pirates though

  16. Good informative post....thanks for sharing.

  17. Well, the part "You're swearing like a sailor and you've not shaved since day three" is not much different from my current office job :-)

  18. wow!...indeed patience pays..and now who can ever say this same man at the sea, is the brain behind the sizzling android!!..:)...that's a motivation to me man!!!

  19. You're a true rockstar, programming on land, by sea.

    Though, I won't be inviting 6 strangers to my closet any time soon-- I'll just live vicariously through you.

  20. Great stuff. It puts my trips to relatively sane places like Qatar and Libya (up until Feb) into sharp perspective.