The Saboteur timed things to perfection.
The company had been fiendishly toiling towards their first major product upgrade in nearly 18 months. Thousands of man-hours had been invested in designing, developing, and testing the tens of new features and hundreds of bug fixes contained in the release.
This was a big deal. Competitors had stolen a march on the firm. Their products looked like shiny new Ferraris lined up against the firm’s dented and smoking Moped.
Customers had been ditching the product, or threatening to.
Senior management had promised the Earth.
A make or break, bet the company, release.
The deadline for locking down the final release candidate was the close of business Friday.
Today.
Now.
To make the cut, features must have been checked in, successfully tested, pass a regression test, and signed off.
Tensions ran high.
Build and test servers were groaning under the load.
Developers were exercising parts of their vocabularies that would make a sailor blush.
Testers were being squeezed under insane amounts of pressure to sign off features.
Feature managers were running around like headless chickens, shouting at everybody, desperate to have the features they were accountable for included in the final build.
Two project managers got into a fistfight, before the CTO grabbed both by their collars and bluntly informed them they would be thrown out the window if they didn’t behave! Nobody within earshot doubted this was a promise, not a threat.
The last six weeks had been a death-march. Everyone working insane hours, tempers fraying, people sleeping under desks, surviving on pizza and red bull, while totally vanishing from the lives of their friends and families. The building reeked of stress, body odour, burnt coffee, stale fast food, and the lingering stench of the kipper that a thoughtless business analyst had microwaved for breakfast three days ago!
The CEO had decreed it would be instant dismissal for anybody who left before the final build was locked down. This wasn’t a threat either. None of the senior management team had been home in four days, all rolling up their sleeves and pitching in with testing, troubleshooting, or ensuring the troops were fed and caffeinated.
Once the build was locked down however, he would personally buy the whole team as much as they could possibly drink, and pour them into taxis to get them safely home afterwards.
Usually a Friday night deadline really means sometime before Monday morning.
This Friday was different.
The England v Columbia game at the Football World Cup was kicking off at 20:00. To a person, the staff were desperate to escape to the pub downstairs, watch the game, and blow off some steam.
Amidst all the chaos the Saboteur patiently waited.
At 19:40 the last of the major components was signed off. A cheer loud enough to shake the windows echoed across the open plan office. All that remained for the build to be locked down was a clean run through the regression testing suite.
The build team started labelling up the thousands of different pieces of code that would collectively make up the final build. Suddenly finding themselves with nothing to do; the managers, analysts and developers started drifting over to congregate around the build team’s desk.
The tension in the air was palpable, yet beneath it was a glimmer of hope, and an undercurrent of excitement the likes of which children experience on the last day of school before the summer holidays. The end was in sight!
The Saboteur calmly walked across to the unlocked computer of the lead Database Administrator, who had “god” priveledges to virtually everything. He quickly copied a small generically named file into the build directory, just one more amongst hundreds of pieces of code that had been checked in over the last couple of hours. Then he closed the application window, and joined the crowd.
Labelling complete, the regression tests were kicked off.
Unlike the movies, there is no animated progress bar. No dramatic musical score building to a crescendo. No rapidly scrolling screens full of unintelligible computer code. Just silence.
They waited.
Anticipation built.
And waited some more.
Time seemed to stand still.
Nobody dared speak.
Somebody farted.
At 19:53 the disembodied voice of Homer Simpson suddenly shouted from a computer speaker: “Woohoo!”
The test successfully completed. The build worked. Their work here was done!
There were high fives and hugs all around, before a stampede for the exits and the pub downstairs.
The Saboteur joined them.
The firm’s largest client was a startup, that had experienced massive growth over a very short space of time. It had recently floated on the stock market, and was touted by the media as the next big thing.
The client’s management loved seeing their names in the press. They spent more time pumping up their own tires giving interviews, than they did building the business.
The client had never turned a profit. They had a burn rate that would give even the most seasoned creative accountant nightmares. Their operations ran at near capacity, in a hugely competitive industry where the only real differentiator was branding. There were razor-thin margins, and a cash flow buffer that would be exhausted in just a few weeks should anything go significantly wrong.
For several months there had been rumours that the client would be acquired by a larger, though equally unprofitable, competitor.
The stock price had surged on the back of those rumours. The industry was in the midst of a feeding frenzy of such mergers and acquisitions, so it was certainly possible.
The rockstar research analysts at most of the large investment backs rated the stock a Strong Buy, encouraging their account holders to invest their hard earned savings into a company that was surely headed sky high!
Behind the scenes, there were rumblings at any given time the client was at best one billing cycle away from a terminal cash flow crisis. The investment banks knew this for certain, for they had prepared the prospectus and conducted the IPO for the client after all!
The Saboteur knew that precarious cash flow position, the egotistical incompetence of its senior management, and the pending takeover left the client ripe for a spot of blackmail.
The Saboteur had learned that the client operated a rolling 30-day backup policy.
Every night at 02:15 the production systems were backed up to tape.
Each morning at 10:00 a courier would collect the backup tape, and see it safely to the offsite secure storage facility located two hours away on the outskirts of Bristol.
The courier would drop off the old backup tape from 30 days ago when he arrived, for reuse that night.
This meant that 31 days after the firm’s major software release had been applied to the client’s systems, they would have no way of backing it out or recovering to a point prior to upgrading.
That was the moment the Saboteur had been waiting for.
At 10:00 on the 32nd day the client’s billing engine switched off.
Support staff and techies investigated, but were left scratching their heads in bewilderment.
Time passed.
The volume of unbilled transactions grew. Wholesale charges on those transactions had already been incurred by the client.
By day 33 fingers were being pointed and accusations made. The client’s senior management shouted that the firm was responsible. The firm’s senior management retorted that it didn’t matter, the client was accountable. There was much shouting and threats of legal action.
If the transactions could not be processed by day 35 then the client’s customer bills could not be generated. They would have no way of paying their wholesalers, and face a cash flow crisis within weeks.
Day 34 saw rumours start swirling: “something was wrong at the client”. The stock price sagged. Scheduled media appearances by the C-suite were cancelled. The prospective acquirer started to ask “please explain” questions.
Only now did the Saboteur act.
An anonymous letter was delivered to the client CEO. It stated something along the lines of:
- The billing engine is unable to process transactions after 10:00 on the 32nd day.
- A life-changing sum of money will procure a special license key to unlock the billing engine.
- Countermeasures to detect whether the system clock was tampered with were activated.
- Countermeasures to detect if the system were moved to another server were in place.
- You have until 17:00 today to make payment.
The firm’s CEO wanted to go to the police. Extortion was a crime, and he was both shocked and outraged that his company’s core product had been weaponised this way.
The client’s CEO and COO adamantly refused.
They knew how precarious the financial footing of his firm actually was.
They had been walking a fine line: keep the tap-dance going until the takeover was completed and they would be very rich; or fail and face the prospect of bankruptcy and jail for potentially fraudulent accounting shenanigans during the client’s IPO.
It doesn’t get much more accountable than that!
As the client’s senior management raided their employee pension fund to meet the extortion demand, their CFO suffered a heart attack… but survived.
The extortion demand gave had given the techies some clues where to look, but there was insufficient time to identify and remove the trojan code before the deadline expired.
The Database Administrator had understandably taken it personally that someone would use his credentials to implant the trojan code. Nobody seriously believed he would have been stupid enough to use his own login to plant the time bomb.
He stepped back and thought about what system clock and server characteristics countermeasures must have in common.
Then he had an epiphany!
The main function of his job was to defend the firm’s systems from rubbish code written by incompetent developers. Good programmers write immutable tests, the answers to which are black and white: they are either true or they are false. There is no grey area.
Therefore any monitoring algorithm the Saboteur had implemented would most likely use that same good programming pattern. Did the server characteristics change? True or False. Was the system clock tampered with? True or False.
He delved deeply into the guts of the server’s innermost workings. Somewhere way down low, below the operating system and the device drivers, was a definition of what False actually meant.
The Database Administrator located it. What would happen if instead of “False” returning a value of false, he simply changed “False” to return a value of true?
With nothing to lose he gave it a try.
The billing engine started up.
The transactions started processing.
It worked!
The Database Administrator’s hack was most definitely not a long-term solution. It would undoubtedly cause many problems. However it did allow the transactions to be processed, and the customer bills to be produced. The client would live to fight another day.
In the end the Saboteur didn’t received any money.
The police were never called.
The customer bills went out on time, and the market rumours of commercial difficulties were dismissed as baseless gossip.
However, the takeover fell through, the potential acquirer got the yips.
The client CEO was soon back in the media, but the markets were never again entirely convinced by his performance. There couldn’t be smoke without there also being fire, right?
The client went bankrupt several months later.
The firm followed suit shortly afterwards, the client’s unpaid bills ensuring their insolvency.
The losers in this story were the employees and shareholders. They each believed the promises made by senior management, and invested their livelihoods or hard-earned savings into them.
The Saboteur suffered no real consequences of note. The firm eventually determined who had inserted the trojan code, but as they had left the company (and the country) shortly after England’s 2-0 win over Columbia in that fateful football game, there was little that could be done.
The Saboteur’s stated reason for leaving: unreasonable work-life balance and managerial bullying.
Sabotage, espionage or subversion
I was reminded of this incident when I read Bloomberg’s recent investigative article about many leading technology firms (and the US government) apparently falling victim to a diabolically clever supply chain attack.
For those who haven’t followed the story, trojan computer chips were apparently added to 10,000s of circuit boards produced in offshore factories by third-party suppliers. The compromised chips permitted a remote bad actor to access and/or control the cloud-based computers in which they had been installed.
The trojan chip interacts with the lowest layer of a computer’s processing, right down at the 1s and 0s level. All the security roles and permissions, encryption, and so on sit several layers of abstraction higher up the technology stack. This means they are potentially circumvented because these protections sit on top of the processor that itself is compromised.
Why bother trying to hack individual apps, databases or passwords one at a time when you can compromise all of them at once? Regardless of whether the motive for the attack was sabotage, espionage, “snooper’s charter” style monitoring, or subversion; this was the true genius of the attack.
A massive fail
If the integrity of the hardware that our applications run on cannot be trusted, this means no firm can reliably meet their compliance obligations with regard to information security and data privacy.
Everything from tax return processing, to the email and messaging platforms we use, to internet banking now run on cloud-hosted infrastructure.
That has potentially huge implications for firms who remain accountable for the data that they possess and process.
Data breaches: who is to blame?
Consider the following “plucked from the headlines” scenario. Your personal private data gets leaked onto the internet. Perhaps it was the details of your rehab stint nobody knows about, or your secret Grindr profile, or career limiting fodder for a #MeToo vigilante mob.
Who would be responsible?
- Is it the firm you supplied your data to?
- Is it the cloud-hosted provider, from whom the firm rents infrastructure capacity?
- Is it the hardware vendor who rents the kit to the infrastructure provider?
- Is it the component supplier who sold the circuit boards to the hardware vendor?
- Is it the engineer who originally designed the plans for those circuit boards?
- Is it the offshore manufacturing subcontractor who physically assembled the circuit boards?
- Is it the individual factory worker who attached the components to the circuit board?
At each link in the supply chain responsibility has been delegated downwards.
Accountability can’t be outsourced
The accountable party is easier to identify. The data privacy regulations pin the blame on the firm we supplied our data to. They were responsible for ensuring it was safely stored and securely transmitted, regardless of how many layers of outsourcing, offshoring and subcontracting that may have been processing that data.
This responsibility versus accountability question also applies to our finances.
We can outsource the responsibility for preparing our tax returns to an accountant, yet it is us who remains accountable for any fines or legal penalties should those returns be incorrectly calculated.
Many folks appoint financial planners or wealth managers to be responsible for investing their money. It is the folks themselves who are accountable for the outcome of those investing activities however, as they reap the gains or suffer the losses that are incurred.
The same is true for our choice of banks, stock brokers, mortgage brokers, insurers, and lenders.
We can delegate as much responsibility as we like, but accountability can’t be outsourced.
youngfiguy 7 October 2018
Fascinating story and thought-provoking. You are an excellent story teller Indeedably!
{in·deed·a·bly} 7 October 2018 — Post author
Thanks Mr YoungFIGuy, very kind of you.
Interestingly Amazon and Apple both called bullshit on the Bloomberg article.
With a bit of luck this means the type of Trojan hardware attack described in the article hasn’t physically occurred… yet… that we know of!