Upgrade of troubled Phoenix could risk yet more payroll snafus for public servants

The Liberal government is planning a $22 million upgrade of the Phoenix pay system’s software that the late auditor general Michael Ferguson warned could pose a “significant risk” and trigger a new wave of pay errors for Canada’s public servants.

The plan is the latest in a nationwide scandal over a system proposed by the former Harper government that was supposed to save money. It was launched by Prime Minister Justin Trudeau's Liberal government in 2016. But since then, the government has spent about $1 billion and counting to fix Phoenix, leaving thousands of public servants underpaid, overpaid or not paid at all. Nearly all public servants have been somehow affected.

Senior public servants at Public Services and Procurement Canada (PSPC) acknowledge that updating the software, which the troubled Phoenix is built on, is a “challenge,” but the benefits far outweigh the risks.

“I stay awake nights more worrying about not doing the upgrade than I worry about the upgrade,” said Gini Bethell, PSPC’s assistant deputy minister overseeing the plan in an interview.

“For me, that was the bottom line. I kept balancing the two and the benefits of an upgrade came out way ahead all the time. “We cannot be not supported. Period. End of Story.”

PSPC has put together a two-year implementation plan to upgrade the software

PeopleSoft 9.1 to PeopleSoft9.2. For many, the risk lies in the fact that the myriad customizations of the Phoenix system were done under version 9.1.

The final plan is scheduled to go to the government’s Enterprise Architectural Review Board for approval April 4.

Massive backlog already

The upgrade of the highly-customized system comes when Phoenix is still struggling with a massive backlog of files and new errors crop up to botch public servants’ pay cheques.

Some critics say the upgrade is risky and fear it will make matters worse; others question the need and some simply can’t understand why the government is willing to take the risk.

“On the balance of probabilities, I think the smart decision is to not attempt an upgrade at this point in time,” said one technology expert.

In his last media interview before his death in February, Ferguson told this journalist that such an upgrade could trigger a new onslaught of errors and problems.

He said the system is unstable and with all the customizations, an “upgrade would be a very complicated piece of business.”

“If they have to do that upgrade while (Phoenix) is still having problems, that’s a significant risk. I think…one of the next points in time when they could have real problems is whenever they have to upgrade Peoplesoft.”

Need to keep Oracle software company support

PeopleSoft 9.1, which underpins the Phoenix pay system, is an off-the-shelf software developed by technology giant Oracle. It has since been updated with a new version, PeopleSoft 9.2.

PSPC began building Phoenix nearly a decade ago. IBM was hired to design and build it, using PeopleSoft 9.1 as the base for the government’s complicated pay regime with 80,000 pay rules and 105 collective agreements for 300,000 employees.

Oracle notified clients in 2015 — before Phoenix went live — that it would stop supporting that version by the end of 2018.

But Bethell said Phoenix can’t risk losing Oracle’s expertise and customer support.

Without it, the government would be on its own to handle any critical problems, defects or bugs. It also wouldn’t have access to Oracle’s improvements or regular updates on legislative changes of the federal and provincial governments, such as tax tables, needed to pay employees properly.

“We can’t be in a position of not having support from Oracle,” said Bethell. “We have to have their code changes to pay people or we will have a worse problem on our hands.”

PSPC put together the implementation based on “all the lessons learned” from the disastrous “big bang” rollout of Phoenix three years ago, Les Linklater, PSPC’s associate deputy minister responsible for fixing Phoenix, said in an interview.

Unlike Phoenix, the upgraded system will face end-to-end testing, trials and pilot projects to iron out all the bugs before used to pay employees. The original and upgraded Phoenix will run “in parallel” so there will be a backup system.

“It’s all about planning, testing, piloting and course correcting as required,” said Linklater.

Risk rooted in complicated customizations

In his report, Ferguson found Phoenix was heavily customized with more than 200 “custom-built programs” added to handle thousands of pay rules and connect to a mishmash of 34 different human resources systems that feed information needed for workers to get paid.

Ferguson expressed worry that rewriting all that code for the upgrade was fraught with risks that would create more mistakes in pay.

The complexity of federal pay is a root cause of the calamities that have dogged Phoenix.

Technology experts have long argued the massive number of rules and processes, which had to be customized, should have been reduced and simplified before Phoenix was built. They will complicate the upgrade and remain the biggest challenge in building a new system.

But Bethell said the upgrade won’t be as risky or costly as some fear. That’s largely because the software was adapted for federal pay rules and processes by adding ‘bolt-ons’ rather than rewriting the basic code.

She said only about four per cent of the hundreds of customizations will have to be retrofitted. As a result, most of the $22-million budget will go to testing rather than rewriting code.

“The original team did the right thing,” she said. “If they had to customize Peoplesoft to do something special to work for the government they did it with a bolt-on instead of going in and changing the coding of the core.

“If we had made all those changes within the code itself we would looking at whole different situation here.”

But some industry experts say the government is downplaying the risks. They argue PSPC would be better off forgoing the upgrade, maintaining Phoenix and adding tax tables on its own.

One long-time IT pay and process expert said Phoenix wasn’t tested properly before launch because the government didn’t have the expertise and questions whether government has the skills now.

“I’d like to be optimistic but I think they are digging the hole deeper. ..They aren’t really making progress and they still have errors. It makes you wonder why they are doing this.”

Was upgrade left too long?

The government has also come under fire for leaving the upgrade so late.

Ferguson chided PSPC in his 2018 report for not having an upgrade plan when it knew Oracle was discontinuing its support. He said an upgrade is standard management for IT projects; takes massive preparation and failing to plan for one “put the system’s long-term viability at risk.”

Bethell said the upgrade should have been done sooner, but Phoenix was unstable and making too many errors to take it on.

Oracle has agreed to extend its support for PeopleSoft for 9.1 for 2019 while the PSPC plans the upgrade. The government is now seeking a further extension of two additional years to implement and test the new version with the customizations.

When completed, the government will have an upgraded system that will be fully supported by Oracle, at least until 2029.

That gives PSPC a lifeline of another decade or more to ‘stabilize’ Phoenix; clear the backlog and clean up pay records to transfer to a new replacement pay system that Treasury Board is working on.

That leaves some questioning whether the government is hoping Phoenix can be fixed and not replaced.

“If it is stabilized, functioning and running well and upgraded doesn’t that beg the question why you are willing to buy a new one when you could run the risk of Phoenix part two,” said one technology expert. “Why go out and buy a new one. Isn’t that a waste of money?”

Light at the end of a ‘nightmarish tunnel’

Unions say the government is sending mixed signals about how fast it intends to replace Phoenix. It’s racing to come up with a short list of vendors with plans for a pilot by the summer. On the other hand, there was no money in last week’s federal budget to build for a new system.

Debi Daviau, president of the Professional Institute of the Public Service of Canada (PIPSC), said she was alarmed the 2019 budget didn’t fund a Phoenix replacement. The budget’s commitment to a new system is “not firm enough to leave me comfortable.”

“The absence of any major funding to ensure it happens without delay is a serious disappointment,” she said. “Our members need to know there is light at the end of this nightmarish tunnel. The government is still paying for the mistakes of the past rather than investing in a fix for the future.”

Bethell said Phoenix will be used for at least five to seven years, perhaps longer if the new system is not ready. She said the government is committed to a central system that integrates human resources and pay - which Phoenix does not.

The $22 million for the upgrade was buried in the $431 million in funding PSPC received for 2018-19 to stabilize Phoenix. The government has already spent more than $1 billion to fix Phoenix — on top of the $309 million that was spent to build Phoenix and set up the central pay centre in Miramichi, N.B.

A 2018 report by the office of the comptroller-general estimated Phoenix could cost as much as $3.5 billion if it took five years to fix.