Varied PCI QSA assessment quality causes PCI compliance issues

At a recent PCI DSS user group meeting, members said inconsistent assessments from poorly trained QSAs are resulting in numerous PCI compliance challenges.

If one QSA's advice differs from another's, get him to explain his reasoning. And then judge which meets the intent of the standard.


Simon Sharp
Director, Illumis Ltd,

Poor and inconsistent advice received during QSA assessments are creating difficulties for project managers working to make their companies compliant with the Payment Card Industry Data Security Standard (PCI DSS).

Members of the U.K.'s PCI DSS User Group vented their frustration at the last meeting in London on May 27, saying that Qualified Security Assessors (QSAs) often provided conflicting advice, even when they worked for the same company, causing serious PCI compliance issues.

"We had one assessor approve some of our compensating controls. Then he left and the next assessor from the same company said the controls were completely inadequate," said a project manager working for a large publishing company.

Others complained that their consultants often lacked relevant experience and had merely undergone a short training course that qualified them to carry out nothing more than a box-ticking exercise, rather than offering valuable PCI compliance advice.

The PCI DSS User Group, which meets regularly to exchange ideas and share information, includes representatives from both the private and public sectors, with members from most segments of industry. Few claimed to be close to reaching compliance with PCI DSS, even though some have already been running projects for more than five years.

One reason cited for the poor rate of progress is inadequate training of many QSAs. Despite recent moves by the PCI Security Standards Council (PCI SSC) to introduce more stringent controls over who can practise as a QSA, user group members said there was little evidence of improvement based on their recent experiences.

"QSAs are still very variable. The quality and continuity are just not there in some of these consultancies," said another programme manager who asked not to be named.

To combat the lack of consistency, some members said merchants should always ask for any decisions to be validated by the consulting company rather than the individual PCI QSA. That way, the advice is more likely to be reviewed by other members of the consultancy and, therefore, be more consistent.

But the PCI SSC is promising imminent improvements. Bob Russo, the organisation's general manager, told that a quality assurance (QA) programme for individual QSAs, introduced more than year ago, is already bearing fruit and helping to raise PCI QSA assessment standards.

"The merchants and acquirers are telling us there's a big difference from what they saw two years ago, and they like it," Russo said.

He admitted that QSA assessments may be "more of an art than a science" and that it was inevitable that QSAs might hold differing opinions about certain details. But he added that a recently introduced QA programme for QSA firms would increase consistency, and ensure that companies manage their own QSAs better. Also, the introduction of version 1.2 of the standard last year set out new quality assurance requirements and a scoring matrix aimed at preventing QSAs from cutting corners and ensuring they produced a thorough report on compliance (known as the ROC).

The scoring matrix forces the QSA to cover all elements of the standard, and is designed to allow the SSC to ensure a level of quality assurance over the work done. QSAs also now need to be recertified every year.

Those QSAs who fail to make the grade are put into remediation -- a process to help them mend their ways -- and are not allowed to practise again until they can demonstrate adequate improvements.

Merchants choosing a QSA can view the status of any QSA on the SSC website. The site is updated weekly and displays in red text any QSAs undergoing remediation. A handful of U.K.-based assessors are currently in that position.

One admirer of the new QSA quality assurance process is Simon Sharp, a director of newly-formed Illumis Ltd., a two-man consultancy based in Cambridge. Having just gone through the approval process to become a QSA, he said it was stringent and that the scoring matrix ensures QSAs do a good job.

His advice to merchants is to check that their QSAs have experience relevant to their respective businesses. He also said that consultants should have their work reviewed by their peers before providing advice to clients.

"You need more than one set of eyes on the problem," Sharp said. "Compensating controls are a sticky subject, but the merchants need to look at the intent of the requirements, and not just ticking the box. If one QSA's advice differs from another's, get him to explain his reasoning. And then judge which meets the intent of the standard."

He added that merchants need to have the right person running the compliance programme. "Compliance is a business and a technology project," Sharp said, "but the programme manager needs to be able to understand the technology."

Return to the PCI learning guide.

Read more on Regulatory compliance and standard requirements