The former GP, who now advises the information policy unit of the Department of Health, made his comments at a "breakout" session during a conference run by the British Computer Society's Health Informatics Committee.
The adviser cannot be named because the organisers gave delegates at the conference an assurance of anonymity to encourage open discussion.
Working groups at the conference looked at aspects of the £2.3bn national programme for IT, with the aim of giving constructive feedback to Richard Granger, the head of the programme.
The adviser admitted that sensitive data could be the subject of attacks from malicious hackers, but said there were risks with all database systems.
"This question of having data in a national database is a 'so what?' matter. All our personal information is on all sorts of national databases, from MI5, police databases, Inland Revenue databases and social security databases downwards," he said.
"Yes, occasionally they get hacked into. We all accept that and we strive for better systems."
The adviser surprised critics of the patient records programme, called the Integrated Care Records Service, by attacking them in a raised voice during a working group.
"Can I just break in here because I am sick to death of listening to all this doom and gloom," he said.
"If this group, of all groups, cannot take a positive view of the national IT programme - we all have a deep interest in the outcome - there is no hope for the programme."
His comments were regarded by some at the conference as representing the views of many senior figures at the Department of Health.