Government departments and businesses face increased scrutiny into the way they collect and use personal data if the government adopts the recommendations from a House of Lords report published today.
Lord Goodlad, chairman of the House of Lords' constitution committee, said the huge rise in surveillance and data collection by the state and other organisations risked undermining the traditions of privacy and individual freedom which were vital for democracy.
The committee spent nearly a year studying the "pervasive and routine" electronic surveillance and collection and processing of personal information.
Its report, Surveillance: Citizens and the State, follows an earlier report from the information commissioner on the rise of " the surveillance state".
The committee looked into government data collection schemes such as the National DNA Database (NDNAD), the NHS Spine for electronic health care records, the children's database, and the national ID card and government plans to share data more between departments.
"If the public are to trust that information about them is not being improperly used, there should be much more openness about what data is collected, by whom and how it is used," Goodlad said.
"There can be no justification for this gradual but incessant creep towards every detail about us being recorded and pored over by the state."
The report criticises the rise in surveillance and data collection by the state and private sector. The lords said it "risks undermining the fundamental relationship between the state and citizens, which is the cornerstone of democracy and good governance".
The Lords made 44 recommendations to protect individuals from invasions of their privacy. They said privacy is an "essential prerequisite to the exercise of individual freedom".
The executive, government agencies and other public bodies should use "restraint" in the use of surveillance and data collection, the report said.
The Lords said the UK led the world in the use of CCTV to watch citizens, with some four million cameras in use. It also had over 7% of its citizens - more than any other country - on record in the National DNA database.
The report said the DNA database potentially infringed civil liberties and could be used for "malign purposes" in future. The government is expected to comply quickly with the recent European Court of Human Rights decision that the DNA profiles of non-convicts be deleted from the NDNAD, the report said.
The Lords said the public is often unaware of the vast amount of information about them that organisations hold and share. They said government and local authorities should help citizens understand the privacy issues for themselves and society that may result from this.
They wanted central and local government to work with the Information Commissioner's Office to raise public understanding of the issues.
The committee called local authorities' use of surveillance powers under the Regulation of Investigatory Powers Act to spy on the public over issues such as littering and to check residential status over school applications "a clear misuse of power".
"The information commissioner should now be given the powers he needs both to investigate private sector organisations' use of personal data in the same way that he can for the public sector, and to have more involvement in scrutinising moves by the public sector to expand its surveillance activities," he said.
"Companies that refuse access to the information commissioner may well have something to hide," they said.
Responding to the report, a Home Office spokeswoman said the government had been clear to balance the interests of the state with those of people on privacy.
"The government has been clear that ... surveillance or data collection should only be used where it is necessary and proportionate," she said.
She said CCTV and DNA were essential crime fighting tools that had revolutionised police investigations. "The home secretary has already set out new common sense standards for use of investigatory powers and retention of DNA profiles and has announced a consultation on the use of the Regulation of Investigatory Powers Act to open a reasoned debate about all these issues." she said.
Main Lords' recommendations requiring judicial oversight of investigations under the Regulation of Investigatory Powers Act 2000 (RIPA), which allows access to internet, e-mail and telephone data:
Compensation to be paid to those who were watched unlawfully under RIPA
Reconsider whether local authorities should use RIPA powers, and if so, under what circumstances
Change the Data Protection Act 1998 to force government departments to produce an independent, publicly available privacy impact assessment (PIA) before they adopt any new data collection or processing scheme
Information commissioner to vet and approve PIAs
A new bill should replace the existing rules that govern the DNA database with great oversight
DNA samples given voluntarily in investigations should be deleted once an inquiry is closed
Legally binding codes of practice for all CCTV schemes
Set up a parliamentary joint committee on surveillance and data powers of the state
Any new legislation that would expand surveillance or data processing powers should be scrutinised
Information commissioner should have the same powers to inspect private sector organisations as it has for the public sector.