IWOOT says that the technique has helped it to cut testing time from days to hours, allowing it to update its site quickly without risking serious errors that could damage sales.
The company is working with US-Israeli firm, uTest, which runs a community of 18,000 testers, who compete online to find bugs in return for cash prizes and kudos in the testing community.
IWOOT used crowdsourcing to test a major upgrade to its web payment system designed to meet the PCIDSS security compliance requirements for online retailers.
The company worked with UK testing consultants TCL, and US-Israeli company uTest, which has pioneered the crowdsourcing approach to testing.
TCL and uTest recruited a team of 45 testers with expertise in Internet Explorer and Firefox, to test a new version of the website, in return for cash payments for each bug they found.
In just over two hours, the testers collaborated online to find and document serious bugs that could have disrupted sales on the site, said Sagar Vadher head of IT at I Want One of Those.
"We had four people testing internally. They all said the new site looks great. Then we had 45 people testing externally saying they had found 30 bugs," he said.
"If there was anything wrong with the checkout and we put it out live, we could lose a lot of money. It is simple to make sure the checkout works, but it is a very important thing."
The testing session cost IWOOT around $500, a fraction of the cost of employing inhouse testers, said Vadher.
TCL gathered four teams of volunteer testers to put the software through final testing a week later. The teams took part in a one-hour race at La Piazza restaurant in Birmingham to find and document bugs, in return for pizza, beer and a prize for the winning team.
The retailer plans to use crowdsourcing to test the next major development of its website in August, which is due to go live for Christmas.
"It is going to have quite an impact on how customers are using the site. We need to have a very high level of performance testing. We would not be able to that in a short space of time using traditional testing, so it is going to be Utest," said Vadher.