Alexey Fedorenko - stock.adobe.c
How startup Once.net and Cloudflare secured the 2023 Eurovision vote
When the Eurovision Song Contest introduced paid-for public voting from outside Europe in 2023, it faced new cyber challenges. Learn how Dutch startup Once.net and Cloudflare teamed up to secure and support the big night
With 64 contests, 67 winners, 101 live shows, 52 participating countries, 1,500 songs, 12 points, seven wins for Ireland, seven for Sweden, five apiece for the UK, France, Luxembourg and the Netherlands, The Eurovision Song Contest is as heavy on statistics as any sporting league. With viewer numbers regularly in the hundreds of millions, and an active and noisy fandom, it’s no surprise that the impact of the contest can be clearly seen on the internet.
As such, each year for over a decade, the contest organisers, the Eurovision Broadcasting Union (EBU) and its membership of national broadcasters, including the BBC, have been working behind the scenes with Cloudflare both to mitigate the impact of the sudden surges of traffic and to ward off the grim prospect of a distributed denial of service (DDoS) attack bringing the contest’s IT systems to their knees.
However, in 2023, for the first time, the EBU opened up audience voting beyond Europe’s borders, inviting voters from 107 non-participating countries to cast paid-for votes online, which were then tallied and added to the final scores as if an extra country – awarding the full spread of points from one through 12 – was participating.
This change created two new challenges for the Eurovision technology partners. First, they had to build an online voting infrastructure that could quickly scale to handle millions of demands in the very short voting window, which lasts barely a quarter of an hour. Second, they had to make sure that the process was secure from start to finish.
Enter Dutch startup Once.net, a firm that was actually first brought in by the Eurovision organisers for the 2021 contest, which took place in the Netherlands after a one-year hiatus due to the Covid-19 pandemic.
The 2021 contest took place with a limited set of Covid-19 restrictions enforced – live audience numbers at the Rotterdam Ahoy conference centre were restricted to approximately 80% of the venue’s capacity, and social distancing measures remained in place.
As a result of these unique needs, the initial engagement between the two partners saw the creation of an interactive Eurovision mobile app on Once.net’s Based.io platform, a project part funded through the EU Horizon 2020 programme.
Based.io is a scalable, live data platform capable of handling up to 400 million concurrent users. It consists of an observable, real-time graph database, a networking layer, cloud functions, analytics and infrastructure orchestration.
Moreover, according to its developers, since all system information, traffic analysis and disruptions are monitored in real-time, the platform can instantly respond to variable demand, enabling the user’s infrastructure to flex as needed during traffic spikes, network outages, or cyber attacks.
The original app that Once.net built served a simple purpose, it allowed audiences at home to contribute to the socially distanced atmosphere via a virtual applause feature; the more cheers (in reality, taps on screens) an act received during their song, the louder the applause they received in the arena after it. This went over very well, and the feature was brought back again for the 2022 and 2023 contests.
But, says Once.net founder and CEO Jim de Beer, almost immediately there was an issue: “We started to get all these DDoS attacks – very clearly tailored attempts. It was clear there was a team of people checking our infrastructure because everything was custom; it’s a new platform, it’s a new protocol that we built. So to attack it you have to have some knowledge or to have done some research in order to launch those attacks.”
These attacks – which reached their peak when Ukrainian folk electronica act Go_A (who were to place fifth in 2021) performed their song Shum – were limited in their size and scope, and were a long way from causing any technical issues that would have been noticed by viewers.
But later that year, when the same app was deployed to support the Junior Eurovision Song Contest – hosted by France and won by Armenia – these DDoS attacks ramped up, with tens of thousands of malicious or co-opted IP addresses involved, bombarding the backend infrastructure with hundreds of millions of junk requests.
Who would attack Eurovision?
Given the ongoing war in Ukraine and Russia’s exclusion from Eurovision and other international events, it is easy to speculate about who might want to attack it, where they might be, and what motivates them.
However, it is important to understand that firm attribution in a DDoS attack is much harder to do than in a ransomware incident. Cloudflare CTO John Graham-Cumming explains.
“We just don’t know because Cloudflare doesn’t do attribution. We don’t tend to try to figure out who the attackers are, partly because the very nature of DDoS attacks means they are distributed – they come from all over the place,” he says.
“So where they come from is very unlikely to actually be who is doing the actual attacking, so it’s hard for us to say. Eurovision obviously brings together a very large number of countries and it’s a very high-profile event, so there are all sorts of motivations for trying to knock something offline.”
Fortunately, the Based.io platform’s infrastructure was already being protected by Cloudflare, and those watching at home were able to enjoy the show as usual. Nevertheless, the team was a little spooked; De Beer describes the experience as “pretty scary”.
And when Once.net was asked to repeat the exercise in 2023 and to take charge of supporting the introduction of paid voting, De Beer and the team knew things needed to seriously change.
“We decided to go and invest a lot of time in rebuilding a big part of our infrastructure with security-first, because we were feeling the heat. This was serious,” he says.
In order to prepare for spikes in network traffic and demand, and determined cyber adversaries, De Beer needed a resilient, scalable platform that could also act as a mitigation layer in front of it. Cloudflare, with its deep historic involvement in Eurovision and its existing relationship with Once.net, was the logical choice, he says.
How Based.io works
The Based.io platform uses a client-side service discovery pattern as opposed to network or HTTP load balancers, which means it selects the most suitable server to connect to at any one time, using Cloudflare’s fast cache propagation infrastructure in order to cope with bumpy traffic.
Each of these servers continuously registers a unique access key that turns over ever 15 seconds, which must be used when a client connects. At the same time, the backend servers ping their health data to the service registry every 300 milliseconds, so when a client shows up it can request the healthiest server URL for that instant (and its associated access key) from a central registry and connect to it. The server will disconnect a number of clients should it become overloaded, who will have to move through the process again.
This central registry would naturally appear to be a bottleneck, and as such a prime target for an attacker. This issue is dealt with by putting it behind Cloudflare’s global network, with a cache time of three seconds. Because the system relies on the real-time health stats to distribute the load of incoming traffic fairly, and uses short-lived access keys, this cache must update quickly and without problems, therefore Cloudflare’s Tiered Caching option also comes into play.
Additionally, by removing load balancers, the platform lets clients connect to the backend servers through Cloudflare, ultimately meaning clients see better system performance backed by a more cyber resilient infrastructure – since the load balancers aren’t there, they can’t constitute an attack surface. Furthermore, connections can be better distributed using the real-time health data.
If the platform needs to scale, which as we shall see it did, this can be done automatically by deploying batches of up to 200 machines capable of handling up to 40,000 connected users apiece each reaching out directly to the Cloudflare API to configure its own DNS record and proxy status. Cloudflare’s high-speed DNS system means these changes can be propagated across the global network within seconds.
At a high level, clients – in this case, voters – arrive at the voting landing page, esc.vote. Rather than build this themselves and expend time and effort on owning, configuring and managing their own infrastructure, the Once.net team decided hosted this page on Cloudflare pages, which meant deploying was as quick as a commit to their Git repository, while reachability or scaling worries melted away.
As an additional barrier to address the potential problem of fraudulent or bot voting attempts to rig the contest, De Beers also deployed Cloudflare Turnstile to protect the payment API endpoints that were being used to validate the vote. Turnstile is a service that makes sure a request is not coming from an emulated browser used by a bot by checking in on various characteristics and processes inherent to browsers and how they behave when humans are using them versus when bots do.
Unlike a Captcha, this process was entirely invisible to the voters, but it allowed the contest to benefit from better conversion rates because fraudulent traffic was filtered out and legitimate users could connect quicker.
All right on the night
The two semi-finals went entirely according to plan, with approximately 200,000 concurrent voters hitting up the system across the two nights.
On the night of the Grand Finale, Saturday 13 May, De Beers and the team first made sure they had enough machines available to handle the initial peak when voting opened, and then conducted ongoing monitoring with Cloudflare.
They did clock a few attempts to DDoS the site, but these were automatically mitigated without any noticeable impact to voters.
As predicted, the central registry server also received some unwelcome attention from attackers, but the combination of Cloudflare’s content delivery network (CDN) and DDoS protection meant that the server’s cache hit ratio – this is a measurement of how many requests a cache is capable of fulfilling versus how many it receives – which for normal traffic should be about 20%, peaked at 80%, well short of becoming overloaded.
When the curtain came down on an historic second victory for Sweden’s Loreen (a feat only matched by Ireland’s Johnny Logan), the Based.io platform had handled a total of 350 million events and served seven million unique users over three hours. At peak, 1.3 million concurrent users were connected, and the voting landing page served 2.3 million requests per second.
“We did get attacks, but they didn’t get further than the first layers,” says De Beers, who reckons the system saw about 100 million malicious requests. “All the malicious IPs were involved got flagged at the access key level. They only attacked for three minutes and then they moved on to other parts of Eurovision…I was super relieved. If they had increased their efforts from last year to Junior Eurovision, we would have seen an increase in threat.”
“It was a great success,” adds Cloudflare CTO John Graham-Cumming. “I was really happy with it, and we obviously internally kept an eye on their account just to make sure there were no problems happening at critical moments. It all went extremely smoothly.”
How the Rest of the World voted
Bonsoir Europe, bonsoir Liverpool, et merci pour ce merveilleux spectacle! Voici les résultats du reste du monde:
- France, one point. La France, un point.
- Spain, two points. L’Espagne, deux points.
- Croatia, three points. La Croatie, trois points.
- Norway, four points. La Norvège, quatre points.
- Ukraine, five points. L’Ukraine, cinq points.
- Albania, six points. L’Albanie, six points.
- Sweden, seven points. La Suède, sept points.
- Armenia, eight points. L’Arménie, huit points.
- Finland, 10 points. La Finlande, dix points.
- Israel, 12 points. L’Israël, douze points.
During the process, the most online votes came from Canada, Chile, Hungary, Kosovo, Luxembourg, Mexico, New Zealand, Slovakia, Turkey, the UAE and the US.
Planning for the future
Of course, Eurovision viewers and fans in the UK – and most other participating countries – were not included in the online voting experiment, and millions of votes were tallied via SMS or, in the UK’s case, by calling a dedicated phone number to listen to a recording of Graham Norton.
As happens every year, the phone systems were quickly overwhelmed and many voters had to make multiple attempts to get through. So when will online voting be coming to the UK – and will it be before our next win?
“It’s really annoying, I know…The reason why it’s still on SMS is the fear of cyber security threats, the fear of bad actors, the fear that it will go down. So that’s why we started with the rest of the world,” explains De Beer.
“But I think over the next couple of years, we’re going to see more and more countries where we allow online voting. It’s way easier and a much better user experience.
“We’re patient from our side – we’re just super happy that we could use our new technology that is still in closed beta – to power this event. It’s incredible for us,” he adds.
And the future of online voting at the Eurovision Song Contest looks assured, as De Beer notes; for the two participating countries where the online voting experiment did take place – Australia and Israel – more votes than usual were recorded because there were no phone lines to get blocked.
Read more about DDoS attacks
- A rise in massive DDoS attacks, some of which target the application layer and cause significant disruptions, might require new defence strategies from cyber security vendors.
- Investigations into recent outages on Microsoft Azure and Outlook services have turned up evidence of a massive DDoS attack.
- The KillNet DDoS gang seems to be becoming more dangerous, but its primary goal remains to create a lot of noise and draw media attention, according to latest analysis.
