3dmentat - Fotolia

How Asean organisations are untangling big data storage challenge

If organisations are to benefit from the promise of big data they must get on top of a legal and regulatory minefield when it comes to storing data. How are Asean IT departments navigating this?

Handling and understanding large and complex sets of data – or big data as it is fashionably known – has become a business imperative, and is providing answers to business problems among the Association of Southeast Asian Nations (Asean). But to get those valuable insights, organisations in the region face a tangle of legal implications, as well as the technical challenges of storing such a vast volume of data.

An obvious obstacle to storing big data is housing this large amount of information. Traditional data storage is expensive both in terms of storage space and performance.

“The deluge of data cannot be easily handled using traditional databases; the use of sample data is no longer sufficient to sate the appetites of users who crave for more accurate data,” says Andy Tan Choo Heng, lead consultant (application) at NCS, the largest IT service provider in Singapore with a presence in 10 countries in Asia, Australia and the Middle East.

To deal with this deluge of data, technology innovations with big data storage in mind include Hadoop.

There are different flavours of Hadoop to suit different requirements, says Tan. There is a high-availability, error-embracing Hadoop that uses commodity grade computers to store data and provide redundancy, with real-time data stream analysis to see and analyse data as it is streaming in. There is also Splice Machine’s fully fledged RDBMS in Hadoop, Apache Tajo’s low-latency databases in Hadoop, and MongoDB’s document-orientated database.

“Most of the Asean countries are using open-source software such as Apache Hadoop, while some companies use other open source which sits on Hadoop or MongoDB,” says Tan.

One e-commerce firm based in Malaysia said it faces challenges not just in storing the data, but also in accessing data. “We need to ensure that the database structures and storage methodologies are designed with the company’s growth in mind. Today, we do more sales in a month than we used to do in the whole year three years ago. So we definitely need strong and sound ways to store data and retrieve information,” according to a spokesman at the company that wished to remain anonymous.

“Besides data handling challenges, the rapid expansion of data creates concerns about database security, as we need to ensure the data integrity and data anonymisation of our sensitive information – be it about our customers, products or about our profitability."

Read more about enterprise IT in the Asean region

To deal with these challenges, the company uses Secure Sockets Layer (SSL) encryption to securely connect its databases, and has experimented with the latest tools and technologies to ensure that it uses strong, powerful and efficient business intelligence tools to retrieve information and perform analysis.

Santhosh Rao, principal research analyst at Gartner, agrees that when personal or sensitive data is stored, data anonymisation techniques should first be deployed to remove personal data before the data is ingested by the big data system. “Another best practice is to keep personal data in an isolated and secure data store, creating a separate data store for anonymised data that is meant to be ingested by the big data platform,” says Rao.

To ensure data immutability and to retain data in the long term in the same state and a secure manner, users are using storage and backup systems that support write once, read many capabilities and encryption. To save on storage costs, some organisations are using storage tiering and compression technologies, adds Rao.

Legal challenges associated with big data

Besides the technology challenges of storing big data, another challenge is data protection and regulatory concerns. Different data protection laws across Asean countries mean navigating these laws can be tricky, as Singapore and Malaysia have data protection laws, but Indonesia, Thailand and the rest of Asean do not. In addition, some countries require expressly given consent for data to be collected, whereas some countries accept implied consent, says Bryan Tan, partner at law firm Pinsent Masons in Singapore.

“Big data is all about utilising data that has been stored. Data protection curtails that in terms of what that data can be used for and how it is stored. Even a process like anonymisation has different interpretations across different borders,” says Pinsent Masons’ Tan.

Added to that, certain industries have to comply with specific data regulations. “Some of the financial regulators frown upon cloud storage,” says Tan. “Regulators in countries like Korea and Japan require personal data to be kept within the country – this means that companies there cannot centralise their data in datacentres, which again poses challenges for big data analysis. Indonesia and China, too, are looking to have local server laws where data service providers can only use servers in their respective countries.”

Aside from legal and regulatory challenges, there are practical and historical issues. Data quality can be an issue as different Asean countries are at differing rates of development, and may have their own unique languages. As a result, the datasets kept by each jurisdiction differ, affecting the quality of data.

Prior to starting on big data projects, organisations need to be mindful of compliance issues with big data projects.

Countries have set limited guidelines on how to secure personal data. These security mechanisms are expected to be reasonable and ensure that unauthorised data access and modification is restricted

“Exposure of personal data in the public domain, data retention periods and intentional data deletion are issues that may result in non-compliance. Organisations are using the existing data protection frameworks that their countries have formulated as a starting point,” says Gartner’s Rao.

Countries have set limited guidelines on how to secure personal data, he says. These security mechanisms are expected to be reasonable and ensure that unauthorised data access and modification is restricted. Some data protection laws – Singapore’s, for example – highlight retention periods for data types in specific verticals.

The spokesman at the Malaysian ecommerce firm agrees that the most important compliance issue for his organisation is Malaysia’s Personal Data Protection Act (PDPA).

“Since November 2013, Malaysia has enforced the PDPA which ensures that we keep information about external parties only upon their consent. We need to be careful with the usage of this information so as to respect the privacy of the customers and merchants and be as explicit as possible when it comes to usage of this information," he says.

“As one of the biggest online marketplace platforms in Malaysia for e-commerce, we make sure that we sign a detailed contract with our merchant while onboarding them to sell their products on our platform, which lays an important emphasis on the personal data protection, data governance and data security of our users.”

Even though Malaysia’s data protection laws do apply to firms with most data storage and data server activities  not carried out in Malaysia, the firms must  still closely follows the personal data protection laws there.

The volume and velocity of data being generated and analysed is set to increase, and these will continue to pose significant storage and compliance issues in the near future. Already, storage suppliers and countries are responding to these changes.

“Most storage vendors have attempted to fulfil regulations such as Security and Exchange Commission (SEC) 17a-4(f), Sarbanes-Oxley Act (SOX), Health Insurance Portability and Accountability Act (HIPAA) and regulation from the Commodity Futures Trading Commission (CFTC),” says Gartner’s Rao. “Expect data protection frameworks in Asean to undergo revisions to provide specific guidelines for data security and retention.”

Read more on Data protection regulations and compliance