Top Menu

Tag Archives | big data case study

Malaysia Bank Adopts Artificial Intelligence Early Warning System to Reduce Risks of Non-Performing Loans (NPLs)

The client is one of the largest international banks, headquartered in Singapore with a prominent presence in the Asian region. As a Financial Institution with significant focus on commercial loans, the challenge of managing credit risk from loans turning delinquent is pervasive and a major concern to the bank. With a clear understanding that high NPL ratios demand greater loan provisions, which reduces capital resources available for lending and dents the bank profitability, the Bank decided to embark in this journey to mitigate the risks of NPLs.

The Bank was using manual excel-based spreadsheets to forecast potential delinquencies based on historical data and repayment behaviors. This manual process was labor intensive, time consuming, lacking futuristic prescriptive analytics modeling and its precision was questionable. Scalability of analysis was a constant challenge with changing economic climate and trends.

Pounded by these challenges, the Bank realized its need for an automated and intelligent solution with the capability to provide early warning for delinquent accounts.


With a significant double-digit increase in the number of approved loans, the Bank prioritized its focus on effective management of loans recovery and delinquencies. The Bank realized the need for automation coupled with deployment of Artificial Intelligence (AI) instead of the labor intensive manual processes which lent a toil on the credit management teams.

The conventional CRISP-DM (Cross Industry Standard Process for Data Modelling) breaks the process of data mining into six major phases of which the sequence of the phases is not strict and moving back and forth between different phases is always required. This conventional model usually requires disparate tools and trained statisticians to navigate usage of these tools for data mining.

Juris Technologies has taken a distinctive forward-looking approach in bringing Machine Learning to the next level.  Leveraging design thinking and embedment of wizards, the business users are empowered with the power of machine learning and artificial intelligence to apply their own domain knowledge and hypothesis in training the different analytical models.  By empowering business users, organizations reap the benefits of significant improvements with the capabilities of continuous training of analytical models.

The Bank engaged Juris Technologies, an end-to-end broad spectrum customer and credit management solutions provider to automate workflow and processes for loans management. With precise understanding of the Bank’s need, Juris Technologies proposed and delivered the Juris Mindcraft solution, a system powered by Artificial Intelligence capabilities, to predict potential Non-Performing accounts as early as 6 months prior to delinquency.

Juris Technologies utilized the Bank’s readily available data, consolidated various attributes of Term Loan, Trade and Overdraft services of accounts, and applied Artificial Intelligence prescriptive models. These models combined historical accounts behavioral analysis with future predictive analytics and algorithms to identify and segregate accounts potential of deterioration to NPL status. In addition to analytics, the solution also provided the Bank’s users with insights, rationale and rules for the identification of such accounts, enabling strategic and tactical actions to be deployed in managing and reducing NPLs.

Benefits Obtained

Combining the capabilities of the Bank’s proprietary Credit Scoring Model with Juris Technologies’ early warning and account monitoring system, the Bank successfully enhanced its existing credit assessment structures and risk assessment capabilities. Coupled with improved economic climate, the Bank was able to reduce NPLs significantly from a sharp hike to a plateau.

With reduction in NPLs, the Bank enjoyed increased profitability and improved their ability to provide new lending and a healthy working capital.

Industry: Banking

Big Data Analytics Solutions ProviderJuris Technologies is a Malaysia-based, MSC Status company providing innovative business solutions in the financial services and telecommunications industry.

Juris Technologies started business operations in 2003 with a primary focus in building solutions serving the Banking, Financial Services and Insurance (BFSI) and Telecommunications industry. Our solution consists of an integrated framework of scalable components which is used to build enterprise quality solutions.

Our solution framework is used to design and develop solutions for customer relationship management with modules such as Contact Management, Campaign Management, and Sales Force Management. In the areas of Credit Management, we provide solutions for Customer Acquisition, Loans Origination, Credit Scoring, Conveyancing and Litigation, and Collection and Recovery. Today, we are the leading customer and credit management solutions provider in Malaysia, offering a collaboration platform for over 1,000 external parties including legal firms, collection agencies and valuers.

Our key clients are Affin Bank, Agro Bank, Alliance Bank, Bank Kerjasama Rakyat Malaysia (BKRM), Bank Persatuan, Bank Simpanan Nasional, CIMB Bank, CTOS, Hitachi Capital, HSBC, JCL, Maybank Etiqa, OCBC Bank, SME Bank, Standard Chartered Bank and Tenaga Nasional Berhad (TNB) Malaysia.

Our global footprint for the licensing of Juris Application Server spans across 22 countries and  the Juris Application Server framework is used by organizations such as Sony USA and The Software Engineering Institute of Carnegie Mellon.

Singapore Bank Embraces Machine Learning To Boost Their Advertising Campaigns

The client is one of the largest banking and financial services corporations in Asia. Headquartered in Singapore, the conglomerate has operations across multiple countries around the region.

The bank has a substantial budget for digital ad campaigns. In the past, however, a large portion of their digital advertisements were not optimized; their ads were reaching many viewers who either had no interest in their products or lacked the propensity to purchase anything from the bank.

In fact, if we look at the current digital ad landscape, 54% of digital advertising in general goes unseen, resulting in wasted impressions and opportunities. Marketing channels as well as audience are highly disjointed in this digital age. Audience fragmentation has made it nearly impossible for marketers to manually segment their target market and perform targeted selling. Also, if they were to miss out key segmentation variables or channels, opportunities to convert would be lost.

The bank realized that there was a need to optimize their ads – automatically and in a very intelligent manner.

Solution: The opportunity for the bank was to continuously find the best target audience for their ad campaigns. And this process had to be done automatically instead of manually tweaking the hundreds of parameters staring at the marketing team in the face.

The bank engaged Optimate, a campaign optimization engine that uses self-learning artificial intelligence (AI) to automate and enhance their digital advertising projects. Optimate has a delicately-designed AI system that uses data-driven logic, learning from historical campaigns and matching promoted ads and products with interest keywords in their proprietary knowledgebase.

Centered on the bank’s desired target market, as well as content from ad creatives, the system would dynamically search for the right audience. Optimate applied sampling-based methodology to augment the scale of data, and to generate the most appropriate target audience based on evidence provided by the data itself.

Optimate AI Overview

The ad optimizing engine was also able to diagnose the status of the client’s ad account via a Competitiveness Index (CI) from all the data within the account linked to their system. A/B testing automatically carried out on different ad creatives by the self-learning algorithms before being pushed to the target audience.

Further budget adjustment analysis was also provided to the bank to complete the system’s set of automatic ad campaign creation and optimization solutions.

Benefits & ROI: The bank’s overall ad effectiveness improved tremendously; the AI engine perpetually worked in the background to fine-tune and deliver the ads to the right audience, all in real-time.

In the two campaigns that were optimized by Optimate’s AI engine, the bank managed to achieve up to 36% increase in reach, 48% reduction in cost per link click, 41% reduction in cost per post engagement, and 50% increase in relevance score.

In a short span of 4 days, the bank managed to cut down their campaign cost by 20%.

And the good ROI will only get better as more campaigns are run for longer periods of time. The more valuable data is generated and analyzed by the AI engine, the more effective the ad optimizations will be.

Did you miss last week’s Optimate Webinar on the Future of Marketing 2017? You can view the recording here. The webinar covered common FB marketing mistakes, 2017 marketing trends, and how SMEs can tap into AI-driven, automated marketing tools for runaway marketing success.

As someone who runs digital ad campaigns, you stand to benefit from Optimate’s sophisticated AI marketing engine. You can sign up for one of their plans using the discount code OPTDECWEB.

Industry: Banking
Big Data Analytics Solutions Provider: Optimate is an AI optimization engine for multiple channels of marketing. Combining self-learning artificial intelligence (AI) to automate and optimize online marketing with a cloud platform, Optimate enhances businesses’ marketing activities by streamlining the process of serving the best content to suitable audiences using machine learning. Optimate now serves over 100 local and regional clients, ranging from Fortune 500 companies to local SMEs.

Malaysia Airports develops data insights to optimize customer satisfaction

Airport operator company, Malaysia Airports, with interests in Turkey, India, and South Asia, is working with Fusionex on a proof-of-concept: to develop a Business Intelligence platform to further enhance Malaysia Airports’ retailer management system within KLIA and provide value-added services for travelers.


Malaysia Airports is an airport management company with a portfolio of almost 40 airports in and around South Asia and Turkey. Apart from the aeronautical business, the client also manages other portfolios including duty-free & retail operations, hotels, free commercial zones, commercial space leasing, and management of parking facilities. It also runs global training centers regarding airport management, airport fire & rescue services, and aviation security.

The Challenge

Malaysia Airports sought to enhance its customer data collection methods and determined to increase the accuracy of gathered information on the performance of its retail units within KLIA. This involves an approximately 400,000 square foot containing retail outlets at various locations including the arrival and departure halls as well as check-in counter areas. These outlets/locations see a fluctuating amount of shoppers at different times and days of the week.

Malaysia Airports aims to ramp up its data collection frequency near the shops area. A higher degree of prompt data collection would serve to accurately determine when promotional activities should be held, or if they were held – what their projected results would be. Accuracy of information garnered from this area is important to determine the customers’ hotspots and their spending trends.

A precise method to track spending trends at the terminal would allow Malaysia Airports to identify local shoppers and international travelers apart. This would also result in a more informed decision being made with regards to what the retail composition should be – whether the terminal should host more shops selling food, garments, souvenirs, or other goods base on the shoppers spending pattern and behaviour.

The Solution

A combination of the Fusionex Business Intelligence platform together with Internet of Things (IoT) devices was utilised to collect and manage the terminal’s retail data. Sensors were installed at the main terminal building’s arrival and departure halls and the contact piers for international and domestic flights.

Tracking the traveler jouney via the Fusionex BI Dashboard

Sensors are also placed in and around the airport’s retail sections to determine hotspots where the most retail activity occurred. The sensors are functioning independently without requiring wiring or lengthy installation methods. Among the sensors used were those with Bluetooth capabilities and Wi-Fi detection capabilities installed in collaboration with solution provider Tapway.

A mobile application was developed to track the basic demographic background of customers including details such as gender and nationality. A campaign was conducted to encourage travelers to sign up for the mobile app. The app could then use the demographic information to craft personalized marketing messages and promotions for the customers.

Using a wide set of connectors, various type of data set and system can be integrated into single platform with speed and ease. The dashboard reporting is then rendered automatic as well, giving the client the ability to view revenues in near real time. These endeavors were supported and co-funded by the Malaysia Digital Economy Corporation (MDEC).

The Benefits

The ability to receive performance data from their existing tenants quickly, together with traveler information from installed sensors, allowed Malaysia Airports to make more accurate decisions:

Understand traveler habits and shopper behavior, preferences. Malaysia Airports will be able to determine travelers’ flow and identify trends of unique shoppers via the installed sensors. Other than that, demographic data such as gender, date of birth and nationality will also be gathered to help Malaysia Airports understands which retail outlets appeal the most to which groups of people. This provides insights into what composition of retail outlets would be optimum in the airport – whether or not certain shops fit, and whether they fit better in domestic or international halls.

Ability to perform more accurate marketing efforts. Passengers’ traffic data gathered, which could be drilled down to daily and even hourly, could inform Malaysia Airports when would be the right time to hold promotional and marketing events within the terminal. Connectivity via travelers’ mobile devices provides a possibility for future marketing undertakings such as customer-specific push notifications. When walking near a particular shop which the traveler might be interested in, the platform pushes a notification to their phones for a promotion on items or services available.


By utilizing IoT devices and gathering useful data on a centralized system, Malaysia Airports is able to get a good overall view of its airport’s retail sector. Being provided with user-friendly BI dashboards also enable management-level personnel to get quick looks at the information to make timely and accurate data-driven decisions.

Industry: Aviation / Retail

Big Data Solutions Provider: Fusionex is an established multi-award winning IT software group that specializes in Analytics and Big Data. Their business is to help clients manage, make sense of and derive useful insights and information from the vast amounts of structured and unstructured data at their disposal. Fusionex is focused on bridging the gap between business and technology, and in doing so, providing an exceptional and positive experience to customers of various markets.

Advanced Analytics in Talent Management Strategy and Operation

The Chief Human Resource Officer of our client, a leading telco in Malaysia, overseeing over 3000 staffs is faced with a sharp rise in HR cost in recent years despite maintaining a tight control on headcount and increments. The key questions we set out to answer was why HR cost is increasing and what can be done about it. It turns out to be a case well-fitted for BIIT Analytics Framework.

Descriptive: Understand Past & Present

We firmed up the problem statement by framing staff cost as a function of key workforce dynamics (i.e. hiring, attrition, promotion and increment) and salary. Using component analysis and a fair bit of calculus, we distilled the four main drivers of cost change: cost per head, headcount, structural change and combined effect. It was then revealed that structural change has been the culprit. This is exactly why while hiring and increment were controlled tightly in the past, HR cost kept rising.

Figure 1 Breakdown of the effects of cost drivers

  • Cost per Head Effect: Change in cost if only cost per head changes at each job level while headcount and structure remain the same
  • Headcount Effect: Change in cost if only total headcount changes while structure and cost per head remains the same
  • Structural Effect: Change in cost if only structure changes while total headcount and cost per head remain the same
  • Combined Effect: Change in cost not explainable by or nonexistent without the other 3 effects

Predictive: Foresee What Is To Come

While stakeholders understand intuitively that such scenario would be disastrous if it were to continue, it is difficult to put an amount to it to compare against other priorities in the business. We studied and ran time series forecasting on the key variables to estimate potential impacts of inaction. The resulting model shows that evolving workforce dynamics will cause the workforce to be fatter at the higher end of hierarchy and thinner at the lower end. If such trend continues, staff cost will very quickly reach an unsustainable level especially when the market is expected to underperform.

Figure 2 Headcount by job level graphs showing evolution into top-heavy workforce structure

 Prescriptive: Pinpoint the Right Course of Action

The next question on everyone’s mind is naturally what should be done to contain the cost increase. To contain the cost issues driven by structural change, one can use the four key workforce dynamics as levers. We modelled staff cost as a function of the dynamics and embedded it into strategic and tactical simulators. With simulators at both levels, the HR management team can simulate scenario with different hiring rate etc., and set policies based on the results.

Figure 3 Screenshots of the strategic simulator tool for adjusting key action levers and simulating potential outcomes

Figure 4 Screenshots of the tactical simulator for line managers to simulate mid-term outcomes of their actions such as promotion, transfer, hiring & etc

Figure 5 Output of the tactical simulator shows outcomes of middle management actions at all levels of organization

Proactive: Enable Action through Technology

No strategy and policy is perfect. The impact of policies should be monitored continuously so that timely adjustments can be made when needed. On that front, we proactively designed and implemented cost tracking dashboard for managers at all levels to ensure their accountability for staff cost incurred. There is also built-in capabilities to customize and periodically push reports to individual managers.

Figure 6 Screenshot of sample cost tracking dashboards at company level. Our solution automates reporting at all levels of organization

Our Solution:

Workforce Analytics – Deep-dive analysis into workforce cost

Using historical data, we perform deep dive analysis to uncover patterns through simple data visualisation techniques. The analysis uncovers general historical trends.

 Workforce Forecasting Models – Forecasting cost and headcount movements

Using historical data, our forecast models are able to predict the following key information on workforce:

  • Headcount by job levels
  • Cost projection
  • New hires, attrition, and promotions

The forecast is able to provide organizations with the correlation between headcount, structure and cost per head impact.

Workforce Simulator – Real-time simulation of organization re-structuring exercises

  • Ability to perform real-time simulation on existing organisation structure (promotion, new hires, termination, relocation, headcount movements etc.)
  • Ability to generate reports on directional impact of the simulated changes with high-level commentary
  • Ability to load and compare multiple versions of the simulations
  • Ability to add new department structures, and new headcount on-the-fly for simulation
  • Ability to export outputs of raw data as inputs to HR systems
  • Encryption of raw data to ensure security

Workforce Cost Tracker – Management Dashboards to monitor cost

  • Ability to monitor and capture historical trends of monthly HR cost and comparison against targets
  • Ability to monitor monthly run-rates of staff cost to give managers a true picture of structural impact
  • Ability to export the dashboards at multiple levels of granularity (e.g. Company level, Division level, Department level etc.)

About us

We are a company that truly believes that insights are strategic assets which; when leveraged properly – have the potential to transform organizations.

A couple of folks who were passionate about Data & Analytics decided to help organization realize the value of analytics across their enterprise. The company was birthed with a simple mission of providing and facilitating businesses (of any shape and size) with invaluable insights through infallible information enabled by technology.

We view technology as the means to achieve the desired business outcomes, and not the end in itself. As such, we place a large amount of emphasis on helping our Clients derive business value from all our engagements.

BIIT Consulting Sdn Bhd


How a Malaysian bank used data science to increase its credit card spend

Client Profile: The bank is one of the largest financial institutions in Malaysia with retail branches in numerous Southeast Asia countries.

Challenge: A substantial segment of the bank’s credit card holders with high spending potential is not utilizing its credit card as their primary card.

Solution: Cardholders with spending potential were segmented based on various demographic attributes such as gender, product holdings and amount spend.

Collaborative filtering – a technique to predict user interest based on the behavior of other users – was applied to detect spending patterns. For example, the bank wanted to know how frequently merchant offerings appeared against customers with specific product holdings. And how often merchant offerings showed up against customers of a particular demographic.


Customer segmentation and collaborative filtering to forecast consumer propensity scores

Each potential customer was scored on their spending propensity for every merchant category like dining, petrol, groceries and airlines.

The bank then matched specific merchant offers to individual customers based on their inclination to spend in that merchant group.

Customer data, card transactions and the effectiveness of previous merchant offers were used to identify potential cardholders to drive higher usage of the bank’s credit cards.


Every customer is scored against every merchant category on their propensity to spend

Business Benefits: With personalized marketing campaigns derived from data analytics, the bank saw an increase in its credit card spend and utilization, resulting in a substantial increase in interchange and interest income.

Industry: Banking

Big Data Solutions Provider: Teradata Malaysia

Tripfez & Salam Standard use big data analytics to provide Muslims a blessed travel experience

In a market saturated with international players all offering variations of the same travel booking products, this Malaysian online travel trailblazer harnessed big data analytics (BDA) to stand out above the rest, aggregating millions of search results through their service.



Through Tripfez and Salam Standard, thousands of users request thousands of hotels, millions of reviews and photos. Tripfez focuses on the speed of processing to create a great consumer experience. At the same time, a lot of emphases are put into understanding user behaviour which is at the core of the entire operation to further increase conversion and improving consumer experience. Salam Standard not only focuses on speed, but a lot of emphases are put on processing millions of requests and data sets every minute, and transforming them. The end result are an analysis of data that will reveal data pattern, including data that shows top destinations, top selected hotels from more than 300,000 hotels worldwide across 35,000 cities.

In both cases, BDA has allowed Tripfez and Salam Standard not only the ability to process millions of datasets, but ultimately to stay ahead of its competitors, through the ability to process vast amounts of real-time data fast and efficiently, and by enabling its analysis team to better understand and predict user patterns on the website as well as user booking patterns.


The tourism industry is highly volatile, where booking trends for certain destinations can change in a heartbeat.  A whole target group can change their preference and purchasing behaviour overnight.  Furthermore, processing of all relevant data was a resource-intensive and expensive process for the company. Tripfez has placed great importance in its ability to process millions of datasets in real-time and without downtime to ensure a smooth customer experience while great emphasis was put by Salam Standard on the ability to process millions of datasets every second in order to understand user specific booking patterns and predict future trends.

The company’s team of analysts must be able to process vast amount of data efficiently as a guide to future marketing decisions as well as trends and booking patterns that will help other travel platforms to better understand customer’s behaviour. It was also important to ensure that any advanced data analytics exercise is not just a mere business intelligence process with a cool dashboard. It is about extracting valuable insights from data and empowering decision makers with analytics that allows them to make strategic decisions and ultimately enable a data-driven organization.



Tripfez and Salam Standard currently store more than 20TB of data on Hadoop with a projection to increase to 100TB in the next 5 years.  The company uses Hadoop to store and process web log data; Hive to query and aggregate the data into tables; SQL Server for reporting against aggregated data; and JMP, a visual discovery software from SAS to further explore the aggregated data.

Hadoop is an open source software project that enables distributed processing of large data sets across clusters of commodity servers. It is designed to scale up from a single server to thousands of machines, with very high degree of fault tolerance. Rather than relying on high-end hardware, the resiliency of these clusters comes from the software’s ability to detect and handle failures at the application layer.

Business Benefits

A big benefit of BDA is the delivery of timely insights from the vast amount of data. This includes those already stored in the company databases, from external third-party sources, the Internet and social media. As a result, this allows real-time monitoring and forecasting events that impact business performance and operations. Such understanding and predictive abilities allow much quicker response to market changes and supports both Tripfez and Salam Standard to stay on top of these changes rather than being surprised by them. It changes the business position from being reactive towards a more proactive stance at the same time enable other travel platforms to leverage on users booking patterns to improve their conversion.


With the implementation of BDA practices, the hardware infrastructure for data processing can be reduced and the speed significantly increased.  BDA also allows Tripfez and Salam Standard to scale the analysis of the vast amount of data and maintain data ingestion without congesting or crashing the entire system. A substantial amount of cost-savings can be brought about for the company.

Overall, Tripfez’s and Salam Standard’s use of BDA helps it to stay ahead of its competitors, not only through the ability to process vast amount of real-time data fast and efficiently, but also to enable its team to better analyse the patterns of users. BDA also supports the company in processing years of historical data, providing its marketing team with the ability to follow user behaviour trends. This allows the team to strategize and to improve its customer experience better than other players.

The Future

From data centres to software architecture, scalability is at the heart of Tripfez and Salam Standard on many levels. Moving forward, BDA will play a big role in all of Tripfez and Salam Standard decision making processes to create a data driven decision enterprise, and at the same time ensuring that user experience is at the core of all decisions.

Doing business now and in the future means harnessing the explosive growth of data. The mobile data generation, real-time connectivity and digital businesses have changed the nature of the game when it comes to protecting data assets. As a result, BDA has an increasingly important role to play in data security, which the company wants to intensify in the future. It also wants to use BDA to transform intrusion detection, differential privacy and malware countermeasures.

Note: This is an updated case study of the previous Lagisatu story, which has since pivoted to Tripfez and Salam Standard.


How our local hero, ServisHero, capitalized Big Data to multiply SME revenue

ServisHero is an on-demand local services marketplace, headquartered in Kuala Lumpur, which saves consumers time and money when looking for reliable local service providers.  The process is simple:  Make a request on our website or mobile application, receive quotes from thousands of trusted local service providers, compare profiles, ratings and reviews, chat to the providers and hire when you are ready.


We have a clear mission to uplift the income levels of millions of small businesses across South-East Asia. We are a proudly purpose-driven and data focused company using information to both shape our own business and grow into new areas


The small-medium size enterprise sector in South East Asia is a major contributor to economic activity, accounting for in the order of 97% of all businesses, 85% of employment and 50% of GDP. Despite this importance to the overall economy, both sides of the market for the local services sector often are faced with a lack of information which hinders efficient market function. This situation has resulted in consumers being uncertain of what a reasonable price is for a service and / or being frustrated by poor quality service experiences, good quality businesses missing out on valuable growth opportunities, and government or industry partners not being able to target support to where it is needed most.

The majority of small-medium sized local service businesses understandably focus on doing what they love, which is selling their services to consumers. However, when it comes to gaining an objective understanding of their performance relative to their competitors, they are often unable to make more than anecdotal comparisons. Even if business owners are to scale the learning curve and make the investment required to install a business intelligence system, they only have information about their own business performance, and still lack the insight into whether that performance is reasonable or not.

Third parties serving the sector, including financial institutions, employment agencies and government bodies, benefit from both aggregated market information for the sector, and also targeted information to allow them to engage with individual businesses within it. However, the disparate or fragmented nature of the market has made collecting primary data an expensive exercise.


High level information architecture, information sources, BDA solutions providers and analytics tools

Internally, we have a challenge to manage hyper-local service-specific markets which cannot be approached with a ‘one size fits all’ approach. We needed to ensure that our data analysis approach was consistent, avoided expensive re-work, while also being flexible enough to cater for the different stages of development of each micro-market, and the analytical needs of each.


ServisHero provides a solution to the timeliness and extent of information capture for the local services market. We have to the minute visibility of economic activity from granular, location-based transaction data which addresses many of the information gaps in the market. We undertake targeted primary data collection from businesses, which is made possible by the frequent and ongoing participation of thousands of local service businesses on the marketplace. An example of this is the ServisHero Small Business Sentiment Index, an ongoing quarterly indicator of business activity across the region.

We capture marketplace information using a combination of tools, notably Snowplow and AWS. A cloud based approach enables mass data ingestion and transformation, providing us with the ability to rapidly scale our business without needing to re-architect the IT solution.  We also are able to take advantage of the interoperability between the variety of tools within the AWS cloud to store and transform data as appropriate to the use case, for example, pushing our natural language data into Elastic Search for processing, Kinesis for event streaming, or batch-loading data from S3 into Redshift, before enabling distributed access to data visualisations or ad-hoc analysis with BI tools like Looker and Kibana.

We manage the competing priorities of analysis standardisation and distribution by leveraging Looker’s data modelling approach. We construct standardised building blocks for analysis which define commonly used metrics or analyses, giving us a consistent data dictionary in both theory and practice. Business users are then free to combine these building blocks to tailor analysis to suit their needs, wherever they are. We integrate the business analytics into internal and external applications, enabling real-time queries from Slack and embedding content in consumer- or Hero-facing products.


Example output from geospatial economic activity data


ServisHero is making radical changes to the efficiency and stability of the local service market across South East Asia. We have saved tens of thousands of consumers’ time and money by providing a vastly improved search and discovery process. Data aggregation has allowed us to improve information transfer in the market, providing consumers with visibility into provider ratings and reviews, prices and comparisons against market benchmarks.

We have been empowering local businesses across South East Asia by providing them with the opportunities and information they need to grow their businesses, and enabling access to cutting edge technological solutions which are otherwise unavailable to SMEs due to their technical nature. In one year of operation, we have facilitated the creation of more than 500 NEW jobs across the region and are generating >US$2.5m in economic impact every month. The thousands of Heroes on our platform have grown their businesses significantly, including individuals who have:

  • Expanded into multiple countries
  • Travelled overseas for the first time
  • Employed more staff
  • Grown their business revenue 10 fold

We have initiated the ServisHero Small Business Sentiment and Friendliness survey, providing a ground level view of the economy from small business owners in different parts of Malaysia and Singapore. The inaugural ServisHero Small Business Sentiment Index was released, to provide third parties with greater insight into business conditions for the under-served SME sector across the region.

The technology solutions we have adopted are allowing us to move rapidly, expanding at >30% month-on-month since launch. We are able to distribute analytics to the business users responsible for acting on data-driven decisions without slowing down by channeling analysis through a centralized data team. The BDA solution allows us to do this while also maintaining control and consistency of approach.

The Future

We will continue to scale the ServisHero marketplace to provide the rich transaction information that enables insights which we can use to empower local businesses. Our BDA solution will likely change as we grow so that we can adopt technologies that are best of breed and right for the rapidly evolving business.

The greater scale of the marketplace will allow us to develop and define advanced analytics techniques, including AI and machine learning, to improve the level of insight we are able to generate.  In parallel with developing additional depth of information, we will extend data insights to Heroes through customized digital platforms, and work with third parties to make anonymized data available to a broader community of stakeholders in the South East Asia services sector.

Industry: Services / Internet

Social Analytics Powers Highly Accurate Election Result Prediction

In my last article I have hinted that one can predict an election outcome using social media data. In this article, I will share a bit more how we have adopted social data with other data sources to predict an election outcome in a small constituency in Malaysia with 97% accuracy.


Unlocking the Value of Predictive Analytics

Predictive analytics is a combination of art and science. It uses a combination of human intervention using anecdotal-based assumptions, ability to dissect multiple data sources with different data formats, deep understanding of statistical modelling (and the numbers behind it) and other data science techniques like machine learning, factor analysis, random forest and many other techniques one could think of. And let’s not forget the hours of refinement and reflection. At times, it involves simple mathematics.

The poll-plus model used by Nate Silver to predict the 2008 US Presidential Election is a living proof that predicting an election outcome or voter’s behaviour is a mixture of art and science. In the upcoming 2016 US Presidential Election, Nate expressed his latest views:

Polls shift rapidly and often prove to be fairly inaccurate,  even on the eve of the election. Non-polling factors, particularly endorsements, can provide some additional guidance,  but none of them is a magic bullet

To unlock the real value of predictive analytics, one needs to move away from believing that an investment in a technology (with a click-of-a-button user expectation) will give you a crystal-ball answer to solve your business problems. In a commercial world filled with buzz words and marketing jargons such as “big data”, “data science”, “world-class product”, “backed by the largest tech VCs”, it is easy to get distracted from seeing what value means when you look at from a single perspective (eg: product / tool). Value means combining people, process and technologies.

We predicted the election results with 97% accuracy. How did we do it ?

With differing opinions around the world on predictive approaches, we embarked on an opportunity to predict a by-election results for small constituency of 42,000 voters in 2016. When the official results were out at 2100 hours, the difference between actual vs. predicted results was 97%.

Hours of statistical modelling techniques were deployed to test the assumptions and predictors; from analysing the significance of Cubes Law, Multiple Linear Regressions (MLR) on factors such as ethnicity and age, statistical analysis on voter’s sentiment from online polling and social media data, census data, historical voting performance, effect of age and internet penetration, review of citizen’s emotions (public mood) at the locality level, national events and other data sets.

We create a crystal ball that gives you a set of realistic answers

Since there is no magic bullet in predicting election results, the best representation of a predictive outcome (i.e. probability of it happening) will fall under a best case, worst case and base case scenario. For the by-election predictive modelling, we predicted the incumbent will win 37% (worst case), 52% (base case) and 57% (best case). Each scenarios were carefully reviewed and tested using a set of weighting filters derived from multiple sources of data to represent the real-life situations. Therefore the actual result will potentially fall within the set of realistic probabilities (i.e. scenarios).


We also performed Monte Carlo simulation, which was used in the past to predict the US Presidential Election, as a final sanity check to validate our predicted results. To sum it up, the overall modelling framework and approach which we have undertaken is shown below.


While the framework shown may appear unexciting to some modellers or data scientists, our real competitive edge lies on the data preparation & cleaning, extrapolation and testing of various predictors that may potentially influence the voters outcome, bias estimation, logistic regression and the layers of assumptions applied in order to achieve near perfect accurate prediction.

Our cutting edge approach includes a development of a customized sentiment algorithm engine using Naïve Bayes classifier to detect local dialects in both languages (English & Malay) to identify patterns on favourability or likelihood of voting on either parties using a large sample size from social media data (i.e. both location based and keyword based). Emotions analytics was also used to measure public moods at localized locations within the constituency.

Last words on Predictive Analytics

More often than not, due to rapid evolution of computing technology and internet, users inadvertently forgot that technology (or tools) is a form of automated enablers that can deceive you to believe that those fancy charts or dashboards you view on the computer screen is the gospel truth. Sadly, many have not question the underlying assumptions and the accuracy of the data that they view every day.

The hard truth is, in the world of data science, data analytics is derived from a classic recipe (e.g.: mathematics & statistics) cooked in a brand new electric oven (i.e.: technology) by an amazing first-class science graduate (i.e. people) who is not afraid to explore new approaches or boundaries (i.e. process).


For more information how we can apply predictive analytics to assist your organization, drop us a message at

(This case study is republished with the permission of Berkshire Media)

Enabling world class airport services through social data analytics

Malaysia Airports Holdings Berhad’s (MAHB) use of Big Data Analytics (BDA) enabled the world-class company to be proactive and responsive toward all public and passenger feedback – a critical factor with some 80 million people passing through their airports yearly. MAHB Customer Affairs and Resolution Excellence (CARE) unit working closely with its operations and commercial departments was designed to improve customer service acting on feedback from the public and passengers through traditional as well as social media channels. Berkshire Media was commissioned to undertake the massive effort to understand passenger behaviour by measuring customers’ perception and also to manage MAHB’s corporate reputation in the digital sphere. Real-time updates, weekly and monthly social data from MAHB passengers and customers were analysed using relevant keywords. Sentiment analysis and opinion mining were also deployed to predict customer sentiment towards MAHB.


Challenges: With more than 80 million passengers passing through MAHB airports annually, the CARE unit was set up in April 2013 to improve the Airport Service Quality (ASQ) scoring set by the Airport Council International (ACI). CARE receives hundreds of feedback a month through traditional channels such as walk-ins, phone calls, emails and letters. These feedbacks, covering major airports within MAHB’s control, include suggestions, lost and found, enquiries, compliments and complaints.  In addition, growing internet usage by passengers and visitors to the major airports in Malaysia provide a new opportunity for MAHB to expand its feedback coverage to enhance its customer experience. Historical data from various social media channels recorded an average of 45,000 social media posts per month with an impression of 240 million users.

In 2014, Berkshire Media was commissioned to undertake the massive effort to understand passenger behaviour, by measuring customers’ perception and also to manage MAHB’s corporate reputation in the digital sphere. For the first time, official MAHB social media channels such as Facebook and Twitter were set up to encourage feedback, and an integrated social media communications campaign in conjunction with the opening of KLIA2 terminal was deployed. With a 45 million passenger capacity, KLIA2 is an iconic terminal to boost MAHB’s performance, mainly contributed by its unique targeted retail and commercial offerings attractive to passengers and airport visitors.

Solution: Real-time updates, weekly and monthly social data from MAHB passengers and customers were analysed using relevant keywords. Sentiment analysis and opinion mining performed using supervised machine learning algorithm in multiple languages were deployed to predict customer sentiment towards MAHB. Results alongside with recommendations were shared with the CARE team on a regular basis. The CARE team members, who are deployed at the heart of the action at various locations of the airports, play a vital role in responding to specific issues based on social media feedback.

The value of big data analytics on customer feedback management is significant as we challenge ourselves to provide better passenger experience in all our airports through new customer engagement channels such as social media. –  Nik Anis, General Manager of MAHB Corporate Communications

A set of management dashboards for the purpose of weekly and monthly reporting were developed to provide the necessary insights, highlights and lowlights of what the public are saying in the social media sphere based on various topics. These reports are highlighted to the management of MAHB as a means to improve the overall customer service through MAHB social media channels, on-ground customer service, operations and others.  Sentiment was measured on a weekly and monthly basis to ensure the overall public sentiment towards MAHB was kept at a manageable level.


Example of a weekly sentiment tracking showing the positive (green), negative (red) and neutral (grey) posts from social media users on airport experiences and MAHB-related topics.

Business Benefits: Six months after the new KLIA2 airport was opened to the public, more than 200k social media posts were analysed to gauge the sentiment or customer satisfaction level of the new airport. The posts were accumulated through weekly and monthly analysis reports given to MAHB’s CARE Team. Using BDA coupled with the right communication strategies, the total number of positive posts was estimated at 72k compared to 27k negative posts, while the remaining posts were tagged as neutral. On a week to week basis, the number positive posts increased by more than 1000%. MAHB’s Facebook likes increased from 2,000 to 90,000 in eight months, while Twitter followers increase from 8,000 to 15,000 within the same period.

Positive posts were in the form of excitement, compliments and other positive experiences from the airports’ retail, ambience and customer travelling experience. Meanwhile, a handful of customer complaints on walking distance, food prices and parking charges were tracked and handled through various operational improvements. For example, on the issue of lengthy walking distance between the departure halls to the boarding gates, MAHB decided to install 16 walkalators in strategic areas of the airport to ease passenger convenience. In addition, free buggy service was aggressively marketed to increase awareness among selected passengers.

Never ignore the voice of social media. 45,000 social media posts by passengers going through the airports per month means customer satisfaction levels change every day. –  Shahid Shayaa, Managing Director of Berkshire Media

The Future: Applying predictive analytics on location-based social data will enable MAHB to anticipate customers’ needs in specific locations of the airport. For example, with location-based social media monitoring technologies, passengers who use social media to post upon arrival or departure can expect pleasant surprises from MAHB’s social media team. Retail promotions on selected products at the duty free shops can be made available by pushing targeted messages to customers at the departure or arrival halls.


Location-based monitoring capabilities will enable better customer feedback management that addresses customer issues or complaints at specific locations of the airports.

Industry: Aviation – Airport Management

Big Data Solutions ProviderBerkshire Media is a fast growing social analytics firm founded by engineers, marketers and communication strategists on the premise that the best communication strategies are built with data-driven campaigns. Their approach enables companies to utilise big data analytics to optimize campaign resources, manage corporate reputation and risks, adopt behaviour-based micro-targeted campaigns and other applications.

The company integrates the best diagnostics and predictive analytics technologies to translate social data into meaningful insights for corporations, advertising agencies, communication strategists and campaign managers.  They have rigorously extracted and analysed a great number of social data to assist clients, which include a Fortune Global 500 company, large conglomerates and public listed companies, to turn knowledge into actionable insights in order to make better or critical decisions that can transform and improve businesses.

Client Profile: Malaysia Airports Holdings Berhad (MAHB) manages 5 international gateways, 16 domestic airports and 18 short take-off and landing ports that serve the rural and remote areas in Malaysia.  MAHB also owns or manages overseas airports namely Istanbul Sabiha Gokcen International Airport in Turkey and Rajiv Gandhi International Airport in Hyderabad.  MAHB recorded a total of 83.3 million passenger movement nationwide in 2014, with more than 40 million passengers travelling through KLIA, its flagship airport.

The Rise against Landslides: Big Data to the Rescue

Challenges: Current slope monitoring exercises are carried out manually. Several methods are being used to monitor slope movements. Some of the most common methods are satellite imageries and conventional Global Positioning System (GPS) which require manual observations. Others prefer to use 3D laser scanners that can scan any topography and provide quick assessments of the slope. Though these methods are efficient in monitoring slope movements, they lack realtime monitoring and predictive analytics of when landslides will happen.


Realtime slope monitoring is still not good enough to prevent the loss of lives and assets during a landslide. Predictive monitoring and an early-warning solution is essential to save lives.

Current slope monitoring is carried out manually and lack realtime and predictive analytics.

Solution: To help tackle the issue of slope management and monitoring, Pembinaan TMC Sistem has taken the leap into Big Data and Analytics and has come up with a revolutionary solution. They developed a remote landslide sensing system with Predictive Monitoring and Early Warning Solution known as Integrated Deformation Early-warning Alert Solution (IDeas). This new system is a huge improvement over the traditional, labour-intensive and costly methods of slope monitoring.

IDeas uses mapping, 3-D scanning and smart sensing—all combined together to create one cohesive ecosystem that is able to analyse and predict slope movements, thus preventing landslides. This state-of-the-art integrated solution runs on a cloud platform, allowing the company to easily scale their resources, making the system flexible and cost-effective.


On site, a robotic station automatically tracks and measures the position of all fixed prisms at a defined interval.

The concept being used is to measure the displacement of prisms fixed on selected critical points on the slope or monitored structure. Then, the position (x.y,z) will be surveyed to produced a set of reference points for displacement calculation. Theoretically, if there is no movement, the position of the prisms will never change. If there are any differences in the measured positions, it can be interpreted that there is soil movement or worse, a landslide. The measurements will not only indicate the size of displacement, but the direction of soil movement as well.

If the displacement value is bigger than the acceptable defined tolerance, it will trigger an alarm and send out warnings to respective personnels via SMS, email and social media. Realtime monitoring might warn you when a movement is detected; sometimes, however, by the time movements are detected, it could be too late. IDeas tries to forecast the incident as precisely as possible before a landslide actually happens.


Accelerated soil erosion is detected 3 months before slope failure. By analysing realtime data, IDeas can tell if a slope failure will happen. Predictive analytics can be used on historical data to forecast when a slope failure will happen.

TMC IDeas integrates high-precision land surveying instrument with realtime communications to monitor any environment 24 hours a day, seven days a week.

Benefits: IDeas has been used in a Malaysian gold mine, one of TMC’s target industries. As safety is paramount in the mining industry, serious damages or life lost due to landslides will lead to the mine being shut down for investigations. With IDeas, the gold mine in Malaysia minimized their risk of a landslide which could potentially cost them US$1.5 million for every week its operations are disrupted..

Industry: Construction, Mining

Big Data Solutions Provider ProfileEstablished in 2009, Pembinaan TMC Sistem has since emerged as a systems integrator and developer for survey & monitoring solutions. TMC owns intellectual property rights to its flagship product – TMC Management System – which is the framework for vertical solutions such as Slope Monitoring, Bridge & Dam Monitoring, 3D Scanning & Mapping, centralized monitoring and analysis system in a web based platform. In 2010, TMC was awarded the MSC status by the government of Malaysia for its contribution to the development of flagship applications and for supporting Malaysia’s K-economy initiatives. Our products and solutions have been utilized in a number of government agencies and various corporations in Malaysia, Brunei & Asia Pacific. We have representatives & associates located across Hong Kong, China, India and USA.

BIGIT Technology Malaysia 2016, the Anchor Event of the Big Data Week Asia 2016 will be featuring concurrent conferences including the 4th Big Data World Show, Data Security World Show and the BIGIT Exhibition. Co-organized by Malaysia Digital Economy Corporation (MDEC), the event is held at KLCC Convention Centre, KL, Malaysia this September 19 & 20.

Join us and be in the presence of our highly targeted audience attending from all over the world with an expected turnout of over 3,000 delegates and trade visitors across various industry verticals and job specializations.

To register, hop over to and use the promo code MY16ADS for great offers.