Top Menu

Archive | Case Studies

Malaysia Bank Adopts Artificial Intelligence Early Warning System to Reduce Risks of Non-Performing Loans (NPLs)

The client is one of the largest international banks, headquartered in Singapore with a prominent presence in the Asian region. As a Financial Institution with significant focus on commercial loans, the challenge of managing credit risk from loans turning delinquent is pervasive and a major concern to the bank. With a clear understanding that high NPL ratios demand greater loan provisions, which reduces capital resources available for lending and dents the bank profitability, the Bank decided to embark in this journey to mitigate the risks of NPLs.

The Bank was using manual excel-based spreadsheets to forecast potential delinquencies based on historical data and repayment behaviors. This manual process was labor intensive, time consuming, lacking futuristic prescriptive analytics modeling and its precision was questionable. Scalability of analysis was a constant challenge with changing economic climate and trends.

Pounded by these challenges, the Bank realized its need for an automated and intelligent solution with the capability to provide early warning for delinquent accounts.

Solution

With a significant double-digit increase in the number of approved loans, the Bank prioritized its focus on effective management of loans recovery and delinquencies. The Bank realized the need for automation coupled with deployment of Artificial Intelligence (AI) instead of the labor intensive manual processes which lent a toil on the credit management teams.

The conventional CRISP-DM (Cross Industry Standard Process for Data Modelling) breaks the process of data mining into six major phases of which the sequence of the phases is not strict and moving back and forth between different phases is always required. This conventional model usually requires disparate tools and trained statisticians to navigate usage of these tools for data mining.

Juris Technologies has taken a distinctive forward-looking approach in bringing Machine Learning to the next level.  Leveraging design thinking and embedment of wizards, the business users are empowered with the power of machine learning and artificial intelligence to apply their own domain knowledge and hypothesis in training the different analytical models.  By empowering business users, organizations reap the benefits of significant improvements with the capabilities of continuous training of analytical models.

The Bank engaged Juris Technologies, an end-to-end broad spectrum customer and credit management solutions provider to automate workflow and processes for loans management. With precise understanding of the Bank’s need, Juris Technologies proposed and delivered the Juris Mindcraft solution, a system powered by Artificial Intelligence capabilities, to predict potential Non-Performing accounts as early as 6 months prior to delinquency.

Juris Technologies utilized the Bank’s readily available data, consolidated various attributes of Term Loan, Trade and Overdraft services of accounts, and applied Artificial Intelligence prescriptive models. These models combined historical accounts behavioral analysis with future predictive analytics and algorithms to identify and segregate accounts potential of deterioration to NPL status. In addition to analytics, the solution also provided the Bank’s users with insights, rationale and rules for the identification of such accounts, enabling strategic and tactical actions to be deployed in managing and reducing NPLs.

Benefits Obtained

Combining the capabilities of the Bank’s proprietary Credit Scoring Model with Juris Technologies’ early warning and account monitoring system, the Bank successfully enhanced its existing credit assessment structures and risk assessment capabilities. Coupled with improved economic climate, the Bank was able to reduce NPLs significantly from a sharp hike to a plateau.

With reduction in NPLs, the Bank enjoyed increased profitability and improved their ability to provide new lending and a healthy working capital.

Industry: Banking

Big Data Analytics Solutions ProviderJuris Technologies is a Malaysia-based, MSC Status company providing innovative business solutions in the financial services and telecommunications industry.

Juris Technologies started business operations in 2003 with a primary focus in building solutions serving the Banking, Financial Services and Insurance (BFSI) and Telecommunications industry. Our solution consists of an integrated framework of scalable components which is used to build enterprise quality solutions.

Our solution framework is used to design and develop solutions for customer relationship management with modules such as Contact Management, Campaign Management, and Sales Force Management. In the areas of Credit Management, we provide solutions for Customer Acquisition, Loans Origination, Credit Scoring, Conveyancing and Litigation, and Collection and Recovery. Today, we are the leading customer and credit management solutions provider in Malaysia, offering a collaboration platform for over 1,000 external parties including legal firms, collection agencies and valuers.

Our key clients are Affin Bank, Agro Bank, Alliance Bank, Bank Kerjasama Rakyat Malaysia (BKRM), Bank Persatuan, Bank Simpanan Nasional, CIMB Bank, CTOS, Hitachi Capital, HSBC, JCL, Maybank Etiqa, OCBC Bank, SME Bank, Standard Chartered Bank and Tenaga Nasional Berhad (TNB) Malaysia.

Our global footprint for the licensing of Juris Application Server spans across 22 countries and  the Juris Application Server framework is used by organizations such as Sony USA and The Software Engineering Institute of Carnegie Mellon.

Singapore Bank Embraces Machine Learning To Boost Their Advertising Campaigns

The client is one of the largest banking and financial services corporations in Asia. Headquartered in Singapore, the conglomerate has operations across multiple countries around the region.

The bank has a substantial budget for digital ad campaigns. In the past, however, a large portion of their digital advertisements were not optimized; their ads were reaching many viewers who either had no interest in their products or lacked the propensity to purchase anything from the bank.

In fact, if we look at the current digital ad landscape, 54% of digital advertising in general goes unseen, resulting in wasted impressions and opportunities. Marketing channels as well as audience are highly disjointed in this digital age. Audience fragmentation has made it nearly impossible for marketers to manually segment their target market and perform targeted selling. Also, if they were to miss out key segmentation variables or channels, opportunities to convert would be lost.

The bank realized that there was a need to optimize their ads – automatically and in a very intelligent manner.

Solution: The opportunity for the bank was to continuously find the best target audience for their ad campaigns. And this process had to be done automatically instead of manually tweaking the hundreds of parameters staring at the marketing team in the face.

The bank engaged Optimate, a campaign optimization engine that uses self-learning artificial intelligence (AI) to automate and enhance their digital advertising projects. Optimate has a delicately-designed AI system that uses data-driven logic, learning from historical campaigns and matching promoted ads and products with interest keywords in their proprietary knowledgebase.

Centered on the bank’s desired target market, as well as content from ad creatives, the system would dynamically search for the right audience. Optimate applied sampling-based methodology to augment the scale of data, and to generate the most appropriate target audience based on evidence provided by the data itself.

Optimate AI Overview

The ad optimizing engine was also able to diagnose the status of the client’s ad account via a Competitiveness Index (CI) from all the data within the account linked to their system. A/B testing automatically carried out on different ad creatives by the self-learning algorithms before being pushed to the target audience.

Further budget adjustment analysis was also provided to the bank to complete the system’s set of automatic ad campaign creation and optimization solutions.

Benefits & ROI: The bank’s overall ad effectiveness improved tremendously; the AI engine perpetually worked in the background to fine-tune and deliver the ads to the right audience, all in real-time.

In the two campaigns that were optimized by Optimate’s AI engine, the bank managed to achieve up to 36% increase in reach, 48% reduction in cost per link click, 41% reduction in cost per post engagement, and 50% increase in relevance score.

In a short span of 4 days, the bank managed to cut down their campaign cost by 20%.

And the good ROI will only get better as more campaigns are run for longer periods of time. The more valuable data is generated and analyzed by the AI engine, the more effective the ad optimizations will be.


Did you miss last week’s Optimate Webinar on the Future of Marketing 2017? You can view the recording here. The webinar covered common FB marketing mistakes, 2017 marketing trends, and how SMEs can tap into AI-driven, automated marketing tools for runaway marketing success.

As someone who runs digital ad campaigns, you stand to benefit from Optimate’s sophisticated AI marketing engine. You can sign up for one of their plans using the discount code OPTDECWEB.

Industry: Banking
Big Data Analytics Solutions Provider: Optimate is an AI optimization engine for multiple channels of marketing. Combining self-learning artificial intelligence (AI) to automate and optimize online marketing with a cloud platform, Optimate enhances businesses’ marketing activities by streamlining the process of serving the best content to suitable audiences using machine learning. Optimate now serves over 100 local and regional clients, ranging from Fortune 500 companies to local SMEs. www.optimate.co

Confess Now, Or The Taxman And Big Data Analytics Will Come After You

Governments worldwide are under mounting pressure to tighten their tax gaps, defined as the difference between what the government expects to collect in taxes versus what is actually collected. Closer to home, in a 2011 Economic Transformation Program (ETP) update by Minister in the PM’s Department and CEO of PEMANDU, Dato’ Sri Idris Jala, Malaysia’s tax gap stands at 20%.

Based on the country’s 2014 direct tax collection of RM133.7 billion, a 20% tax gap leaves an estimated (and whopping) RM33 billion worth of taxes uncollected and unaccounted for.

In contrast, many developed nations have tax gaps hovering between 10% – 15%.

Fraud is a major component of a nation’s tax gap, and it comes in many flavors. People and companies evade taxes by underreporting their income. Moreover, they tend to underpay or avoid filing their taxes altogether. The hidden economy – such as money laundering, prostitution, arms and drug trafficking – is also another key reason for a country’s tax gap.

In the past, efforts to widen the field audit and investigation coverage to narrow the tax gap were met with limited success due to manpower constraints and difficulties in monitoring and identifying these fraudulent individuals and organizations.

This is about to change. Governments are increasingly turning to sophisticated techniques like big data analytics to identify tax fraud and increase its tax revenue.

Numerous advanced data analytics tax collection business cases have been developed in recent years. These include initiatives such as propensity to pay, tax collection risk scoring, anti-money laundering, and GST fraud identification.

The Tax Big Data Analytics Maturity Model (Figure 1) shows that high-impact tax business cases that will bring in serious returns for the government – like GST fraud identification, anti-money laundering and real-time fraud detection – will require the deployment of sophisticated analytics that goes beyond traditional reporting, statistical and predictive models. These high-value projects will require the implementation of advanced data analytics such as text mining, path analysis, connections, affinity and visualization.

tax big data maturity model

Figure 1: Tax Big Data Analytics Maturity Model (Image Credit: Teradata)

Let’s take GST scams as an example. The government will be able to use big data analytics to clamp down on companies that charge GST but do not pay the output tax. They will also be able to use advanced analytics techniques to detect GST carousel fraud (Figure 2).

Traditionally, it is impossible for tax fraud analysts to scrutinize millions of transaction data points. The execution of the carousel fraud, for instance, usually takes place in just a few weeks, and by the time it is detected by tax agents, the fraudsters would have gone missing with stolen tax money.

With big data analytics, millions of transactions can be automatically processed. A graph data model can be built to represent GST fraud. Powerful visualization tools can be used to present the chains of suspicious transactions. Fraud analysts can drill down to see companies and individuals connected to these transactions. These suspicious cases can then be flagged for audit. Anti-fraud and investigation teams will be able to act faster to avoid the crime from being committed.

Yep, the taxman is comin’ after you with big data.

gst carousel fraud

Figure 2: An Example of a GST Carousel Fraud (Image Credit: Inland Revenue Authority of Singapore)

 

Malaysia Airports develops data insights to optimize customer satisfaction

Airport operator company, Malaysia Airports, with interests in Turkey, India, and South Asia, is working with Fusionex on a proof-of-concept: to develop a Business Intelligence platform to further enhance Malaysia Airports’ retailer management system within KLIA and provide value-added services for travelers.

Introduction

Malaysia Airports is an airport management company with a portfolio of almost 40 airports in and around South Asia and Turkey. Apart from the aeronautical business, the client also manages other portfolios including duty-free & retail operations, hotels, free commercial zones, commercial space leasing, and management of parking facilities. It also runs global training centers regarding airport management, airport fire & rescue services, and aviation security.

The Challenge

Malaysia Airports sought to enhance its customer data collection methods and determined to increase the accuracy of gathered information on the performance of its retail units within KLIA. This involves an approximately 400,000 square foot containing retail outlets at various locations including the arrival and departure halls as well as check-in counter areas. These outlets/locations see a fluctuating amount of shoppers at different times and days of the week.

Malaysia Airports aims to ramp up its data collection frequency near the shops area. A higher degree of prompt data collection would serve to accurately determine when promotional activities should be held, or if they were held – what their projected results would be. Accuracy of information garnered from this area is important to determine the customers’ hotspots and their spending trends.

A precise method to track spending trends at the terminal would allow Malaysia Airports to identify local shoppers and international travelers apart. This would also result in a more informed decision being made with regards to what the retail composition should be – whether the terminal should host more shops selling food, garments, souvenirs, or other goods base on the shoppers spending pattern and behaviour.

The Solution

A combination of the Fusionex Business Intelligence platform together with Internet of Things (IoT) devices was utilised to collect and manage the terminal’s retail data. Sensors were installed at the main terminal building’s arrival and departure halls and the contact piers for international and domestic flights.

Tracking the traveler jouney via the Fusionex BI Dashboard

Sensors are also placed in and around the airport’s retail sections to determine hotspots where the most retail activity occurred. The sensors are functioning independently without requiring wiring or lengthy installation methods. Among the sensors used were those with Bluetooth capabilities and Wi-Fi detection capabilities installed in collaboration with solution provider Tapway.

A mobile application was developed to track the basic demographic background of customers including details such as gender and nationality. A campaign was conducted to encourage travelers to sign up for the mobile app. The app could then use the demographic information to craft personalized marketing messages and promotions for the customers.

Using a wide set of connectors, various type of data set and system can be integrated into single platform with speed and ease. The dashboard reporting is then rendered automatic as well, giving the client the ability to view revenues in near real time. These endeavors were supported and co-funded by the Malaysia Digital Economy Corporation (MDEC).

The Benefits

The ability to receive performance data from their existing tenants quickly, together with traveler information from installed sensors, allowed Malaysia Airports to make more accurate decisions:

Understand traveler habits and shopper behavior, preferences. Malaysia Airports will be able to determine travelers’ flow and identify trends of unique shoppers via the installed sensors. Other than that, demographic data such as gender, date of birth and nationality will also be gathered to help Malaysia Airports understands which retail outlets appeal the most to which groups of people. This provides insights into what composition of retail outlets would be optimum in the airport – whether or not certain shops fit, and whether they fit better in domestic or international halls.

Ability to perform more accurate marketing efforts. Passengers’ traffic data gathered, which could be drilled down to daily and even hourly, could inform Malaysia Airports when would be the right time to hold promotional and marketing events within the terminal. Connectivity via travelers’ mobile devices provides a possibility for future marketing undertakings such as customer-specific push notifications. When walking near a particular shop which the traveler might be interested in, the platform pushes a notification to their phones for a promotion on items or services available.

Summary

By utilizing IoT devices and gathering useful data on a centralized system, Malaysia Airports is able to get a good overall view of its airport’s retail sector. Being provided with user-friendly BI dashboards also enable management-level personnel to get quick looks at the information to make timely and accurate data-driven decisions.

Industry: Aviation / Retail

Big Data Solutions Provider: Fusionex is an established multi-award winning IT software group that specializes in Analytics and Big Data. Their business is to help clients manage, make sense of and derive useful insights and information from the vast amounts of structured and unstructured data at their disposal. Fusionex is focused on bridging the gap between business and technology, and in doing so, providing an exceptional and positive experience to customers of various markets.

Advanced Analytics in Talent Management Strategy and Operation

The Chief Human Resource Officer of our client, a leading telco in Malaysia, overseeing over 3000 staffs is faced with a sharp rise in HR cost in recent years despite maintaining a tight control on headcount and increments. The key questions we set out to answer was why HR cost is increasing and what can be done about it. It turns out to be a case well-fitted for BIIT Analytics Framework.

Descriptive: Understand Past & Present

We firmed up the problem statement by framing staff cost as a function of key workforce dynamics (i.e. hiring, attrition, promotion and increment) and salary. Using component analysis and a fair bit of calculus, we distilled the four main drivers of cost change: cost per head, headcount, structural change and combined effect. It was then revealed that structural change has been the culprit. This is exactly why while hiring and increment were controlled tightly in the past, HR cost kept rising.


Figure 1 Breakdown of the effects of cost drivers

  • Cost per Head Effect: Change in cost if only cost per head changes at each job level while headcount and structure remain the same
  • Headcount Effect: Change in cost if only total headcount changes while structure and cost per head remains the same
  • Structural Effect: Change in cost if only structure changes while total headcount and cost per head remain the same
  • Combined Effect: Change in cost not explainable by or nonexistent without the other 3 effects

Predictive: Foresee What Is To Come

While stakeholders understand intuitively that such scenario would be disastrous if it were to continue, it is difficult to put an amount to it to compare against other priorities in the business. We studied and ran time series forecasting on the key variables to estimate potential impacts of inaction. The resulting model shows that evolving workforce dynamics will cause the workforce to be fatter at the higher end of hierarchy and thinner at the lower end. If such trend continues, staff cost will very quickly reach an unsustainable level especially when the market is expected to underperform.


Figure 2 Headcount by job level graphs showing evolution into top-heavy workforce structure

 Prescriptive: Pinpoint the Right Course of Action

The next question on everyone’s mind is naturally what should be done to contain the cost increase. To contain the cost issues driven by structural change, one can use the four key workforce dynamics as levers. We modelled staff cost as a function of the dynamics and embedded it into strategic and tactical simulators. With simulators at both levels, the HR management team can simulate scenario with different hiring rate etc., and set policies based on the results.


Figure 3 Screenshots of the strategic simulator tool for adjusting key action levers and simulating potential outcomes

Figure 4 Screenshots of the tactical simulator for line managers to simulate mid-term outcomes of their actions such as promotion, transfer, hiring & etc

Figure 5 Output of the tactical simulator shows outcomes of middle management actions at all levels of organization

Proactive: Enable Action through Technology

No strategy and policy is perfect. The impact of policies should be monitored continuously so that timely adjustments can be made when needed. On that front, we proactively designed and implemented cost tracking dashboard for managers at all levels to ensure their accountability for staff cost incurred. There is also built-in capabilities to customize and periodically push reports to individual managers.

Figure 6 Screenshot of sample cost tracking dashboards at company level. Our solution automates reporting at all levels of organization

Our Solution:

Workforce Analytics – Deep-dive analysis into workforce cost

Using historical data, we perform deep dive analysis to uncover patterns through simple data visualisation techniques. The analysis uncovers general historical trends.

 Workforce Forecasting Models – Forecasting cost and headcount movements

Using historical data, our forecast models are able to predict the following key information on workforce:

  • Headcount by job levels
  • Cost projection
  • New hires, attrition, and promotions

The forecast is able to provide organizations with the correlation between headcount, structure and cost per head impact.

Workforce Simulator – Real-time simulation of organization re-structuring exercises

  • Ability to perform real-time simulation on existing organisation structure (promotion, new hires, termination, relocation, headcount movements etc.)
  • Ability to generate reports on directional impact of the simulated changes with high-level commentary
  • Ability to load and compare multiple versions of the simulations
  • Ability to add new department structures, and new headcount on-the-fly for simulation
  • Ability to export outputs of raw data as inputs to HR systems
  • Encryption of raw data to ensure security

Workforce Cost Tracker – Management Dashboards to monitor cost

  • Ability to monitor and capture historical trends of monthly HR cost and comparison against targets
  • Ability to monitor monthly run-rates of staff cost to give managers a true picture of structural impact
  • Ability to export the dashboards at multiple levels of granularity (e.g. Company level, Division level, Department level etc.)

About us

We are a company that truly believes that insights are strategic assets which; when leveraged properly – have the potential to transform organizations.

A couple of folks who were passionate about Data & Analytics decided to help organization realize the value of analytics across their enterprise. The company was birthed with a simple mission of providing and facilitating businesses (of any shape and size) with invaluable insights through infallible information enabled by technology.

We view technology as the means to achieve the desired business outcomes, and not the end in itself. As such, we place a large amount of emphasis on helping our Clients derive business value from all our engagements.

BIIT Consulting Sdn Bhd www.biitconsulting.com

 

How a Malaysian bank used data science to increase its credit card spend

Client Profile: The bank is one of the largest financial institutions in Malaysia with retail branches in numerous Southeast Asia countries.

Challenge: A substantial segment of the bank’s credit card holders with high spending potential is not utilizing its credit card as their primary card.

Solution: Cardholders with spending potential were segmented based on various demographic attributes such as gender, product holdings and amount spend.

Collaborative filtering – a technique to predict user interest based on the behavior of other users – was applied to detect spending patterns. For example, the bank wanted to know how frequently merchant offerings appeared against customers with specific product holdings. And how often merchant offerings showed up against customers of a particular demographic.

tdata1

Customer segmentation and collaborative filtering to forecast consumer propensity scores

Each potential customer was scored on their spending propensity for every merchant category like dining, petrol, groceries and airlines.

The bank then matched specific merchant offers to individual customers based on their inclination to spend in that merchant group.

Customer data, card transactions and the effectiveness of previous merchant offers were used to identify potential cardholders to drive higher usage of the bank’s credit cards.

tdata2

Every customer is scored against every merchant category on their propensity to spend

Business Benefits: With personalized marketing campaigns derived from data analytics, the bank saw an increase in its credit card spend and utilization, resulting in a substantial increase in interchange and interest income.

Industry: Banking

Big Data Solutions Provider: Teradata Malaysia
tdata-logo

Big Data Use Cases in Retail

Dynamic pricing across multiple channels is not new, but big data allows for a much more refined set of indicators for price elasticity in comparison with traditional influencers such as time and availability. Other indicators include the weather, the location, the complete buying profile of a customer, and the social media presence of a customer (prescriptive analytics).

Fuzzy matching helps when people search for jobs, hotels, secondhand cars, houses and other goods with many characteristics. Fuzzy matching pairs results that are almost fitting, like a dark blue car instead of a black one, or a hotel on a different Greek island from the one requested (prescriptive analytics).

Counter-dynamic pricing is the opposite of dynamic pricing. If big data analytics can refine price-elasticity models, the optimization algorithms can also be reversed and put to use for consumers. Personal analytics help determine the best moment to buy goods or services for the lowest price, and carry out the order immediately (prescriptive analytics).

Fraud detection is important in most industries. Specifically for the retail sector, one can think of tracking fraud in returns and abuse of customer service, or credit risk for larger purchases, based on, for example, uncovering fraud rings, the social media activity of customers and detecting patterns (descriptive analytics).

Dynamic forecasting complements traditional demand and supply chain forecasting, taking more external factors into account, such as traffic conditions, weather forecasts, shop video feeds suggesting demand, and sensors (moving from sensors attached to checkpoints and monitoring objects [products, materials, parts] to sensors attached to every single object to be monitored) (predictive analytics).

Recommendations are the mostly widely adopted big data use in the retail sector. Based on what other customers have bought, a customer could also be interested in another product. This is also called “best next offer” recommendation. Recommendations can benefit from a much broader context, not only checking which combinations are most likely, but also, based on a very fine-grained “graph analysis,” identifying a closely related peer consumer group. It can also work with people, such as when LinkedIn recommends other people to connect to. It can additionally be used with locations, based on common or complementary characteristics (descriptive analytics, prescriptive analytics).

Retail data can be monetized by selling it upstream to partners and suppliers, to give them more insight into how their products are selling and in which circumstances. Information on, for instance, buying patterns throughout the day, in relation to the weather and how busy stores are, helps suppliers optimize their marketing (general management).

Market basket analysis traditionally matches products that are purchased together. Big data adds more context, including time of day, music played in a store, store visit duration, weather, length of queue and so forth (descriptive analytics).

Mall experience gamification is a new concept, based on smartphone location data (or data from any other location-aware device). Visitors to a mall can be tempted to buy more and stay longer by tracking their movements through a mall and sending them special offers — for instance after checking in at more than 10 stores (descriptive, predictive analytics, prescriptive analytics).

Real-time offers respond in real-time to changing patterns in visitor numbers. Not only online, where dynamic promotions were pioneered, but also in the “real world.” For instance, in airports, shops can put different items on sale based on flights to or from specific locations. Shops can change their promotions not only based on the weather forecasts or commuter streams, but also based on more fine-grained algorithms, including social media analytics (what are people talking about in the neighborhood) (predictive analytics, prescriptive analytics).

Shopping cart defection is not a new form of analysis in the online retail sector. But big data enables consideration of many more factors, next to clickstream analysis (diagnostic analytics).

Loyalty management benefits from big data by extending channel reach from point of sale, Web and call center to include mobile and social capabilities. Rewards may be accrued by more than purchases; people may also earn them for being good product or brand ambassadors. Rewards may also come from more than personal contributions — social relationships may be included (descriptive analytics).

Multichannel location analysis involves researching and evaluating optimal locations in which to develop profitable retail stores in conjunction with e-commerce and mobile commerce market analyses. It includes the use of store location analysis techniques, including the analog method, gravity-modeling multiple regression analyses, and the use of geographical information systems and e-commerce and mobile commerce trade market analysis tools to analyze multichannel trading in geographic areas (descriptive analytics).

Real-time store task management helps with the allocation of staff to, for instance, shelf restocking, customer service, checkout support and order picking, based on actual customer traffic, as determined by, for example, video analytics (prescriptive analytics).

Customer-centric merchandising helps retailers improve their product- and supply chain-centricity. Instead of selecting products based on the offerings of their suppliers and pushing them to customers, big data analytics help identify customer needs and aid the selection of new products that could increase a provider’s “wallet share” on the basis of demand (descriptive analytics).

(Courtesy of Gartner Leader’s Toolkit: Big Data Business Opportunities From Over 100 Use Cases)

Tripfez & Salam Standard use big data analytics to provide Muslims a blessed travel experience

In a market saturated with international players all offering variations of the same travel booking products, this Malaysian online travel trailblazer harnessed big data analytics (BDA) to stand out above the rest, aggregating millions of search results through their service.

tripfez1

tripfez2

Through Tripfez and Salam Standard, thousands of users request thousands of hotels, millions of reviews and photos. Tripfez focuses on the speed of processing to create a great consumer experience. At the same time, a lot of emphases are put into understanding user behaviour which is at the core of the entire operation to further increase conversion and improving consumer experience. Salam Standard not only focuses on speed, but a lot of emphases are put on processing millions of requests and data sets every minute, and transforming them. The end result are an analysis of data that will reveal data pattern, including data that shows top destinations, top selected hotels from more than 300,000 hotels worldwide across 35,000 cities.

In both cases, BDA has allowed Tripfez and Salam Standard not only the ability to process millions of datasets, but ultimately to stay ahead of its competitors, through the ability to process vast amounts of real-time data fast and efficiently, and by enabling its analysis team to better understand and predict user patterns on the website as well as user booking patterns.

Challenges

The tourism industry is highly volatile, where booking trends for certain destinations can change in a heartbeat.  A whole target group can change their preference and purchasing behaviour overnight.  Furthermore, processing of all relevant data was a resource-intensive and expensive process for the company. Tripfez has placed great importance in its ability to process millions of datasets in real-time and without downtime to ensure a smooth customer experience while great emphasis was put by Salam Standard on the ability to process millions of datasets every second in order to understand user specific booking patterns and predict future trends.

The company’s team of analysts must be able to process vast amount of data efficiently as a guide to future marketing decisions as well as trends and booking patterns that will help other travel platforms to better understand customer’s behaviour. It was also important to ensure that any advanced data analytics exercise is not just a mere business intelligence process with a cool dashboard. It is about extracting valuable insights from data and empowering decision makers with analytics that allows them to make strategic decisions and ultimately enable a data-driven organization.

tripfez3

Solution

Tripfez and Salam Standard currently store more than 20TB of data on Hadoop with a projection to increase to 100TB in the next 5 years.  The company uses Hadoop to store and process web log data; Hive to query and aggregate the data into tables; SQL Server for reporting against aggregated data; and JMP, a visual discovery software from SAS to further explore the aggregated data.

Hadoop is an open source software project that enables distributed processing of large data sets across clusters of commodity servers. It is designed to scale up from a single server to thousands of machines, with very high degree of fault tolerance. Rather than relying on high-end hardware, the resiliency of these clusters comes from the software’s ability to detect and handle failures at the application layer.

Business Benefits

A big benefit of BDA is the delivery of timely insights from the vast amount of data. This includes those already stored in the company databases, from external third-party sources, the Internet and social media. As a result, this allows real-time monitoring and forecasting events that impact business performance and operations. Such understanding and predictive abilities allow much quicker response to market changes and supports both Tripfez and Salam Standard to stay on top of these changes rather than being surprised by them. It changes the business position from being reactive towards a more proactive stance at the same time enable other travel platforms to leverage on users booking patterns to improve their conversion.

tripfez4

With the implementation of BDA practices, the hardware infrastructure for data processing can be reduced and the speed significantly increased.  BDA also allows Tripfez and Salam Standard to scale the analysis of the vast amount of data and maintain data ingestion without congesting or crashing the entire system. A substantial amount of cost-savings can be brought about for the company.

Overall, Tripfez’s and Salam Standard’s use of BDA helps it to stay ahead of its competitors, not only through the ability to process vast amount of real-time data fast and efficiently, but also to enable its team to better analyse the patterns of users. BDA also supports the company in processing years of historical data, providing its marketing team with the ability to follow user behaviour trends. This allows the team to strategize and to improve its customer experience better than other players.

The Future

From data centres to software architecture, scalability is at the heart of Tripfez and Salam Standard on many levels. Moving forward, BDA will play a big role in all of Tripfez and Salam Standard decision making processes to create a data driven decision enterprise, and at the same time ensuring that user experience is at the core of all decisions.

Doing business now and in the future means harnessing the explosive growth of data. The mobile data generation, real-time connectivity and digital businesses have changed the nature of the game when it comes to protecting data assets. As a result, BDA has an increasingly important role to play in data security, which the company wants to intensify in the future. It also wants to use BDA to transform intrusion detection, differential privacy and malware countermeasures.

Note: This is an updated case study of the previous Lagisatu story, which has since pivoted to Tripfez and Salam Standard.

 

How our local hero, ServisHero, capitalized Big Data to multiply SME revenue

ServisHero is an on-demand local services marketplace, headquartered in Kuala Lumpur, which saves consumers time and money when looking for reliable local service providers.  The process is simple:  Make a request on our website or mobile application, receive quotes from thousands of trusted local service providers, compare profiles, ratings and reviews, chat to the providers and hire when you are ready.

servishero3

We have a clear mission to uplift the income levels of millions of small businesses across South-East Asia. We are a proudly purpose-driven and data focused company using information to both shape our own business and grow into new areas

Challenges

The small-medium size enterprise sector in South East Asia is a major contributor to economic activity, accounting for in the order of 97% of all businesses, 85% of employment and 50% of GDP. Despite this importance to the overall economy, both sides of the market for the local services sector often are faced with a lack of information which hinders efficient market function. This situation has resulted in consumers being uncertain of what a reasonable price is for a service and / or being frustrated by poor quality service experiences, good quality businesses missing out on valuable growth opportunities, and government or industry partners not being able to target support to where it is needed most.

The majority of small-medium sized local service businesses understandably focus on doing what they love, which is selling their services to consumers. However, when it comes to gaining an objective understanding of their performance relative to their competitors, they are often unable to make more than anecdotal comparisons. Even if business owners are to scale the learning curve and make the investment required to install a business intelligence system, they only have information about their own business performance, and still lack the insight into whether that performance is reasonable or not.

Third parties serving the sector, including financial institutions, employment agencies and government bodies, benefit from both aggregated market information for the sector, and also targeted information to allow them to engage with individual businesses within it. However, the disparate or fragmented nature of the market has made collecting primary data an expensive exercise.

servishero1

High level information architecture, information sources, BDA solutions providers and analytics tools

Internally, we have a challenge to manage hyper-local service-specific markets which cannot be approached with a ‘one size fits all’ approach. We needed to ensure that our data analysis approach was consistent, avoided expensive re-work, while also being flexible enough to cater for the different stages of development of each micro-market, and the analytical needs of each.

Solution

ServisHero provides a solution to the timeliness and extent of information capture for the local services market. We have to the minute visibility of economic activity from granular, location-based transaction data which addresses many of the information gaps in the market. We undertake targeted primary data collection from businesses, which is made possible by the frequent and ongoing participation of thousands of local service businesses on the marketplace. An example of this is the ServisHero Small Business Sentiment Index, an ongoing quarterly indicator of business activity across the region.

We capture marketplace information using a combination of tools, notably Snowplow and AWS. A cloud based approach enables mass data ingestion and transformation, providing us with the ability to rapidly scale our business without needing to re-architect the IT solution.  We also are able to take advantage of the interoperability between the variety of tools within the AWS cloud to store and transform data as appropriate to the use case, for example, pushing our natural language data into Elastic Search for processing, Kinesis for event streaming, or batch-loading data from S3 into Redshift, before enabling distributed access to data visualisations or ad-hoc analysis with BI tools like Looker and Kibana.

We manage the competing priorities of analysis standardisation and distribution by leveraging Looker’s data modelling approach. We construct standardised building blocks for analysis which define commonly used metrics or analyses, giving us a consistent data dictionary in both theory and practice. Business users are then free to combine these building blocks to tailor analysis to suit their needs, wherever they are. We integrate the business analytics into internal and external applications, enabling real-time queries from Slack and embedding content in consumer- or Hero-facing products.

servishero2

Example output from geospatial economic activity data

Benefits

ServisHero is making radical changes to the efficiency and stability of the local service market across South East Asia. We have saved tens of thousands of consumers’ time and money by providing a vastly improved search and discovery process. Data aggregation has allowed us to improve information transfer in the market, providing consumers with visibility into provider ratings and reviews, prices and comparisons against market benchmarks.

We have been empowering local businesses across South East Asia by providing them with the opportunities and information they need to grow their businesses, and enabling access to cutting edge technological solutions which are otherwise unavailable to SMEs due to their technical nature. In one year of operation, we have facilitated the creation of more than 500 NEW jobs across the region and are generating >US$2.5m in economic impact every month. The thousands of Heroes on our platform have grown their businesses significantly, including individuals who have:

  • Expanded into multiple countries
  • Travelled overseas for the first time
  • Employed more staff
  • Grown their business revenue 10 fold

We have initiated the ServisHero Small Business Sentiment and Friendliness survey, providing a ground level view of the economy from small business owners in different parts of Malaysia and Singapore. The inaugural ServisHero Small Business Sentiment Index was released, to provide third parties with greater insight into business conditions for the under-served SME sector across the region.

The technology solutions we have adopted are allowing us to move rapidly, expanding at >30% month-on-month since launch. We are able to distribute analytics to the business users responsible for acting on data-driven decisions without slowing down by channeling analysis through a centralized data team. The BDA solution allows us to do this while also maintaining control and consistency of approach.

The Future

We will continue to scale the ServisHero marketplace to provide the rich transaction information that enables insights which we can use to empower local businesses. Our BDA solution will likely change as we grow so that we can adopt technologies that are best of breed and right for the rapidly evolving business.

The greater scale of the marketplace will allow us to develop and define advanced analytics techniques, including AI and machine learning, to improve the level of insight we are able to generate.  In parallel with developing additional depth of information, we will extend data insights to Heroes through customized digital platforms, and work with third parties to make anonymized data available to a broader community of stakeholders in the South East Asia services sector.

Industry: Services / Internet

Social Analytics Powers Highly Accurate Election Result Prediction

In my last article I have hinted that one can predict an election outcome using social media data. In this article, I will share a bit more how we have adopted social data with other data sources to predict an election outcome in a small constituency in Malaysia with 97% accuracy.

election1

Unlocking the Value of Predictive Analytics

Predictive analytics is a combination of art and science. It uses a combination of human intervention using anecdotal-based assumptions, ability to dissect multiple data sources with different data formats, deep understanding of statistical modelling (and the numbers behind it) and other data science techniques like machine learning, factor analysis, random forest and many other techniques one could think of. And let’s not forget the hours of refinement and reflection. At times, it involves simple mathematics.

The poll-plus model used by Nate Silver to predict the 2008 US Presidential Election is a living proof that predicting an election outcome or voter’s behaviour is a mixture of art and science. In the upcoming 2016 US Presidential Election, Nate expressed his latest views:

Polls shift rapidly and often prove to be fairly inaccurate,  even on the eve of the election. Non-polling factors, particularly endorsements, can provide some additional guidance,  but none of them is a magic bullet

To unlock the real value of predictive analytics, one needs to move away from believing that an investment in a technology (with a click-of-a-button user expectation) will give you a crystal-ball answer to solve your business problems. In a commercial world filled with buzz words and marketing jargons such as “big data”, “data science”, “world-class product”, “backed by the largest tech VCs”, it is easy to get distracted from seeing what value means when you look at from a single perspective (eg: product / tool). Value means combining people, process and technologies.

We predicted the election results with 97% accuracy. How did we do it ?

With differing opinions around the world on predictive approaches, we embarked on an opportunity to predict a by-election results for small constituency of 42,000 voters in 2016. When the official results were out at 2100 hours, the difference between actual vs. predicted results was 97%.

Hours of statistical modelling techniques were deployed to test the assumptions and predictors; from analysing the significance of Cubes Law, Multiple Linear Regressions (MLR) on factors such as ethnicity and age, statistical analysis on voter’s sentiment from online polling and social media data, census data, historical voting performance, effect of age and internet penetration, review of citizen’s emotions (public mood) at the locality level, national events and other data sets.

We create a crystal ball that gives you a set of realistic answers

Since there is no magic bullet in predicting election results, the best representation of a predictive outcome (i.e. probability of it happening) will fall under a best case, worst case and base case scenario. For the by-election predictive modelling, we predicted the incumbent will win 37% (worst case), 52% (base case) and 57% (best case). Each scenarios were carefully reviewed and tested using a set of weighting filters derived from multiple sources of data to represent the real-life situations. Therefore the actual result will potentially fall within the set of realistic probabilities (i.e. scenarios).

election2

We also performed Monte Carlo simulation, which was used in the past to predict the US Presidential Election, as a final sanity check to validate our predicted results. To sum it up, the overall modelling framework and approach which we have undertaken is shown below.

election3

While the framework shown may appear unexciting to some modellers or data scientists, our real competitive edge lies on the data preparation & cleaning, extrapolation and testing of various predictors that may potentially influence the voters outcome, bias estimation, logistic regression and the layers of assumptions applied in order to achieve near perfect accurate prediction.

Our cutting edge approach includes a development of a customized sentiment algorithm engine using Naïve Bayes classifier to detect local dialects in both languages (English & Malay) to identify patterns on favourability or likelihood of voting on either parties using a large sample size from social media data (i.e. both location based and keyword based). Emotions analytics was also used to measure public moods at localized locations within the constituency.

Last words on Predictive Analytics

More often than not, due to rapid evolution of computing technology and internet, users inadvertently forgot that technology (or tools) is a form of automated enablers that can deceive you to believe that those fancy charts or dashboards you view on the computer screen is the gospel truth. Sadly, many have not question the underlying assumptions and the accuracy of the data that they view every day.

The hard truth is, in the world of data science, data analytics is derived from a classic recipe (e.g.: mathematics & statistics) cooked in a brand new electric oven (i.e.: technology) by an amazing first-class science graduate (i.e. people) who is not afraid to explore new approaches or boundaries (i.e. process).

—————————————————————————————————–

For more information how we can apply predictive analytics to assist your organization, drop us a message at enquiry@berkshiremedia.com.my

(This case study is republished with the permission of Berkshire Media)