The 3L’s that kill #data projects

The typical data project starts with the BA or systems architect asking: “fast, cheap or good – which one do you want?” But in my experience, no matter how much time you have, or how much money you are willing to throw at it, or what features you are willing to sacrifice, many initiatives are doomed to fail before you even start because of inherent obstacles – what I like to refer to as the 3L’s of data projects.

Image taken from "Computers at Work" © 1969 The Hamlyn Publishing Group

Image taken from “Computers at Work” © 1969 The Hamlyn Publishing Group

Reflecting on work I have been doing with various clients over the past few years, it seems to me that despite their commitment to invest in system upgrades, migrate their content to new delivery platforms and automate their data processing, they often come unstuck due to fundamental flaws in their existing operations:

Legacy

This is the most common challenge – overhauling legacy IT systems or outmoded data sets. Often, the incumbent system is still working fine (provided someone remembers how it was built, configured or programmed), and the data in and of itself is perfectly good (as long as it can be kept up-to-date). But the old applications won’t talk to the new ones (or even each other), or the data format is not suited to new business needs or customer requirements.

Legacy systems require the most time and money to replace or upgrade. A colleague who works in financial services was recently bemoaning the costs being quoted to rewrite part of a legacy application – it seemed an astronomical amount of money to write a single line of code…

As painful as it seems, there may be little alternative but to salvage what data you can, decommission the software and throw it out along with the old mainframe it was running on!

Latency

Many data projects (especially in financial services) focus on reducing systems latency to enhance high-frequency and algorithmic securities trading, data streaming, real-time content delivery, complex search and retrieval, and multiple simultaneous user logins. From a machine-to-machine data handover and transaction perspective, such projects can deliver spectacular results – with the goal being end-to-end straight through processing in real-time.

However, what often gets overlooked is the level of human intervention – from collecting, normalizing and entering the data, to the double- and triple-handling to transform, convert and manipulate individual records before the content goes into production. For example, when you contact a telco, utility or other service provider to update your account details, have you ever wondered why they tell you it will take several working days for these changes to take effect? Invariably, the system that captures your information in “real-time” needs to wait for someone to run an overnight batch upload or someone else to convert the data to the appropriate format or yet another person to run a verification check BEFORE the new information can be entered into the central database or repository.

Latency caused by inefficient data processing not only costs time, it can also introduce data errors caused by multiple handling. Better to reduce the number of hand-off stages, and focus on improving data quality via batch sampling, error rate reduction and “capture once, use many” workflows.

Which leads me the third element of the troika – data governance (or the lack thereof).

Laissez-faire

In an ideal world, organisations would have an overarching data governance model, which embraces formal management and operational functions including: data acquisition, capture, processing, maintenance and stewardship.

However, we often see that the lack of a common data governance model (or worse, a laissez-faire attitude that allows individual departments to do their own thing) means there is little co-operation between functions, additional costs arising from multiple handling and higher error rates, plus inefficiencies in getting the data to where it needs to be within the shortest time possible and within acceptable transaction costs.

Some examples of where even a simple data capture model would help include:

  • standardising data entry rules for basic information like names and addresses, telephone numbers and postal codes
  • consistent formatting for dates, prices, measurements and product codes
  • clear data structures for parent/child/sibling relationships and related parties
  • coherent tagging and taxonomies for field types, values and other attributes
  • streamlining processes for new record verification and de-duplication

From experience, autonomous business units often work against the idea of a common data model because of the way departmental IT budgets are handled (including the P&L treatment of and ROI assumptions used for managing data costs), or because every team thinks they have unique and special data needs which only they can address, or because of a misplaced sense of “ownership” over enterprise data (notwithstanding compliance firewalls and other regulatory requirements necessitating some data separation).

Conclusion

One way to think about major data projects (systems upgrades, database migration, data automation) is to approach it rather like a house renovation or extension: if the existing foundations are inadequate, or if the old infrastructure (pipes, wiring, drains, etc.) is antiquated, what would your architect or builder recommend (and how much would they quote) if you said you simply wanted to incorporate what was already there into the new project? Would your budget accommodate a major retrofit or complex re-build? And would you expect to live in the property while the work is being carried out?

Next week: AngelCube15 – has your #startup got what it takes?

Australia 3.0 – beyond the mining boom….

In the wake of the G20 Brisbane meeting, Australia’s place in the world has been under scrutiny, in particular our role in Asia Pacific. With the announcement of a Free Trade Agreement with China (following similar treaties with Japan and Korea), a flurry of extra-mural visits by G20 leaders, and our current Presidency of the UN Security Council, you’d be forgiven for thinking that Australia was now front and centre of the world stage. Well, I hate to disappoint anyone, but I’ve recently spent 3 weeks overseas, and the only news I heard from home was the death of Gough Whitlam.

However, this does seem like a timely opportunity* to consider the question: “What’s next?” after the resources bubble has burst. This was the topic of discussion at this month’s Directors Suite luncheon, where I delivered some opening remarks based on the following text: 

Introduction

Our theme of Australia 3.0 is not to be confused with the think tank of the same name. Although it is interesting to note that their four areas of interest are Infrastructure, Health, Government Services and Mining.**

Historical Perspective?

I’m not a political or economic historian, but I would suggest that Australia’s policy agenda has followed a rough but discernible narrative:

  • Australia 1.0 – from the launch of Federation to the 1960’s – post-colonial era, bookended by WWI and the Vietnam War, and despite the dominant figure of Menzies, largely a protectionist, semi-nationalised, highly collective and quasi-socialist mixed economy
  • Australia 1.5 – The Whitlam Upgrade (or Experiment) – radical, short-lived, too much too soon?
  • Australia 2.0 – The Hawke/Keating System Reboot – currency and interest rate reforms, major privatization, re-engagement with Asia
  • Australia 2.5 – Rudd/Gillard bug fixes – a micro-managed response to the GFC, but despite the hype/promise, not much was actually achieved in macro terms, witness the 2020 summit…

What Issues Will Define Australia 3.0?

If we take it as read that there are demographic and environmental challenges ahead, I see that there are 5 Key Drivers for social and economic change, each with their own particular issues and consequences:

1. THE BIG ONE:
Economic activity post-GFC, post-mining boom, post-dollar parity
The “new normal”: slow/low/no growth and the struggle for sustainable growth; sunset on the baby boomer era; how to get internationally competitive, streamline SME regulations, remove the burden of tax administration
2. THE TECH TREND:
The age of mobile, cloud and social technology
Digital innovation backed by a new spirit of Gen Y/Gen I entrepreneurial start-ups; no more “job for life” employment – 1.3m non-employing businesses in Australia…. (40% of US workers will be freelance/self-employed by 2020)
3. THE END OF EMPIRE(S):
Declining respect for/relevance of political structures & public institutions
Minority governments, heightened clash of ideologies, power shift from Federal/State to Regional/Community; also reflects a failure of leadership within political parties, unions, corporations, religious bodies, professional sporting codes, armed forces etc.
4. OUR PLACE IN THE REGION
Free Trade Agreements with Asia, realigning regional interests
At what price? Implications for our traditional political allegiances? Challenges to Australia’s regional relevance if it’s one-way traffic only? Threat to food security?
5. NEW NATION BUILDING
Upgrading declining infrastructure and building capacity for the future
Who decides? Who pays? NIMBY? Too little too late?

Some international perspectives

Based on my recent travels to the UK and Hong Kong, we can make some interesting comparisons with conditions here at home. For example, like Australia, both UK and HK have very unpopular governments at present (but for different reasons); they are currently enjoying relatively higher (albeit still sluggish) GDP growth rates compared to other developed economies; and like the Australian dollar, Sterling has also declined recently against the US dollar (HK’s dollar is, of course, pegged to the US).

I got the impression that the cost of living in the UK has not gone up much since my last visit just over two years ago, although like Australia’s capital cities, London house prices are probably achieving/exceeding pre-GFC levels. (However, GDP growth is mainly due to pent-up demand from continuing austerity measures.) Relations with the EU are strained by budget issues and immigration polices. Following the Scottish referendum, there has been increased discussion on regional devolution, and Manchester looks set to acquire new regional powers (similar to the Mayor of London model). London remains as an important international financial centre, while selected manufacturing and services industries are enjoying renewed growth. There were numerous signs of major infrastructure projects (notably the Crossrail in London) and urban renewal initiatives (such as the Manchester City Library upgrade).

Meanwhile, HK is going through yet another constitutional crisis under the post-handover Basic Law (“One Country, Two Systems”). The Occupy Central protests, aka the Umbrella Movement were the most orderly demonstrations I have ever seen. The protests are multi-faceted; they are not just about Universal Suffrage, but also reflect social, economic and cultural struggles/challenges. There is another (speculative) property boom, fuelled in part by new subway systems, new commercial buildings, and a harbour front tunnel to by-pass the CBD; and in part by hundreds of new apartments (attracting mainland buyers). Property prices are at another all-time high (new developments can cost US$4-5m for less than 1,000 sq. ft.) – no wonder that about half of the population now live in public housing projects, and nearly one-fifth are estimated to be living below the poverty line. But food, clothing, public transport, eating out and general consumer goods can still be bought at modest prices (as long as you avoid high-end brands in high-end malls).

Making The Right Connections

I spent two days at a major Asia Pacific financial services conference in HK aimed at stock exchanges, banks and data vendors, where I only saw a couple of delegates from Australian banks, nobody from the ASX and no-one from the Australian superannuation or asset management sectors.

Does this matter? I think it does.

There was much talk about the convertibility (or internationalization) of the RMB, and one currency broker I spoke to suggested that Australia will be the next target for major RMB investment – it’s not just about Toorak mansions. There are huge RMB deposits sitting in HK, and Australia is an approved investment destination (and Australian-managed funds are an approved asset class) for approved mainland investors. The money has to go somewhere.***

By standalone stock market capitalisation, ASX is ranked 14th globally, but represents only about 2.2% by value. Furthermore, when taking into account recent stock exchange mergers and the new HK/Shanghai Stock Connect trading platform, the combined Hong Kong/Shanghai/Shenzhen market cap will leapfrog into 2nd place globally, and into 1st place in Asia Pacific, displacing Japan from its long-held position. And even though conference delegates often talked about the 4 key regional markets of HK, Japan, Singapore and Australia, the ASX comprises a mere 6% of regional market value, and the only exchange ASX has had serious (but failed) merger talks with is Singapore – which does not even make the global top 20.

The ASX market cap is $1.5tn; total superannuation funds and assets under management are about $1.6tn; while the equity in family owned businesses that needs to be refinanced over the next 5-10 years is estimated to be about $3.5tn.

Even financial market experts in Asia were acknowledging that wealth management, retirement planning and private banking services are gaining more significance than IPOs and equities trading. This in turn places greater emphasis on long-term investments, asset management for future returns, a new role for private equity, and more allocations to fixed income and bonds. But regulatory and operating costs threaten to erode any value that is being created in these asset classes, unless service providers and intermediaries can generate better efficiencies and/or develop additional, high-value products and services.

For our part, do we need to explore the role of alternative stock exchanges and non-traditional fund-raising platforms (especially for emerging companies and infrastructure projects)? And what is happening with Australia’s anticipated role as a regional fund and asset manager?

Implications for NEDs

As Non-Executive Directors, does this mean we should be shifting our focus from the “holy grail” of a seat on a public board, and instead look at how we can help, support and build value in the small businesses that will continue to be the long-term drivers of economic growth, and ensure that the boards of super funds have adequate governance?

Footnotes:

*We were not alone: “Head of PwC Australia addresses National Press Club”

**See my own “3 Pillars of the Digital Economy”

***As part of the FTA with Australia, China has opened a RMB clearing house in Sydney, and granted Australia a portion of RQFII asset allocation. And soon after the FTA was announced, the NSW Treasury issued an RMB bond.

Next week: Managing Big Data Analytics and Visualization

 

Who’s making money from market data?

In recent years, market data vendors and their clients have been fixated on supporting the demand for low-latency feeds to support high-frequency, algorithmic and dark pool trading while simultaneously responding to the post-GFC regulatory environment. New regulations continue to place increased operating burdens and costs on market participants, with a current focus on know your customer (KYC), pre-trade analytics and benchmark transparency.

For banks and asset managers, the cost of managing data is now seen as big an issue as the cost of acquiring the data itself. Furthermore, the need to meet regulatory obligations at every stage of every client transaction is adding to operating expenses – costs which cannot easily be recovered, thereby diminishing previously healthy transactional margins.

I was in Hong Kong recently, and had the opportunity to attend the Asia Pacific Financial Information Conference, courtesy of FISD. This annual event, the largest of its kind in the region, brings together stock exchanges, data vendors and financial institutions. It has been a few years since I last attended this conference, so it was encouraging to see that delegate numbers have continued to grow, although of the many stock exchanges in the region, only a few had taken exhibition stands; and representation from among buy-side institutions and asset managers was still comparatively low. However, many major sell-side institutions and plenty of vendors were in attendance, along with a growing number of service providers across data networking, hosting and management.

Speaking to delegates, it was clear that there is a risk of regulation overload: not just the volume, but also the complexity and cost of compliance. Plus, it felt like that despite frequent industry consultation, there appears to be limited co-ordination between the various market regulators, resulting in overlap between jurisdictions and duplication across different regulatory functions. Are any of these regulations having the desired effect, or simply creating unforeseen outcomes?

One major post-GFC development has been the establishment of a common legal entity identifier (LEI) for issuers of securities and their counterparts. (This was in direct response to the Lehman collapse, as a result of a failure or inability to correctly and accurately identify counterparty risk in their trading portfolios, especially for derivatives such as credit default swaps.)  However, despite a coordinated international effort, a published standard for the common identifier, and a network of approved LEI issuers, progress in assigning LEIs has been slow (especially in Asia Pacific), and coverage does not reflect market depth. For example, one data manager estimated that of the 20,000 reportable entities that his bank deals with, only 5,000 had so far been assigned LEIs.

Financial institutions need to consume ever more market data, for more complex purposes, and at multiple stages of the securities trading life-cycle:

  • pre-trade analysis (especially to meet KYC obligations);
  • trade transaction (often using best execution forums);
  • post-trade confirmation, settlement and payment;
  • portfolio reconciliation;
  • asset valuation (and in the absence of mark-to-market pricing, meaning evaluated pricing, often requiring more than one independent source);
  • processing corporate actions (in a consistent and timely fashion, and taking account of different taxation rules);
  • financial reporting and accounting standards (local and global); and
  • a requirement to provide more transparency around benchmarks (and other underlying data used in the creation and administration of market indices, and in constructing investable products).

Yet with lower trading volumes and increased compliance costs, this inevitably means that operating margins are being squeezed. Which is likely having most impact on data vendors, since data is increasingly seen as a commodity, and the cost of acquiring new data sets has to be offset against both the on boarding and switching costs and the costs of moving data around to multiple users, applications and locations.

The overloaded data managers from the major financial institutions said they wished stock exchanges and vendors would adopt more common industry standards for data licensing and pricing. Which seems reasonable, until you hear the same data managers claim they each have their own particular requirements, and therefore a “one size fits all” approach won’t work for them. Besides, whereas in the past, data was either sold on an enterprise-wide basis, or on a per-user basis, now data usage is divided between:

  • human users and machine consumption;
  • full access versus non-display only;
  • internal and external use;
  • “as is” compared to derived applications; and
  • pre-trade and post-trade execution.

Oh, and then there’s the ongoing separation of real-time, intraday, end-of-day and static data.

This all raises the obvious question: if more data consumption does not necessarily mean better margins for data vendors (despite the need to use the same data for multiple purposes), who is making money from market data?

While the stock exchanges are the primary source of market data for listed equities and exchange-traded securities, pricing data for OTC securities and derivatives has to be sourced from dealers, inter-bank brokers, contributing traders and order confirmation platforms. The major data vendors have done a good job over the years of collecting, aggregating and distributing this data – but now, with a combination of cost pressures and advances in technology, new providers are offering to help clients to manage the sourcing, processing, transmission and delivery of data. One conference delegate commented that the next development will be in microbilling (i.e., pricing based on actual consumption of each data item by individual users for specific purposes) and suggested this was an opportunity for a disruptive newcomer.

Finally, other emerging developments included the use of social media in market sentiment analysis (e.g., for algo-based trading), data visualisation, and the deployment of dedicated apps to manage “big data” analytics.

Next week: Australia 3.0

Online Pillar 2: #Finance

Along with the launch of the iPhone 6, Apple also announced a new mobile payments system. OK, so it’s not the first smart phone app that will help you manage (read: SPEND) your money, but it’s likely to be a market leader very quickly. After all, financial services mean big money in the interconnected online economy.

This week’s blog is #2 in my mini-series on the Three Pillars. Away from NFC solutions, digital wallets and virtual currencies, what else is helping to drive online innovation in financial services?

First, as with last week’s look at Health, it’s important to consider that despite being both a defined business vertical, and a highly regulated industry, the financial services sector is also vulnerable to market disintermediation, horizontal challengers and disruptive technologies.

Although most of us tend to stick with a single financial institution for the bulk of our banking products and services, we will likely use different providers across our credit cards, insurance policies, personal investments, retirement plans and foreign currency. The major banks don’t always do a good job of being a single provider of choice because they tend to manage their customers from a product perspective, and not always from the vantage point of a life-cycle of different needs.

Most retail banks have launched customer apps – mainly for account management and transaction purposes – and likewise, other platforms such as PayPal offer smart phone solutions. As with our other two pillars (Health and Education), Finance apps proliferate – e.g., calculators, account aggregators, budgeting tools and branded customer products from major financial institutions. But unlike Health apps, at least the Australian retail banks have to comply with consumer information requirements – although I suspect this is more a requirement of APRA than Apple. (Question: should apps offering stock market data, or enabling customers to plan investment strategies have to include product disclosure statements, or ensure customers have first completed a mandatory risk profile?)

Disruption in the banking and finance sector is coming from a variety of directions:

  • traditional retailers extending their existing credit card and insurance services into deposit accounts and investment products;
  • technology startups creating online payment systems;
  • trading platform Alibaba offering microfinance, trade finance services, deposit accounts and investment funds; and
  • online retailers and market places collecting a lot of useful behavioural data on customer creditworthiness and implied financial risk – for example, platforms like eBay and PayPal are using transactional data to assign customers a quasi-credit rating score or ranking.

Elsewhere, the financial services sector drives the use of data and technology to streamline stock trading and settlement – across algorithmic trading strategies, low-latency trading, straight-through securities processing, transaction and security data matching, market identifiers and real-time data analytics. The use of social media sentiment and stock #hashtags is also creating new trading strategies among savvier investors – one major Australian bank I spoke to recently boasted of having a Media Control Centre, where they can monitor client engagement, customer activity and brand profiles across the social web.

Crowdsourcing services, along with other platforms for raising capital and early-stage funding (plus new online listing and share trading platforms) threaten to disintermediate established stock exchanges, investment banks and stock brokers. Yet I see a huge opportunity for traditional bank and non-bank lenders to use these techniques for themselves. For example, banks love asset-backed and secured lending, as opposed to overdraft or cashflow lending. However, most startups don’t have physical assets such as plant or machinery, and young entrepreneurs are less likely to own property that can be put up as collateral.

So, what if banks see startup clients as a new channel to market? By investing part of their marketing costs or R&D budgets to underwrite new business ventures, they could help fund early stage ideas, and gather valuable information on customers and suppliers. Some banks are sort of moving in this startup direction – NAB and RBS, for example – but they have yet to demonstrate new business models or innovative product solutions that align with the lean startup and new entrepreneurial generation. I have observed many founders bemoan the lack of support from banks when it comes to offering merchant services that align with the needs of startups.

On another level, banks could do more to connect ideas with capital, customers with vendors, and buyers with suppliers – as the increasingly online and highly networked economy introduces new supply chains and innovative business models. (Hint to my bank manager: referrals and recommendations are often the most cost-effective way to acquire new customers – so, maybe we can help each other?)

Of course, where financial institutions really need to lift their game is in coming to grips with the shared economy. If consumers no longer see the need to buy or own assets outright (thereby reducing the reliance on mortgages, personal loans, hire purchase agreements and even credit cards….) what are the implications for financial services? Maybe banks need to take more interest in these “shared” asset eco-systems. For example, if I have taken out an investment loan to buy an apartment, which I plan to list on Airbnb, wouldn’t it be in the bank’s best interest to make sure I am getting as many bookings as possible – by helping to market my property to their other customers, or by making it really easy for people to book and pay for the accommodation via their smart phone banking app, or by enabling me to run online credit checks on prospective customers?

It’s nearly ten years since the term “distributed economy” was coined to encapsulate the new approaches to innovation, collaboration and sustainable resource allocation. Apart from microfinance and some developments in CSR and ethical investing, I’m not sure that financial institutions really grasped the opportunities presented by the distributed economy – sure, they were quick to outsource and offshore back office operations, but this was largely a cost-cutting exercise. Innovation in financial products mainly resulted in complex (and risky) derivative instruments – and ultimately, led to the GFC.

In the current low/slow/no growth economic climate, banks have to look at new ways of generating a return on their capital. They can’t just keep paying out higher shareholder dividends (not when banking regulations require them to increase their risk-weighted capital allocation); so they must engage with the new business models and the people behind them, and they must be willing to do so with a new mindset, not one built on staid financing models. Sure, they need to maintain prudent lending standards, and undertake relevant due diligence, but not at the risk of stifling innovation in the markets where their customers increasingly operate.

(For a related article on this topic, see here. Since I drafted this blog, PayPal has launched an SME loan platform, and it has just been announced that the ex-CEO of bond fund PIMCO has taken a key equity stake in an online Peer-to-Peer lending platform.)

Next week: Online Pillar 3: #Education