The 3L’s that kill #data projects

The typical data project starts with the BA or systems architect asking: “fast, cheap or good – which one do you want?” But in my experience, no matter how much time you have, or how much money you are willing to throw at it, or what features you are willing to sacrifice, many initiatives are doomed to fail before you even start because of inherent obstacles – what I like to refer to as the 3L’s of data projects.

Image taken from "Computers at Work" © 1969 The Hamlyn Publishing Group

Image taken from “Computers at Work” © 1969 The Hamlyn Publishing Group

Reflecting on work I have been doing with various clients over the past few years, it seems to me that despite their commitment to invest in system upgrades, migrate their content to new delivery platforms and automate their data processing, they often come unstuck due to fundamental flaws in their existing operations:


This is the most common challenge – overhauling legacy IT systems or outmoded data sets. Often, the incumbent system is still working fine (provided someone remembers how it was built, configured or programmed), and the data in and of itself is perfectly good (as long as it can be kept up-to-date). But the old applications won’t talk to the new ones (or even each other), or the data format is not suited to new business needs or customer requirements.

Legacy systems require the most time and money to replace or upgrade. A colleague who works in financial services was recently bemoaning the costs being quoted to rewrite part of a legacy application – it seemed an astronomical amount of money to write a single line of code…

As painful as it seems, there may be little alternative but to salvage what data you can, decommission the software and throw it out along with the old mainframe it was running on!


Many data projects (especially in financial services) focus on reducing systems latency to enhance high-frequency and algorithmic securities trading, data streaming, real-time content delivery, complex search and retrieval, and multiple simultaneous user logins. From a machine-to-machine data handover and transaction perspective, such projects can deliver spectacular results – with the goal being end-to-end straight through processing in real-time.

However, what often gets overlooked is the level of human intervention – from collecting, normalizing and entering the data, to the double- and triple-handling to transform, convert and manipulate individual records before the content goes into production. For example, when you contact a telco, utility or other service provider to update your account details, have you ever wondered why they tell you it will take several working days for these changes to take effect? Invariably, the system that captures your information in “real-time” needs to wait for someone to run an overnight batch upload or someone else to convert the data to the appropriate format or yet another person to run a verification check BEFORE the new information can be entered into the central database or repository.

Latency caused by inefficient data processing not only costs time, it can also introduce data errors caused by multiple handling. Better to reduce the number of hand-off stages, and focus on improving data quality via batch sampling, error rate reduction and “capture once, use many” workflows.

Which leads me the third element of the troika – data governance (or the lack thereof).


In an ideal world, organisations would have an overarching data governance model, which embraces formal management and operational functions including: data acquisition, capture, processing, maintenance and stewardship.

However, we often see that the lack of a common data governance model (or worse, a laissez-faire attitude that allows individual departments to do their own thing) means there is little co-operation between functions, additional costs arising from multiple handling and higher error rates, plus inefficiencies in getting the data to where it needs to be within the shortest time possible and within acceptable transaction costs.

Some examples of where even a simple data capture model would help include:

  • standardising data entry rules for basic information like names and addresses, telephone numbers and postal codes
  • consistent formatting for dates, prices, measurements and product codes
  • clear data structures for parent/child/sibling relationships and related parties
  • coherent tagging and taxonomies for field types, values and other attributes
  • streamlining processes for new record verification and de-duplication

From experience, autonomous business units often work against the idea of a common data model because of the way departmental IT budgets are handled (including the P&L treatment of and ROI assumptions used for managing data costs), or because every team thinks they have unique and special data needs which only they can address, or because of a misplaced sense of “ownership” over enterprise data (notwithstanding compliance firewalls and other regulatory requirements necessitating some data separation).


One way to think about major data projects (systems upgrades, database migration, data automation) is to approach it rather like a house renovation or extension: if the existing foundations are inadequate, or if the old infrastructure (pipes, wiring, drains, etc.) is antiquated, what would your architect or builder recommend (and how much would they quote) if you said you simply wanted to incorporate what was already there into the new project? Would your budget accommodate a major retrofit or complex re-build? And would you expect to live in the property while the work is being carried out?

Next week: AngelCube15 – has your #startup got what it takes?

Defining RoDA: Return on #Digital Assets

How do we measure the Return on Investment for digital assets? It’s a question that is starting to challenge digital marketers and IT managers alike, but there don’t appear to be too many guidelines. Whether your social media campaign is being expensed as direct marketing costs, or your hardware upgrade is being capitalised, how do you work out the #RoDA?

In most businesses, measuring the expected RoI of plant or equipment is usually quite easy: it’s normally a financial calculation that takes the initial acquisition price, amortized over the useful life of the asset, and then forecasts the “yield” in definable terms such as manufacturing output or capacity utilisation.

However, when we look at digital assets, many of those traditional calculations won’t apply, either because the usage value is harder to define, or the benchmarks have not been established. Also, while hardware costs may be easy to capture, how are digital assets such as websites, social media accounts, software (proprietary and 3rd party) and domain names being reported in the P&L, cash-flow analysis and balance sheet?

Sure, most hardware (servers, PCs and physical networks) can be treated as capex (e.g., if the purchase price is more than $1,000 and the useful life is 2-5 years). But how do you make sure you are getting value for money – is it based on some sort of productivity analysis, or is it simply treated as fixed overhead – regardless of your turnover or operating costs?

As we move to cloud hosting and #BYOD, many of these assets utilised in the course of doing business won’t actually appear on the company balance sheet. Yet they will have some sort of impact on the operating costs. Most software is sold under a licensing model, where the customer does not actually own the asset. (But, if the international accounting standards change the treatment of operating leases longer than 12 months, that 2-year cloud hosting fee might just became a balance sheet item.)

I was once involved in the acquisition of a publishing business that was converting legacy print products to digital content. Not only did they capitalise (and amortize) the servers and the conversion software, they also capitalised the data entry costs (using freelance editors) to avoid the expense hitting the P&L. Nowadays, that’s a bit like putting the HTML coding team on the balance sheet and not the payroll…

In some cases, the costs associated with maintaining an e-commerce website or registering a URL, will remain as overhead or operating expenses. But over time, businesses will want to have a better understanding of their RoI for different online sales and digital marketing channels, especially if they have been investing considerably in their design, build and maintenance. Measuring online visitor data, customer conversion rates and average yield per sale, etc. are becoming established metrics for many B2C sites. Having a good grasp of your #RoDA may just give you a competitive edge, or at least provide a benchmark on effective marketing costs.


Smart Designs: 5 Trends for Digital Products

There are 5 key themes emerging in new digital products* that are grounded in the analogue world. It seems designers and developers are having to find ways to embrace analogue once more, and integrate it into digital solutions. While not everything old is new again, there are some distinct echoes of the past in many of these new developments.

Eno Hyde - Someday World

#1. Revivalism

Sony is reviving magnetic tape as a data storage medium – prompting some pundits to suggest that cassettes might be making a comeback. (Not if the participants in this video have their way…..) Interesting to note that tape storage is far more energy-efficient than traditional hard drive storage. And last week, Telstra announced the development of a major public Wi-Fi network which sounds like the stuff of the future, but looks back more than 25 years, with the launch of Telepoint services.

#2. Hybridisation

The combination of analogue and digital technologies** is not new (remember the Advanced Photo System and the Digital Compact Cassette from the 1990s?). But modern polymath Brian Eno and his latest musical collaborator Karl Hyde have just put out an iOS app that is designed to interact with the vinyl edition of their new album, “Someday World”. It’s not quite augmented reality, but the app uses that concept to project animated graphics to accompany the music when the user points the iPhone’s camera at the record label.*** This could just be the first example of making the vinyl record a digital artefact!

#3. Simulation

As someone who dabbles in iOS music apps (as well as beta-testing a few in my spare time), I have become used to replicating the analogue experience of old-school analogue synthesizers and drum machines on my iPhone and iPad. This has now been taken a step further with the launch of the iVCS3, an iPad version of one of the first portable analogue synthesizers from the late 1960s (an instrument made famous by The Who and Brian Eno, among others). A notoriously difficult piece of hardware to operate, it is almost the antithesis of digital predictability, yet makes perfect sense in the digital context when simulated via the touch screen interface of the iPad.

#4. Sensorial

Despite some concerns about smart phone biometric security tools, the use of biometrics in banking is a near certainty. Sensory-based smart phone applications and add-on devices in the areas of health (diagnostics), the environment (air quality monitoring) and even cooking (taste tests) will soon be commonplace.

#5. Interconnectivity

The Internet of Things is starting to get interesting (beyond the fridge that can do your grocery shopping), especially when combined with robotics (although this April fool spoof from Sphero was probably a bit too real for comfort….). A couple of physical devices that could find extended use when hooked up to an internet connection are the Auug (featured in the new Apple ads) and the SwatchMate Cube (a winner in the 2013 Melbourne Design Awards). For example, the Auug could be used in remote control or simulation applications, while SwatchMate could be modified to analyse surface materials beyond their colour properties.


* I’ve been re-reading “Grounded Innovation: Strategies for Creating Digital Products” by Lars Erik Holmquist which has helped shaped some of my thinking on this topic.

** I thought I may have invented a new word as a possible title for this blog – Digilogue – until I came across this book. (But I took heart from the fact that the author, futurist Anders Sorman-Nilsson, like me also holds an LL.B.)

*** If you install the app and point your iPhone camera at the picture below, it should also have the same result as scanning the record label itself:



Amazon finally comes to Australia; local retailers still want government action on sales tax

A short (and seasonal) post this week, as everyone starts easing off for the holidays.

It may just be coincidence, but about the same time Amazon launched their new Australian website local retailers renewed their campaign to lower the $1,000 sales-tax exemption for online purchases from overseas retailers. And both events came at the start of the Christmas shopping season….

Obviously too early to say which way this will go, but here are a few personal observations:

First, the local Amazon site is limited to e-books, games and Android apps. So, no access to music, television or film content (digital or physical), no sales of print books and no Amazon marketplace. For these products and services, customers are directed to the US site. (Previously, the dormant domain name referred customers to the UK site.)

[Note: neither the US nor the UK sites allow overseas customers to buy mp3 content, but they can download digital music via Amazon’s AutoRip service when purchasing physical goods – confused? Me too….]

Second, prices for e-books on Amazon’s Australian site appear to be comparable to the US store, and presumably include local sales tax (GST) to keep on-side of the local real world and online retailers (as well as the ATO, of course).

Third, the general consensus is that if the $1,000 threshold was lowered or even abolished, the amount of sales tax to be collected would be more than outweighed by the additional costs of processing, administration and remittance (which would likely be passed on to local consumers at a “cost-plus” rate by overseas online retailers).

Fourth, many local retailers who voice their opposition to the $1,000 tax-free exemption fail to understand some of the reasons why local shoppers prefer to buy from overseas online retailers:

1. Price – even if overseas sales attract the 10% GST, in some cases this would still be cheaper than buying locally (especially so when the A$ was above parity with the US$). For example, from time to time, Amazon’s UK store offers free shipping on physical goods to Australia….

2. Choice – many products available from overseas online stores just aren’t available in Australia. This is primarily due to geo-blocking, confusion over local distribution rights, and simple lack of interest in stocking some items for the local market

3. Service – from recent personal experience, buying from a local online retailer took much longer than buying the same product from an overseas site, because the supply chain logistics were woefully inept.

[Note: As a separate but related example, I recently ordered a new iPhone 5S direct from Apple’s local website, and received it within 3 days, including a weekend; whereas my telco provider – which prides itself on its on-line business model and customer service standards – took more than 2 weeks to send me a new nano SIM card….. I had also been told by a couple of local Apple re-sellers that it would take 3-4 weeks to order the new phone, unless I took out a new mobile plan with them – which may say more about Apple’s trading policies than the resellers’ business operations.]

My advice to local bricks and mortar and even some online retailers is to look at their own limitations before insisting that the government amend the GST-free threshold on overseas online purchases.

As for Amazon, I wish them well in developing their local service. Much has been made of the stated intention to focus on Australian titles, and the opportunity for local authors to self-publish via Amazon. But already there have been some rumblings that this new site may cannibalize Kindle sales made via some of Amazon’s local retail channel partners.