Big Data, Big Rubbish?

Answers and Tips to prevent your Big Data Deployments from becoming Big Rubbish

Big Data has been trending in social media for years now. The concept is not a new one and it includes a set of different technologies that provide different applications for different industries. Leveraging from diverse Big Data sets that are available internally or externally, Big Data is characterized by the variety, the volume and the velocity that is produced and therefore, cannot be processed or analysed by more conventional methods such as Databases.

However, there is still a lot of confusion around the mix of technologies that are required, with productised solution platforms currently available in the market, still at their very early stage and with a total cost of ownership in order to gather, store, maintain and analyse Big Data still very high.

So, let’s agree on one thing; Big Data still remains a hype and a buzzword rather than a feasible solution option, especially for organisations that are not large enough to invest big on Big Data.

On the other side, and by all means, Big Data investments will certainly be proven to be beneficial in the near future, especially when combined with analytics capabilities and a clear vision for the future. This article provides answers to key questions and tips for creating a clear strategy to deliver Big Insights and prevent Big Data investment from becoming Big Rubbish.

• “76% of all organizations are planning to increase or maintain their investment in big data over the next two to three years.”
• “52% of enterprises globally see Big Data as an opportunity, and only 4.5% see it as a threat”
• “Despite widespread optimism regarding Big Data’s potential, only 23.5% of enterprises have a clear strategy today.”

Source: Forbes

What should a Big Data solution be?

As for most problems in today’s complicated world, there is not a single solution that fits all problems. You will need to identify the optimal mix of technologies and create a scalable, flexible environment that will serve your needs and plans for your future and align your business objectives and strategic priorities.

“Forrester’s TechRadar™ methodology provides an evaluation tool to help you identify, evaluate and clarify 22 different technologies, all related with Big data.“

Source: Forrester’s TechRadar™

But, as Big Data will continue to grow and become even Bigger, Advanced Analytics always needs to be the core component in any Big Data solution in order to effectively operationalise and optimise data related processes and derive actionable insights and business value.

Big Data without Advanced Analytics capabilities will simply result in data overload, noise and unavoidably Big Rubbish. It is not the increasing volume of Big Data that will answer your business problems or the increasing computational power or storage capacity, but the analytical algorithms that will allow you to mine Big Data sets and discover hidden patterns and associations.

Useful Tips for Transitioning Data Insights into Business as Usual are provided here. Guidelines on how to plan your Analytics Strategy are provided here.

What type of Big Data do I need?

Again, there is not a single golden rule that should always be followed, as the answer relies on the industry, the activities and the environment in which each organisation operates, and most importantly on the business problem that needs to be addressed.

Big Data sources vary in terms of structure, origin and nature. For example, for a financial institution that aims to improve customer service, Big Data can originate from structured CRM/Marketing, Sales and Transactional data sets, unstructured Social Media text and external Social Economic data from surveys. For a manufacturing company that wants to reduce production downtime, Big Data can originate from Asset Management Systems, Supervisory control and data acquisition (SCADA) systems, Inspections and Production yields, and also Weather Data and Internet of Things (IoT) Sensors readings from machines’ equipment.

Customers success stories on Big Data are available here and here.
To help you identify your Big Data Sources go here and here.

How Big should my Big Data be?

Analysing Big Data sources can result in Big Insights, however they can also result in Big Noise and Big Rubbish. A balance between Big Data sources and targeted, representative samples (Small Data), and also between traditional modelling techniques and artificial intelligence, deep learning and machine learning algorithms, should be maintained. Variable selection and sampling is still important for an accurate statistical model and access to Big Data is the key component for that. It will provide data scientists the required bandwidth to build differentiated analytical models that can address different business problems and reveal associations and correlations, while Small Data, or appropriate relevant data samples will help you to reveal the root causes and produce accurate predictions and targeted actions for future events.

What other things should I consider?

Even if Big Data is technology-intensive deployments, peoples’ skills are still and will remain important for deriving business value from data processes. They will prevent your Big Data efforts becoming Big Rubbish and will increase return on investment. Experienced data scientists with appropriate skills and training, combined with a project governance system will definitely add clarity to your efforts and create a competitive advantage for the future.

Information on the importance of Project Management in Analytics engagements is available here.
For the skills required to be a good analyst click here.
Help to identify your training needs is provided here.
For lessons learned during analytics implementations click here.

Version 1’s experienced consultants are on hand to help you understand your SPSS needs – from consultancy and training to finding the best software and license type for your analytical and usage requirements. Contact us to discuss your requirement and identify the best SPSS solution for you.