newvantage partners llc is a management consulting firm founded in 2001 to advise fortune 1000 businesses on information management and strategy .

the firm is a leader in big data strategy consulting and executive thought-leadership .

newvantage was founded by industry leaders randy bean , ceo and managing partner , and paul barth , phd , supported by a group of business leaders , technologists , and academic leaders , including noted author thomas davenport .

bean writes a monthly guest column in the wall street journal , and contributes to business publications including mit sloan management review and harvard business review .

he is known for his observations on business culture and the emergence of big data .

newvantage partners host a series of executive roundtables with fortune 1000 c-level executives , and publishes an annual big data executive survey of c-level business and technology executives .

the company is headquartered in boston , ma , with operations in new york and san francisco .

newvantage partners was highlighted in the february 23 , 2016 wall street journal op ed article , “ the big-data future has arrived ” .

newvantage partners

newvantage partners <EOT> newvantage partners is based in boston with operations in new york and san francisco .

randy bean is ceo and managing partner of consultancy newvantage partners . you can follow him at @ randybeannvp .

about newvantage partners founded in 2001 , newvantage partners exclusive focus is the delivery of world - class expertise in data strategy and big data execution for fortune 1000 clients . newvantage partners is based in boston , new york , and san francisco , with operations in austin , tx and charlotte , nc . visit urltoken for more information .

find out more about the newvantage partners big data fast track here or request more information at info @ urltoken .

about newvantage partners founded in 2001 , newvantage partners llc is a world - class provider of data management and analytics - driven strategic consulting services to fortune 1000 firms , and is the industry leader in big data strategy consulting , thought - leadership , execution , and business value realization .

in a new q & a , newvantage partners cofounder randy bean reveals how veterans of big data management are dealing with the new era of big data .

newvantage partners has been featured in a number of recent articles and thought - leadership pieces on big data and issues in data management . recent articles include :

newvantage partners has released the results of its 4th annual big data executive survey , entitled “an update on the adoption of big data in the fortune 1000 . ”

newvantage partners hosted a series of big data executive thought - leadership roundtable discussions in new york ( march 1 , 2012 ) and boston ( january 25 , 2012 ) .

" this is the first and only end - to - end big data program in the industry today which moves a company from strategy and design through execution and implementation , ” said randy bean , ceo of newvantage partners . “ " sixteen leading companies have been able to accelerate business value by leveraging newvantage partners’ big data capabilities and expertise . ”

newvantage partners announces its big data fast track program , the result of three years of big data strategy and execution with more than a dozen fortune 1000 and industry leading clients .

mit sloan management review . interview with newvantage managing partners paul barth , phd and randy bean . " quick wins help avoid cultural obstacles on the path to value . " october 27 .

newvantage partners has been named an ibm big data ecosystem executive thought - leadership partner . randy bean , managing partner at newvantage , noted , " we are pleased to partner with ibm in educating the business community on the tremendous opportunities that big data provides to accelerate the time - to - answer from their information assets . we are honored that ibm , which is at the forefront of the big data revolution , has identified and selected newvantage partners to be a key partner in building out executive thought - leadership on big data . for more information , see urltoken .

last summer , newvantage partners held discussions with c - level executives ( chief data officers , chief information officers , chief technology officers , chief analytics officers , chief information architects ) as well as line - of - business heads and senior function heads at more than 50 large companies , most of them with more than 30 , 000 employees . executives from financial services firms represented about 50 percent of the respondents , but newvantage partners also interviewed executives from insurance , government and other businesses .

the newvantage partners’ third - annual survey targeted companies in the fortune 1000 and drew responses from 125 high - ranking executives in 59 such firms , including american express , aetna , fidelity investments , general electric , johnson & johnson , kaiser permanente , lincoln financial and wells fargo .

boston , ma - - ( marketwire - may 25 , 2011 ) - newvantage partners , a boutique management consulting firm that serves as trusted advisors and senior consultants to a blue - chip roster of fortune 1000 clients , today announced the expansion of its executive thought - leadership activities .

that growing importance , according to newvantage partners , is reflected in its finding that roughly two - thirds of the firms it surveyed have big data initiatives already up and running , with many respondents saying the information is leading to new strategic insights as well as faster and better - informed decisions .

" most of the companies spend over $ 1 billion a year in technology , " says paul barth , founder and managing partner of newvantage partners . " they aren ' t usually quick to respond , but when they do they have a lot more resources to put behind their initiative . "

newvantage has also recently published a series of executive perspectives authored by newvantage associates and partners including paul barth , phd , on information as a strategic asset , blaise heltai , phd , and former evp / e - business and internet strategy for fleet / bank of america , on mitigating business risk , and paul bergamo , former chief technology officer for liberty mutual and aetna , on data governance for the long - term .

newvantage partners surveyed c - level and other senior executives from across disciplines , but emphasizes the value of the findings to marketers . “big data initiatives may have their greatest impact on marketing effectiveness , ” says randy bean , managing partner , newvantage partners . “our big data executive survey indicated that marketing and sales were the primary drivers of big data corporate investments : 77 % of executives indicated that marketing and sales were the business functions driving big data investment ; 87 % noted the need to accelerate time - to - answer for critical business questions . cmos stand to be the greatest beneficiaries of big data . ”

boston , ma - - ( marketwire - nov 14 , 2011 ) - newvantage partners , a boutique management consulting firm to fortune 1000 clients , and early pioneers in the field of " big data , " today announced plans for additional " big data " thought - leadership activities in response to fortune 1000 client interest .

about newvantage partners newvantage partners ( nvp ) is a boutique management consulting firm founded in 2001 and serving as trusted advisors and senior consultants to a blue - chip roster of fortune 1000 clients . nvp helps clients mitigate the risk of large - scale business and technology transformation initiatives by providing a critical link between business and technology strategy and capabilities . newvantage is recognized for an innovative delivery model - - a core team of former c - level business and technology executives , industry advisors , and subject experts , augmented by an extended subject - matter network . newvantage is focused on up - front planning to mitigate project risk - - current state assessment , future state design , business roadmap and business case , and follow on requirements , capabilities , and business architecture . clients include fortune 1000 financial service leaders , and business and technology executives across industries with responsibility for customer - facing , back - office , and shared - service functions .

) - - newvantage partners , a management consulting firm whose exclusive focus is the delivery of world - class expertise in data strategy and big data execution for fortune 1000 clients , has released the results of its 4th annual big data executive survey , entitled “an update on the adoption of big data in the fortune 1000 . ”

large enterprises , especially financial services firms , are adopting big data at a more rapid pace than expected , according to a recently released study by big data consulting firm newvantage partners . but how are organizations using big data to make better business decisions ? success , it turns out , is as much an art form as it is a technology solution .

“big data was a new topic just a few years ago , with many companies grappling with its role in their organization , ” randy bean , ceo and co - founder of newvantage partners , said in a statement accompanying the survey . “now , big data is largely viewed as a mainstream activity that is an integrated component of how they do business . ”

it seems that there ' s as much substance as there is hype around big data . according to the study “big data executive survey 2013 : the state of big data in the large corporate world , ” by consultancy and analyst firm newvantage partners , 91 % of fortune 1000 senior executives report that their organization has a big data initiative in place , or is planning one . of those respondents , 60 % say that they ' ve implemented a big data initiative . and 32 % of executives note that their firm ' s big data initiative is already operationalized across the organization . newvantage partners defines big data as “new database management and analytical approaches developed for analyzing , storing , and manipulating large or complex data . ”

boston , ma - - ( marketwire - mar 14 , 2012 ) - newvantage partners , a boutique management consulting firm that advises and provides executive thought - leadership to fortune 1000 clients on big data and enterprise data management , has announced an expanded range of big data executive thought - leadership activities and initiatives , in response to fortune 1000 client and industry interest .

“big data has reached a point of mainstream adoption within fortune 1000 firms , ” said randy bean , ceo and managing partner of newvantage partners , who also writes a column on big data for the wall street journal . “this survey has over the past 4 years reflected the evolution of executive perspectives as firms have come to terms with the opportunity represented by big data . ”

" newvantage partners big data solution center provides a data & analytics lab to evaluate new technologies and solutions approaches” said avi kalderon , nvp partner and head of the big data solutions center . “we accelerate analytics to support established use cases , enable ideation and validate new business strategies through data science , and deliver analytics - as - a - service to enable rapid insights . "

newvantage has recently authored a series of articles and been the subject of an in - depth interview on the topic of " big data . " recently published articles include :

the big data executive survey 2014 report released monday by the consultancy newvantage partners found that firms now investing more than $ 10 million on big data projects expect their spending to rise in a range of 35 % to 75 % by 2017 , while companies that already have a considerable outlay of more than $ 50 million expect spending to grow by 6 % to 28 % in the same period .

for the third year in a row , newvantage partners has conducted a survey of fortune 1000 senior business and technology executives regarding their companies’ investments in big data . this annual survey ( see my summary of the 2013 report here ) provides a unique glimpse into the big data initiatives of large enterprises and how fast they have adopted the set of technologies , processes , and skills associated with it .

traditionally , companies have spent 80 percent or more of their time on the third and fourth steps , newvantage says . but big data solutions offer up new ways of approaching these steps .

" we intend to increase the number of dinners we hold each year , " noted bean . newvantage will be hosting its next dinner on june 16th at l ' espalier restaurant in boston .

newvantage is also expanding its public speaking activities . newvantage managing partner and founder paul barth , phd , was selected as a panel moderator for the north american financial information summit 2011 that was held in new york city , ny on tuesday , may 24 th , and will be a featured speaker at the american payroll association conference in salt lake city , ut on friday , may 27th .

about newvantage partners decades before the advent of big data , there were a body of business leaders , technologists , and academic thinkers engaged by the prospect of leveraging information to help companies gain critical business insights that would lead to more informed business decisions . these individuals were pioneering new information - driven initiatives through database marketing ( later crm ) , introduction of the internet as a new business channel , business process optimization , and customer analytics and business intelligence .

making this leap—integrating traditional analytics and big data while tearing down boundaries between it and business units—is a critical early step in creating organizational initiatives that leverage big data to affect the business , newvantage concludes .

newvantage partners ( nvp ) is a boutique management consulting firm established in 2001 , comprising many of the early business and technology pioneers of information - driven processes . today , nvp are trusted advisors and senior consultants to a roster of fortune 1000 clients , operating as a core team of experienced c - level business and technology executives , and industry advisors , augmented by subject - matter experts , working in small teams with executive management . more info : urltoken .

“piloting a big data solution for a top 20 pharmaceutical firm , we loaded over 1 billion claims and developed a persistence analysis that ran in 20 minutes vs . the five to ten days it took in their data warehouse environment , ” said rich sokolosky , partner and life sciences practice leader at newvantage partners . “after that success , we implemented a solution across eight therapeutic areas that is now being used as the model for their global big data initiative . ”

newvantage also announces the expansion of its publication activities . newvantage associate kate pugh has published her latest book on knowledge management , sharing hidden know - how . " this book addresses one of the time - honored problems in organizations , " notes professor tom davenport . " how do you get people with experience , solutions , and knowledge to share them effectively with those who need these valuable assets ? "

about newvantage partners decades before the advent of " big data , " there were a body of business leaders , technologists , and academic thinkers engaged by the prospect of leveraging information to help companies gain critical business insights that would lead to more informed business decisions . these individuals were pioneering new information - driven initiatives through database marketing ( later crm ) , introduction of the internet as a new business channel , business process optimization , and customer analytics and business intelligence .

newvantage has announced plans for an expanded series of executive dinner and breakfast discussions focused on the topic of " big data " and issues of business process efficiency and marketing optimization , in response to fortune 1000 client demand .

newvantage partners ( nvp ) is a boutique management consulting firm established in 2001 , focused on " big data " and enterprise data management . the firm is comprised of many of the early business and technology pioneers of information - driven processes . today , nvp are trusted advisors and senior consultants to a roster of fortune 1000 clients , operating as a core team of experienced c - level business and technology executives , and industry advisors , augmented by subject - matter experts , working in small teams with executive management . more info : urltoken .

most executives responding to the 2012 newvantage survey cited “the need to integrate a greater variety of data sources” as their main focus with big data . findings of the 2013 survey show that a nearly equal number of respondents cite variety ( 82 % ) , volume ( 81 % ) , and analytical velocity ( 87 % ) —which newvantage defines as the speed by which organizations can obtain answers to critical business questions—as the primary benefit they expect to gain from their big data initiatives .

newvantage helps clients execute on their big data strategy , from planning and design , through program implementation and execution . the firm’s core areas of expertise include data strategy , data governance , data architecture , data integration , and business alignment and adoption .

newvantage is known for its executive thought - leadership roundtable dinners and the expertise it brings to clients through executives who have " sat in the chair . " " the response to our executive roundtable dinners has been very gratifying , " commented newvantage managing partner and founder randy bean . " we have been organizing these dinners for several years to bring together a very select community of c - level executives , noted authors , and thought leaders . we are expanding the frequency of these dinners now in response to the demand . "

“it should be noted , ” the newvantage partners report state , “that the level of skepticism [ about big data ] has dropped significantly since we first conducted this survey in 2012 . ” indeed , most of the executives surveyed ( 74 % ) now believe that big data warrants “serious attention” and only 3 % call it “same old stuff . ” but the level of comfort with the term itself is still very low . most respondents said that the term is not helpful in focusing a serious discussion about the value and benefits of big data to their organizations and 83 % of executives claim to dislike the term big data ( 53 % ) , or find it to be overstated ( 30 % ) .

fortune 500 companies are rushing to make big data investments . who is leading this charge ? what are they doing ? what are they trying to avoid ? a recent survey of c - level and function heads from fortune 500 companies offers a unique glimpse into how the captains of industry are thinking about big data and how their companies are changing because of new insights gleaned from big data analyses . randy bean , a coauthor of the survey and cofounder of newvantage partners , which sponsored the study , sat down with david kiron , executive editor of mit sloan management review ' s big ideas initiatives , to discuss how top executives at some of the largest companies and organizations in the united states are managing big data .

newvantage recently completed a series of summer and fall executive thought - leadership events , featuring a select group of c - level business and technology executives , and noted business authors . the dinner and breakfast events held on june 16 , july 20 , july 27 , september 22 , and october 19 in boston and new york , including noted industry thought - leaders :

one big advantage to this type of analytics is the shortening of the " time to answer " ( tta ) , according to paul barth , founder and managing partner of new vantage partners , a boutique information management and analytics consulting firm . the queries , or models , that used to take data scientists months to build in order to answer forward looking business questions about supply chain or production schedules can now be done , in some instances , in hours , and in bulk .

as the saying goes , you’re known by the company you keep . in our case , our extended community of contacts , what we call the emerge collaborative , is an additional value we offer our clients . we are plugged into the business community , from the venture capital community to publishing houses to dynamic ceo’s and cio’s and complementary marketing , design and advertising firms , both domestically and internationally . our business partners form a collaborative community that thrives on networking and relationship brokering . we make this a way of interacting with everyone in our community .

“one financial services firm , looking for a fast and efficient way to look at data across their systems to find profitable opportunity , came to us for a big data solution , ” said chris pierpan , chief solutions architect and senior principal at newvantage . “not only were we able to build an integrated analytics environment in a week , we built their self - service analytics capability in less than four weeks . now , they can access data in hours instead of months . ”

" the " new " and the " known " are not islands ; they must be symbiotic systems connected to and feeding each other , " newvantage says . " " new " analyses require rapid access to all the " known " data representing the reality of today ' s business . conversely , there must be a disciplined approach to promoting new insights , data and models to evolve the " known . " without this linkage , the systems diverge into incoherence that does not reconcile or scale . "

" emerge pr has been an excellent partner for newvantage . we face a distinct brand challenge – maintaining our executive thought leadership position , which is relationship driven – while expanding awareness of our firm to executives beyond word - of - mouth and personal referrals . emerge has helped us work with the press and analyst communities to extend the reach of our executive thought leadership message in a way that is tasteful , discrete , and consistent with our role as trusted advisors to fortune 500 c - level management . "

" while this may seem wasteful , it eliminates two important delays : writing programs to select just the data needed , and going back to the source systems multiple times as new insights generate new questions that need new data , " newvantage says . " building traditional data marts and data warehouses is extraordinarily complex and costly . the broad range of open source offerings coupled with flexible , scalable grid systems create an environment that not only drives down costs , but also offers the potential of decreasing query times exponentially . "

" there is a growing interest on the part of the fortune 1000 clients we advise to understand how they can harness ' big data ' to gain business insight , " noted newvantage managing partner randy bean . " fortunately , as early business and technology leaders and pioneers in information - driven processes , we are able to bring some valuable and practical perspectives to bear based on our long - history helping company ' s leverage their information assets . hopefully , we can help others avoid the pitfalls and realize value from their initiatives quickly . "

unlike traditional relational databases , big data platforms allow analysts to organize , clean and integrate data selectively , ignoring records and fields that are not the current focus of analysis . this is a significant departure from data warehouses , where a great deal of effort is spent on data engineering to make sure it ' s production - worthy before it ' s released to users . newvantage notes that by deferring full data engineering , big data platforms accelerate tta during discovery - oriented analysis and eliminate the engineering effort on data that doesn ' t deliver value .

respondents gave a number of reasons for their investments in big data , from reducing risk to creating higher - quality products and services . but two reasons were clear leaders : achieving better , fact - based decision - making and improving the customer experience . of course , these are leading reasons for investments in traditional business intelligence ( bi ) analytics , too . barth said that when newvantage dug deeper , the real " quantum leap " for companies when it comes to big data is accelerating the speed at which they can get to a decision , or time - to - answer ( tta ) .

" emerging technologies and methodologies—including hadoop , cloudera , database appliances , accelerators and self - learning and genetic algorithms—can dramatically reduce tta , " newvantage adds . " the key to streamlining the time from a corporate question to game - changing business insight is to right - size the approach to analysis : rapid iterations during discovery and rigorous engineering into production . strong governance and oversight of known data capabilities must coexist with agile data analysis that paves the way for new data discoveries . enterprises with the capability to create an ebb and flow between dynamic discovery and scalable execution position themselves for sustainable success and dominance over their competitors . "

bean noted that each dinner is limited to twelve guests who participate in a roundtable discussion . recent dinners have included chief marketing officers , chief information officers , and line - of - business presidents and executive vice presidents representing firms including aetna , bank of america , bank of new york , citigroup , fidelity investments , the hartford , ing , marsh mclennan , metlife , putnam investments , thomson reuters , state street bank , and wellington financial , as well as noted thought leaders and newvantage advisory board members tom davenport , professor at babson college and best - selling author of competing on analytics , and lynda applegate , martin marshall professor of business administration at harvard business school and chair of the hbs entrepreneurial management program .

big businesses are adopting analytical solutions to capitalize on big data , a new survey finds .

according to the survey , 68 % of executives expect their organizations to invest more than $ 1 million on big data initiatives in 2013 . the percentage of companies investing more than $ 10 million is expected to rise from 19 % in 2013 to 50 % in 2016 . additionally , the number of organizations investing $ 50 million or more is projected to increase from 6 % in 2013 to 14 % in 2016 .

respondents perceive executive sponsorship as the most important factor to ensuring successful adoption of big data within a large corporation . in fact , 83 % of executives cite this as a critical factor .

respondents ' attraction to big data is primarily its likelihood to help them make better , fact - based decisions . additionally , these executives cite as valuable the ability of big data initiatives to accelerate the speed with which their organization can gain insight , thus answer critical business questions . as a result , 87 % of respondents consider the accelerating time - to - answer as an essential metric for successful big data initiatives .

nearly half of respondents ' companies ( 48 % ) are considering or appointing executives to new roles , such as chief data officer . another 26 % already have a chief data officer in place . and 42 % is launching change management initiatives like new processes and updating organizational structures to support the adoption of big data .

a significant number of respondents note that their organizations are implementing database management technologies such as aster , hadoop , and netezza , as well as analytics solutions such as revolution r , sas , and tableau , to sustain their big data initiatives .

urltoken delivers business - building content in print , online , via email and social media . stay up to date by subscribing , or manage your subscription here . put powerful urltoken content to work for you . subscribe today !

adam burden , edwin van der ouderaa , ramnath venkataraman , tomas nyström , and prashant p . shukla

this is a summary of the full article . to enjoy the full article sign in , create an account , or buy this article .

we conducted the survey in june and july of 2012 . it came out of our executive thought leadership breakfasts and dinners , where we bring together cios , ctos , chief marketing officers and the chiefs of analytics from fortune 500 companies .

the survey was biased to organizations that have participated in our executive discussions . about 53 % were financial services firms , which include banking , asset management firms and major credit card companies . another 19 % were insurance firms . if you put those numbers together , almost three - quarters of survey participants were from some form of financial services firms .

all of the respondents were senior executives in the c - suite or a level or two below . so , this was a unique perspective on what top executives in very large companies are thinking about big data . we had over 50 senior executives complete a very detailed set of 65 questions .

part of the motivation for doing the survey was that we had heard from many of those surveyed that they wanted to know what their peers were doing in terms of big data . was big data a real phenomenon across their industry ? what kinds of investments were organizations making in big data ? what type of applications were they looking at ? and did they have people with the right skills to execute big data initiatives or did they need to find people with these skills ?

one was the promise of improved data - driven decision making . everybody agreed on that in principle .

3 free articles per month , $ 6 . 50 / article thereafter . free newsletter .

david kiron is executive editor of mit sloan management review ‘s big ideas initiatives .

we answer three questions about the findings in our recent big idea initiative research report , “the talent dividend . ”

increased access to data can be turned into better decision making by focusing on both the production and consumption of analytics .

artificial intelligence helps doctors make better diagnoses . it can do the same for corporate leaders .

copyright © massachusetts institute of technology , 1977 - 2018 . all rights reserved .

this is such an amazing article , with a depth of context not found in your typical media article on the subject . great job !

i loved this article also grabbed few potential ideas featured over here . i believe big data is not just a big idea for tomorrow . it is an absolute change in company’s strategy for serving customers , managing operations , and business models . big data analytics can be used to forecast or avoid many types of failures and , to reduce big risks . cio’s must consider certain tools before jumping into cloud computing , where risks are often equal to the rewards .

you must sign in to post a comment . first time here ? sign up for a free account : comment on articles and get access to many more articles .

firms currently investing more than $ 10 million on big - data projects expect their spending to rise 35 % to 75 % by 2017 , a survey finds .

senior executives at u . s . firms are betting big on big data — projecting much higher spending on collecting and analyzing high - tech information over the next three years and increasingly hiring chief data officers to oversee that work , according to a new survey .

one key finding that spotlighted the trend : 43 % of the executives said their company now has a chief data officer , up sharply from 19 % as recently as 2012 and 26 % last year .

the consulting firm , which advises clients on using data and analytics to improve their strategy , said that 42 % of the executives who responded were either in the c - suite or business - unit heads , another sign of big data’s surging importance .

the survey findings dovetail with other recent reports that big data — collecting and analyzing the flood of fast - moving new metrics from internet usage , from other high - tech devices such as sensors and cameras , and elsewhere — is one of the fastest growing corporate line items . the technology research firm gartner said cumulative spending on big data from 2011 through 2016 will reach $ 232 billion .

“surveyed executives are bullish on the importance of big data to their businesses , ” thomas h . davenport , a professor at babson college who specializes in data and analytics , said in the survey’s forward . “only 3 % feel that it’s the ‘same old stuff . ’”

big data is clearly a disruptive technology , but using it successfully is as much art as it is science . the key is integrating big data with traditional bi to create a data ecosystem that allows you to generate new insights while executing on what you already know .

[ beware the 9 warning signs of bad it architecture and see why these 10 old - school it principles still rule . | sign up for cio newsletters . ]

and these large enterprises see value in big data and are bringing their resources to bear . barth says 85 percent of respondents reported they already have big data initiatives under way .

" over 75 percent were investing over $ 1 million a year already , " barth says . " twenty - five percent were investing over $ 10 million a year . there ' s a real commitment to using this technology for one program or a series of programs . they made it absolutely clear they could not do the programs they were doing without big data . "

" if your time - to - answer is 30 minutes in one case and 30 seconds in another , it really changes your business processes , " barth says . " it makes you much more effective as a business analyst . "

data warehousing and business intelligence are critical to business success . take this online course and get fluent with the fundamentals .

by using new big data technologies , organizations can answer questions in seconds rather than days and in days rather than months , barth says . this acceleration , in turn , allows businesses to answer questions that have resisted analysis , develop test and learn processes that quickly adapt to the market and automate complex workflows .

however , reaping the benefits of accelerated tta requires following a careful process based upon a clearly defined and governed relationship between big data and traditional analytics solutions .

" historically , there has been much talk about the difference between traditional analytics and big data , and organizational responsibility for each within an enterprise , " the report says . " the survey , however , shows the two are becoming closely intertwined and must work together to deliver the promised results of big data . further , breaking down organizational boundaries and creating close integration between it organizations and the business units is a critical step for any organization hoping to build a winning strategy for big data . "

" data management and analytics have often resided in different parts of the organization , " the report adds . " it departments usually controlled the data and analytics was conducted in either a special group or within a business unit . this is contrary to the entire principle of big data and the survey confirms that organizations understand close integration is necessary . sixty - five percent say , " big data is an integral part of data management , " and 68 percent further felt that " big data is part of the advanced analytics toolbox . "

" integrating real - time , full analytic capabilities into the business and operating units will enable the type of quick reactions to key business questions and challenges that can build competitive advantage and improve performance , " the report says .

" think about your data and data quality as having different stages that we call bronze , silver and gold , " barth adds . " data in your data warehouse is gold . when you go to that gold source , you know you ' re getting data that has been really worked through . but what if data is also available in a raw form and you can get it to me in a week or a month , if you can dump all the data in one place and organize it just a little bit ? the data is useful before it ' s perfect . "

the idea then , barth says , is that big data platforms become one piece—albeit an important one—of a data ecosystem that is designed to constantly look for new insights into customers , markets , products and risks , while at the same time building upon what is already known . in other words , pursue the " new " while operating on the " known , " a healthy , continuous improvement model .

think of it this way : whether it comes from big data or traditional analytics , the important thing is to provide valuable answers . the value of an answer , barth says , is based on its accuracy and the speed with which it can be delivered . to get an accurate , speedy answer , it ' s important to ask the right questions . and that ' s where big data comes in : it ' s about pursuing the " new . "

" the art part of big data is associated with discovery and explanation , " barth says . " you ' re looking for something you can ' t quite articulate . there ' s a phase of analytics that ' s exploration and discovery , in which you ' re generating hypotheses . then comes modeling and application . in my view , traditional bi tends to be really far down the road after you understand the underlying analytics and correlations that relate to an issue you ' re seeing . the phase in which you don ' t understand it , the discovery phase , is where big data is useful . "

first and foremost , because of the relatively low cost and high capacity of big data platforms , organizations can load all of the data from their source systems rather than choosing particular data for the question at hand .

for instance , barth points to a large financial services firm that wanted to perform multi - channel pathing analysis of its customers to understand which elements led to a sale and which led to attrition . to do so , the firm needed to integrate six months of session data with other channel data . the first attempt , using traditional relational databases , took tens of thousands of lines of sql code and the firm soon realized that it could only afford to access six days worth of data rather than six months . the firm abandoned the attempt after calculating that the effort would take weeks .

" in a big data environment , they were able to write and execute it in under 100 lines of code , " barth says . " they executed it in less than 24 hours , processing hundreds of terabytes of session data . the analysis was on data they already had ; they had it inside the bank . they just wanted to know what their own customers were doing on their own channels . this really unlocked that visibility into the way their bank was running . "

but the key , barth says , is to take it one more step . once you understand what you ' re seeing , you can develop a model that explains it and metrics to measure your execution in improving your business against that model . that ' s where traditional bi comes in .

thor olavsrud covers it security , big data , open source , microsoft tools and servers for urltoken . follow thor on twitter @ thorolavsrud . follow everything from urltoken on twitter @ cioonline and on facebook . email thor at tolavsrud @ urltoken

thor olavsrud covers it security , big data , open source technology , microsoft tools and servers for cio . com .

what are you doing to accelerate it agility ? learn about the it model that serves as a catalyst for digital transformation .

unlock the potential of your data . how well are you harnessing information to improve business outcomes ? a new cio playbook will help .

imagine for a moment the classic i love lucy scene that takes place in kramer’s kandy kitchen . lucy and ethel are trying to wrap candy but the conveyor belt is delivering it too quickly . some of the candy gets wrapped , some of it gets through unwrapped , and some of it ends up hilarious places better left unmentioned . and then venerable character actress elvia allman yells , “speed it up a little ! ”

now imagine that you’re lucy and every piece of candy is actually a piece of data , coming at you with overwhelming speed . it’s a horrifying thought to have to wrap it – or , in this case , identify it and figure out whether it’s important enough to analyze , not to mention determine how it should be analyzed . but potential scenarios exist in real 21 st century life : analyzing security data , video feed data , health - care data . any scenario where finding an answer faster will fight crime , reduce costs , or save lives .

ever since i wrote about the concept of fast data for computerworld a few months ago , i’ve found the topic fascinating . that’s why i was so gratified to see it pop up on more radar screens recently .

perhaps the most relevant approach to fast data appeared recently in the wall street journal , where consultant randy bean noted , “in contrast [ to big data’s historical focus ] , fast data is about “data in motion” and immediate response and action . it’s the velocity component of the big data triad . while large corporations have been focused on the variety and volume of data they manage , fast data applications are being developed to seize on the opportunities presented by data velocity . ”

emagine ceo david peters wrote in techradar earlier this month about the importance of fast data for communications service providers – some of which applies to other industries – but since his company sells a “real - time event decisioning platform , ” consider the source .

, nicely encapsulating the issues behind real - time data collection , processing , and insight availability . rather than whitewashing the concept , de montcheuil addresses some of the pitfalls as well :

beyond the technical challenges of running smaller / faster batches and streaming / messaging technology , the major difficulty of real - time business intelligence resides in the differences in velocity between the different types and sources of data . because not all data enters the data warehouse / mart / lake at the same time , and because it is not refreshed at the same frequency , updating the data analysis and reference structures becomes very challenging . ”

infoworld continued with the question of the viability of fast data last week in matt asay’s look at the streaming future of big data , thanks to new technologies such as spark , storm , and kafka ( man , what a literary nightmare those names connote ) . though much of his article is a volley between those who believe in batch and those who prefer real - time , he rightly notes that there’s room for both . as one of his sources says , “batch isn’t going anywhere as there will always be a place for large - scale analytics with gobs of data . ” the same source , he adds , has “a ton of interest in streaming analytics [ but adds that ] it’s ' way too early to say ' how it will all shake out . "

it’s universally acknowledged that big data is now a fact of life , but while large enterprises have spent heavily on managing large volumes and disparate varieties of data for analytical purposes , they have devoted far less to managing high velocity data . that’s a problem , because high velocity data provides the basis for real - time interaction and often serves as an early - warning system for potential problems and systemic malfunctions .

moreover , data proliferation has been accelerating . emc recently reported that data volumes can be expected to double every two years , with the greatest growth coming from the vast amounts of new data being produced by intelligent devices and sensors . oracle president mark hurd has predicted that the number of devices connected to the internet will grow from 9 billion to 50 billion by the end of this decade .

what makes device data , sensor data , and other forms of “fast data” distinctive is that , unlike historical data , it is live , interactive , automatically generated , and often self - correcting . historical data is used to identify patterns that inform future decision - making , while fast data is designed for real - time decisions and real - time responses . think of fast data as the continuous processing of events and data in order to gain instantaneous insight and take instantaneous action .

while fast data is not really new , it has been largely restricted to a couple of high - value uses : complex event processing ( cep ) activities that operate on event streams , examples of which include algorithmic trading and fraud monitoring in financial services ; and event correlation , which includes the systems that monitor and manage complex industrial components such as jet engines .

so , what has changed ? first , the explosion of fast data has driven the demand for instant action . second , innovators in social media and services like uber have shown that businesses can be differentiated based on their ability to act on data instantly . uber knows where you are , where you’re going , and how you will pay to get there because it can capture , analyze , and act on data in real time . the availability of lower - cost memory is making fast data accessible for a broader set of applications , including :

organizations that know how to use fast data will be more nimble , adaptive , and competitive . what can companies do differently to prepare for the opportunities created by high - velocity data ? here are a couple of suggestions :

automate decision - making to increase customer engagement . monitor customer activities to identify—and respond to—patterns , thresholds , and triggers . one major retailer is seeking to engage with customers in real time while they are online , but is hampered by traditional systems and batch processing environments . they are now creating an environment that marries customer and inventory data with streaming data so they can , for example , report to the customer whether a product is in stock and address any inventory gaps immediately .

integrate machine - generated data to personalize interactions . emc projects that soon nearly two - thirds of all data will be generated by machines , not people . that presents a technical challenge : how can companies capture and analyze many flows of data concurrently ? the good news is that next - generation data systems that prioritize real - time data are in the early stages of adoption . by integrating new sources of machine - generated data , and combining it with traditional data sources , firms can further personalize their customer interactions . ericsson , the mobile broadband company , has developed real - time visibility into its system performance , using device and user data as it is generated . this enables ericsson to identify and improve slow performance as needed and to serve customers with programming tailored to them .

firms that take these steps now will be well positioned to improve their operations and better serve their customers using high - speed data .

copyright © 2018 harvard business school publishing . all rights reserved . harvard business publishing is an affiliate of harvard business school .

big data has gone mainstream with 67 % of executives reporting that they have big data initiatives running in production , up from 32 % last year . a remarkably rapid adoption rate , since only two years ago 85 % of the respondents had a big data initiative “planned or in progress . ” while a third of respondents in 2012 had big data investments of $ 1 million or more , 45 % reported big data spending of greater than $ 10 million this year , and 74 % estimated that their companies will spend that amount in 2017 . 82 % of the executives surveyed said big data is already integrated into the mainstream of their organizations while only 12 % said it is managed as a separate environment .

who leads this new mainstream corporate activity which 82 % of the executives surveyed view as being highly important or mission - critical within their organizations ? 37 % of executives reported that the primary executive sponsor is the ceo , coo , or business president , stressing the importance of sponsorship from the top of the organization . and who does the executive sponsor turn to for big data accountability ? leading the charge in many situations is the chief data officer ( cdo ) with 43 % of executives reporting that their organization has defined and established such a role . this is a sharp jump from previous years - - in 2012 only 19 % of executives reported that their organization had established a cdo function , growing only slightly to 26 % in 2013 .

the business function that is the primary driver of investment in big data for 36 % of the respondents is sales and marketing . but other functions also lead big data initiatives , including those focused on internal activities : risk , security , regulatory , and compliance ( 29 % ) ; r & d and new products ( 21 % ) ; it and operations ( 10 % ) ; and finance , hr and administrative ( 4 % ) .

what are these diverse functions expecting from their investments in big data ? three quarters of executives cited greater insight and learning , the ability to answer business questions faster , make informed decisions faster , and accelerate speed - to - market as the primary business driver for their big data initiatives .

as tom davenport writes in his introduction to the report : “everything is sunny in the world of big data… except , its name ! perhaps never has there been such enthusiasm about a business topic , and less satisfaction with the name of it . . . [ big data is ] clearly not just hype , and it’s not just experimental . companies are getting real value from it in production applications . now if we could only figure out what to call it ! ”