What the Intelligence Community Is Doing With Big Data

Description: What if the government could know the future?

It’s trying. Armed with billions of tweets, Google (GOOG) searches, Facebook (FB) posts, and other publicly available social-media and online data, the Office of the Director of National Intelligence is sponsoring research projects involving 14 universities in the United States, Europe, and Israel with the goal of using advanced analytics to predict significant societal events.

Source: Businessweek.com

Date: Feb 5, 2013


This is an ambitious goal. Even nerd savant Nate Silver has yet to predict the likeliness of a popular revolt toppling a government, a deadly disease outbreak, a sudden currency collapse, or war. Still, the recent explosion in data analytics ushered in by the social-media era holds huge promise for making increasingly accurate predictions about the future, says Thomas Malone, director of the Center for Collective Intelligence at Massachusetts Institute of Technology and author of The Future of Work. But, he cautions, “it’s not magic. There are a lot of things about the future that are essentially unknowable.”

Matheny, too, understands the limitations of predictive analytics. That’s part of the point. At the end of the government’s three-year research program, he says, they hope to know with more certainty just what is predictable—that is, what categories of events are relatively more knowable.  Read Rest of Story 

  Questions for discussion:

1. Are there dangers to our society as the result of various uses of Big Data?  Why or Why Not

2.  what are some positive opportunities for the use of Big Data?


16 thoughts on “What the Intelligence Community Is Doing With Big Data

  1. Abraham Sarthak

    The use of Big Data by the government has both positive and negative impacts on the society .The government can use these data to predict the riots or social gathering which can harm the society .They can be well prepared with it .As today internet has become a major source of communication between people so i feel in this regard that it would help the government a lot .But also it can have an adverse effect on the society as it can lead to wrong use of the information by the government, as they can use it to curb the oppositions attempt to through down the government .Today most of the large gatherings all around the world are planned on either facebook , google or any other social networking site .So it can use this to identify how many people are with the government and how many are against it , in way they can predict the future results of the election and be prepared according to it . So according to me the use of Big Data is critical issue in the society as it can lead to massive tranformations in it .But that totally depends on the intentions of the goverenment and the authorities using it.

  2. Antonia

    The only danger that I could see from the use of Big Data is human error. You can’t predict the future and by trying to might just set you up for failure. Say the government predicted that there was going to be a world war, so they mobilized their troops and got them ready. If this war was incorrectly predicted then another nation may see this mobilization as a threat and it may actually cause a war that was never supposed to happen in the first place. I think if something is supposed to happen it will, if there is going to be a deadly disease outbreak hospitals will know before Facebook (hopefully). I think it would be a waste of time, resources, and money. I don’t think if a group of people were trying to topple over the government they would go on Twitter or Facebook and tell the whole world about it; if they did they would be much of a threat because they obviously aren’t that smart. If they did try this I would hope they would be able to find alternative uses for the information they collect. They could use this information for statistics or maybe for solving crimes that have already been committed.

  3. Brandon Yadernuk

    There was a movie that came out a couple of years ago, staring Tom Cruise, about a task force in the future using a predictive means to stop crime before it even happened. This movie was called the Minority Report. Now I am not about to say that technology couldn’t come to this point, but there is the possibility that there are errors to be made. The use of technology to predicate patterns with people is a risky business. The dangers of calling something a crime before it happens also exist. Therefore, I do not think that Big Data is really ready to be used in this manner. People are not robots. They do not make predictable choices every moment of everyday. People are creatures of chaos, everything be do everyday changes our future.

    Now there may be some positive ways that big data can be used. In marketing, for example, to be able to give every website visited a more custom tailored feeling would be quite advantageous for both parties. So, I guess what I am trying to say is this. If a company can use big data to make my online shopping experience better, well then they should start doing that before trying to predict the population’s future. Or, they could always try predicting the weather, there is ALWAYS room for improvement there.

  4. Ollie Sherwood

    Are there dangers to to our society with the use of “Big Data”, well of course but isn’t that the case with every new and unknown technology. People were asking the same about cell phones when that technology started to hit the market. I see the real danger in the uncertainty that this technology bring with it. At the end of the day it will not be 100% accurate however someone will think that is and this is where the danger lies. Also we should be concerned with who gets this ad how it is used as well. The main pro that can be used from this type of new and changing technology is we would be able to learn from our past mistakes and maybe change the path that we are on for the better. This type of crazy technology could be very useful people within the force like army and the police force, but still here we should be careful to who gets to use this type of tech within their field. But the Calgary Police is already using a type of technology right now and that seems to be effective. As I see it the world of technology is rapidly changing and this type of tech maybe here sooner than we think and i can only see the pros.

  5. charleschow

    There always have some risks because the future is uncertain. If bad people develop the higher technology than the government, the big data might probably used for illegal activities. The most important thing for government is how to use this big data with safety. Otherwise, it might cost a big loss instead of gain any convenience. If the government officer use the big data in a wrong path ,it must cost a disaster for the countries.
    However, Government could be easier to general any gov analysis because it has a fast and easy way to collect data. Also, it could make the data transfer quicker and effectively. Moreover, it could bring a more stabilized society if the government use the big data in the right way because using the big data in security could reduce the crime and nature loss.

  6. Navreet Dhillon

    NOT POSSIBLE! I don’t believe that anybody or anything has the power to predict the future: be it science or astrology. We could do everything in our power but the unpredictable will stay unknown. Yet, with access to a lot of meaningful information and statistics, some things could be predicted. For example: Calgary police has a website to find out areas with high crime rates, hence they would try to patrol that area more often and hopefully be able to regulate and reinforce law. However, it can never be predicted that the resident of house number 101 would be murdered by his next door neighbor without invading the privacy of innocent people. Predicting natural disasters is also somewhat possible even though, Mother Nature will not shot an email before an avalanche. This arena of predicting the future is very ambiguous and claiming to do so would definitely need a lot of scientific back up to convince non- believers like me. Even if they crack the code, they need to make sure their security is top notch and ensure that an evil villain wants to rule the world doesn’t get access to the information. Also, what is the guarantee that the people who have access to this information would not misuse it? There are a lot of questions that need to be answered before figuring out the unknown.

  7. Dallon Martin

    There are some pretty cool things that can be done with the use of “Big Data”, but with that being said its almost like taking a shot in the dark on somethings. Being able to predict the future is a touchy subject because it would be hard to become 100% accurate in predicting what is going to happen because things could change, or someone could just change there mind.
    With that being said, using “Big Data” in this way doesn’t pose a threat to anyone, it could only help. the question that needs to be asked is are we just wasting money though paying someone to use this research that doesn’t really pay off. If there was a precise way to use this information it would be beneficial.
    As technology continues to increase Big Data is only going to get more and more useful.

  8. Kristian

    The ‘Big Data’ idea is one of particular interest to all. However, the thought of knowing that is unknowable is a lofty goal . No doubt however, big data has potential to predict many things, maybe on the lesser extreme (ie. market trends). On a micro level, big data can provide organizations with competitive advantage if used correctly. On the Macro level, there are events that can be predictable from social media, and can have a substantial impact on society as a whole (ie. Arab Spring). There is no doubt in my mind that the pros that come from big data will definitely outweigh the cons, the cons and potential problems being that of privacy and security. Big data should be used for of a control measure however, and not a predictor. As a society we can learn from past accomplishments and mistakes, and use big data to help us understand why something worked, or why it didn’t. Big data will need to have control measures, which may be a daunting task. Because of the almost infinite data ‘big data’ can obtain, policies are going to need o be put in place to ensure ethical and legal practices. A policy that will be tougher and tougher to make as data keeps exponentially increasing.

  9. Katie Guccione

    The idea of being able to use big data to stop or predict major world problems would be amazing, if it would actually work. By using this tools you will likely be able to track trends that happen before a disaster, and thus use these to hopefully be able to be one step ahead of the problem. Fortunately, i think this is too good to be true and i think that this will only do one thing and that is it will just add to peoples constant desire to be able to predict the future, control it and change it. It will overall cause a larger problem for society as a whole. This will only give the government a larger influence over us and allow them to have even more power over our everyday life. If they are able to ‘predict’ a world crisis they will allow us a small amount of information to leak out, but only enough to influence society as a whole into doing exactly what they want. Overall I think that we can learn from the past and that we should always keep what has happened to us in mind and how it was casued. I do not think that we should go as far as to try to control or predict the future, this will only allow for people to think that they have more control the world than they are already do.

  10. YWesley

    1-800-PHYSCHIC much! No way, the government is spending taxpayer’s dollars on a high-tech psychic carnival sideshow. My guess is the government can follow the progression of an outbreak of a deadly flu, up-to-date minute to minute on a natural disaster, or chitter chatter on potential threats of war or terrorism through social media network sites. Then follow those research findings by applying a contingency statistical formula to predict the next probabilities but it cannot and will not be able to predict the future. The government might as well hire a psychic on staff, if they are going to use the possibility of connections to animism. The problem with regular every day Joes using and sharing on the Internet is they can be influenced by imprecise information or fallacious occurrences. With the simple hit of the “share” button, without checking the facts of the claim or if it is someone’s personal opinion to a cause or an event in time. Where is the creditability in those scientific research methods? The government should not spend the taxpayer’s money irresponsibly but save our money and put it towards education or towards a school for gifted children. They should spend a minimal amount on posting a sign: Oracles, Prophets, Prognosticator, and Fortune Teller needed, apply within your crystal ball, if this is the route they want to follow through with.

  11. Colin

    One could easily state there are pros and cons to the Intelligence use of social media “big data.” The main goal of this using this data is to further refine what can be known and what is essentially unknowable. With such a massive amount of data to sort through it is hard to imagine that any one person would be adversely affected by this practice. However, if used in an unethical manner it can present a danger to individual privacy in the world. You can take the Arab Spring as an example of how social media affected those societies in a profound way. This gluttony of information is simply another tool that the Intelligence community is going to use regardless of whether people agree with it. The Patriot Act allowed the Government abilities to use information that were never heard of before 9/11. Information is truly one of the most valuable commodities in the world, so it is normal for Governments to want to utilize this information. If terrorist events or crimes can be prevented through the use of this information then that alone justifies its use. As with all new technologies there is the omnipresent danger that they can be used unethically. It is the age old question of weighing the benefits versus the downside of this data. In terms of the net benefit for protecting the safety of society, one could argue utilizing this data is essential.

  12. Brogan

    I do not know much about the risks of Big Data, but I think there are some definite positives which can make it very useful. I think predicting the future is not possible based on just google searches and Facebook updates, however if predictive analysis could give the right people a small insight into future possibilities, preventative (and perhaps extreme) measures can be taken. For example, if someone was able to predict that a nuclear plant was to explode, all the workers and surrounding citizens could be evacuated. If the result is that it does explode, potentially hundreds of lives would be saved. If the plant doesn’t explode, no harm done, and perhaps more research could be done to fine tune these predictions. However, completely freak accidents, I believe, are impossible to predict using data from social and internet sources. But I believe if Big Data could save even 1 life, I think more research should be put towards it.

  13. Daryabeygi, Mohadese

    The meaning of the prediction of the future would be more than just historical trend. Predicting the future depends on many unknown factors that the specialist based on their big Data might not be able to predict. There are lots of thing that can happen in the future that we did not experience in the past. For example, in 1980, people may not even have dreamt that they can contact face to face people who are far from them, thousands of kilometers. If we assume that they could predict some trend about our society, it would hurt their audience. For example, some people believed that on December of 2012 the world would end forever. It caused fear in many people but it did not happen. So, the results of this prediction would cause unnecessary pressure on society. In addition, they would spend lots of money for useless research or results.
    On the reverse side, we would have some useful information that would help out society to be more prepared for or prevent some harmful events such as the recession in 2008. The results from the research would help economists and management to look at the root of the problem and prevent it to happen again.

  14. ibrahim.akinola

    I believe that our society do not face any dangers as a result of the uses of big data; this is because big data benefits the society in a lot of ways; as stated in the article it is meant to help the police or government in able to predict events of the future so as to combat it before it happens, this technology can save millions of life in places before a natural disaster occurs and can also allow the system to get rid or change bad behaviours before the society feels the effect e.g. Connecticut elementary school shooting and John Holmes theater shooting.
    Some people might argue that it brings about danger to the society in form of privacy concerns, but I believe they are showing lack of intelligence and they should not turn a blind eye at it, instead they should see it from another perspective and how it can save the world from a state of calamity.
    The big data brings about a lot of positive opportunities like reduction in crime rates, saving millions from natural disasters and able to control individual bad behaviours before they get worst. Based on this new intelligence, it can actually bring the world closer into eradicating undesirable effects

  15. Josh

    Even though the article was published under a blog page entitled “The Very Near Future”, this story is kind of old news. Pre-crime technology has been implemented in the United States and around the world for at least several years now, with varying levels of advancement and success. In 2002, the film “Minority Report” was released, depicting a police-state society where criminals were apprehended before they committed a crime. The idea of this type of technology has been around for a while, but its implementation has been greatly sped up by the prevalence of online social media.
    Going as far back as the beginning of tyranny among men, despotic rulers throughout history have attempted to preserve their power by quelling opposition before it arose. The paranoia of Stalin led to the murderous widespread purge of government officials, generals, soldiers, and even innocent citizens he suspected of being potential saboteurs. Granted, in this article we’re not talking about anything this extreme. The reason I bring up this point is, because the future is ultimately unknowable, there is great danger in accusation and predictive action.
    I have no problem with police agencies using crime statistics and information systems to become more effective and efficient in their efforts to protect and serve the public. I do have a major problem with peoples’ rights, such as privacy and liberty, being threatened by the very organizations that are designed to preserve those rights. Part of the issue here is that most of this information is probably being collected in a completely lawful manner, and there is no way to stop that. Every day people willing post their own personal information, making the work of surveillance and investigation much easier for those with access to it. The main problem I have with the use of this information is when it threatens or denies the agency of individuals. People make poor choices every day, but there is no way to prevent them from doing so without taking away their individual freedom. This is the potential threat I see with ever increasing use of predictive crime systems.

  16. Marc

    1. Are there dangers to our society as the result of various uses of Big Data? There are always dangers and “unintended consequences” of every political decision or technology (like Nobel and the invention of dynamite). Big Data is scary because it’s not well understood, and it has global consequences.

    2. What are some positive opportunities for the use of Big Data? The predictive analysis the article talks about is pretty cool. If we can do the same thing with school shootings that we can with hurricanes (the ability to see them coming with a few hours’ notice, as the result of patterns in collected data), I’d support that whole-heartedly. As quoted, “There are a lot of things about the future that are essentially unknowable,” that’s very true. Using Big Data analysis of social and internet content, it would be very difficult to predict things that are not included in the data. Using meteorological data works to predict storms, but not political revolutions; if the next villan isn’t connected to the internet, Big Data doesn’t do us much good.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s