WikiLeaks embassy cables: download the key data and see how it breaks down
http://www.guardian.co.uk/news/datablog/2010/nov/29/wikileaks-cables-dataThe WikiLeaks embassy cables release has produced a lot of stories but does it produce any useful data? We explain what it includes and how it breaks down - plus you can download the key data for every cable
• Get the data
• Wikileaks cables: the interactive guide to what we've published
• Get the data
• Wikileaks cables: the interactive guide to what we've published
• 21:35pm, 28.11.2010: We've just added a CSV download - see below. Remember this is the date, time, sender and tags for each cable - NOT the text of the cable itself
WikiLeaks embassy cables revelations cover a huge dataset of official documents: 251,287 dispatches, from more than 250 worldwide US embassies and consulates. It's a unique picture of US diplomatic language - including over 50,000 documents covering the current Obama administration. But what does the data include?
The cables themselves come via the huge Secret Internet Protocol Router Network, or SIPRNet. SIPRNet is the worldwide US military internet system, kept separate from the ordinary civilian internet and run by the Department of Defense in Washington. Since the attacks of September 2001, there has been a move in the US to link up archives of government information, in the hope that key intelligence no longer gets trapped in information silos or "stovepipes". An increasing number of US embassies have become linked to SIPRNet over the past decade, so that military and diplomatic information can be shared. By 2002, 125 embassies were on SIPRNet: by 2005, the number had risen to 180, and by now the vast majority of US missions worldwide are linked to the system - which is why the bulk of these cables are from 2008 and 2009.
An embassy dispatch marked SIPDIS is automatically downloaded on to its embassy classified website. From there, it can be accessed not only by anyone in the state department, but also by anyone in the US military who has a security clearance up to the 'Secret' level, a password, and a computer connected to SIPRNet - which astonishingly covers over 3m people. There are several layers of data in here - ranging up to the "SECRET NOFORN" level, which means that they are designed never be shown to non-US citizens. Instead, they are supposed to be read by officials in Washington up to the level of current Secretary of State Hillary Clinton. The cables are normally drafted by the local ambassador or subordinates. The "Top Secret" and above foreign intelligence documents cannot be accessed from SIPRNet.
Google fusion table storyline of cables sent in the weeks around 9/11 - click on a cable to find out the tags. Get the fullscreen version here or click here to download the data
• The state department sent the most cables in this set, followed by Ankara in Turkey, then Baghdad and Tokyo
• 97,070 of the documents were classified as 'Confidential'
• 28,760 of them were given the tag 'PTER' which stands for prevention of terrorism
• The earliest of the cables is from 1966 - with most, 56,813, from 2009
What can you do with the data?
An embassy dispatch marked SIPDIS is automatically downloaded on to its embassy classified website. From there, it can be accessed not only by anyone in the state department, but also by anyone in the US military who has a security clearance up to the 'Secret' level, a password, and a computer connected to SIPRNet - which astonishingly covers over 3m people. There are several layers of data in here - ranging up to the "SECRET NOFORN" level, which means that they are designed never be shown to non-US citizens. Instead, they are supposed to be read by officials in Washington up to the level of current Secretary of State Hillary Clinton. The cables are normally drafted by the local ambassador or subordinates. The "Top Secret" and above foreign intelligence documents cannot be accessed from SIPRNet.
Google fusion table storyline of cables sent in the weeks around 9/11 - click on a cable to find out the tags. Get the fullscreen version here or click here to download the data
We've broken down the data for you - and you can download the basic details of every cable (without the actual content) below. Each cable is essentially very structured data. This is what's included:
• A source, ie the embassy or body which sent it
• There is a list of recipients - normally cables were sent to a number of other embassies and bodies
• There is a subject field - basically a summary of the cable
• Tags - each cable was tagged with a number of keyword abbreviations. We've put together a downloadable Google glossary spreadsheet of most of the important ones here
• Body text - the cable itself. We have opted not to publish these in full for obvious security reasons
Thanks to Guardian developer Daithi Ó Crualaoich we've performed some analysis of the data - which you can download for yourself below. The key points are:
• 251,287 dispatches• The state department sent the most cables in this set, followed by Ankara in Turkey, then Baghdad and Tokyo
• 97,070 of the documents were classified as 'Confidential'
• 28,760 of them were given the tag 'PTER' which stands for prevention of terrorism
• The earliest of the cables is from 1966 - with most, 56,813, from 2009
What can you do with the data?
Download the data
• DATA: every cable with date, time and tags, EXCLUDING BODY TEXT (via Google fusion tables, subject to heavy traffic)
• DATA: every cable with date, time and tags, EXCLUDING BODY TEXT (Zipped CSV file, 3.1MB)
• DATA: our analysis of the cable by location and tag
• DATA: glossary of keywords and tags
• DATA: every cable with date, time and tags, EXCLUDING BODY TEXT (Zipped CSV file, 3.1MB)
• DATA: our analysis of the cable by location and tag
• DATA: glossary of keywords and tags
World government data
Development and aid data
Can you do something with this data?
• Flickr Please post your visualisations and mash-ups on our Flickr group
• Contact us at data@guardian.co.uk
• Contact us at data@guardian.co.uk
