Investigative Data Warehouse (IDW) is a searchable database operated by the FBI. It was created in 2004. Much of the nature and scope of the database is classified. The database is a centralization of multiple federal and state databases, including criminal records from various law enforcement agencies, the U.S. Department of the Treasury's Financial Crimes Enforcement Network (FinCEN), and public records databases. According to Michael Morehart's testimony before the House Committee on Financial Services in 2006, the "IDW is a centralized, web-enabled, closed system repository for intelligence and investigative data. This system, maintained by the FBI, allows appropriately trained and authorized personnel throughout the country to query for information of relevance to investigative and intelligence matters." [1]
This section needs to be updated.(February 2017) |
The size of the database appears to be growing rapidly. In 2004, according to a government solicitation for bids to manage the project, it was approximately 10TB in size. In 2005, according to one FBI official, the IDW contained approximately 100 million documents. In 2006 it contained more than 560 million documents and was accessible by more than 12,000 individuals. According to the FBI's website, as of August 22, 2007, the database contained 700 million records from 53 databases and was accessible by 13,000 individuals around the world.
As of 2007 [update] , the FBI is the subject of a lawsuit brought by the EFF (Electronic Frontier Foundation) because of a lack of public notice describing the database and the criteria for including personal information, as required by the Privacy Act of 1974. The lawsuits are a result of two Freedom of Information Act requests filed by the EFF in 2006.
It was built in part by Chiliad corporation, [2] [3] the FBI Office of the Chief Technology Officer, [4] and others. Companies listed on the FOIA files include Northrop Grumman [5] and others.
Investigative Data Warehouse–Secret (IDW-S) "provides data and data processing/analysis services to FBI agents and analysts as they perform counter-terrorism, counter-intelligence, and law enforcement missions". The core subsystem supports the Counter-Terrorism Division (CTD), the Special Event Unit, and via DOCLAB-S, the Joint Intelligence Committee Investigation (JICI) and IntelPlus. [6]
According to a 2005 email, "IDW will also be used for criminal and other authorized non-CT investigations as it evolves." (CT being counter terrorism) [7]
Within the system, there were subsystems named IDW-S Core, SPT, and DOCLAB-S [8]
The special projects team (SPT):
allows for the rapid import of new specialized data sources. These data sources are not made available to the general IDW users but instead are provided to a small group of users who have a demonstrated "need-to-know". The SPT System is similar in function to the IDW-S system, with the main difference is a different set of data sources. The SPT System allows its users to access not only the standard IDW Data Store but the specialized SPT Data Store. [9]
According to internal emails, the FBI performed several Privacy Impact Assessments (PIAs) of the IDW system. They worked with lawyers from their National Security Law Branch (NSLB) to attempt to make sure their system was complying with various laws regarding sharing of information and secrecy [10] (for example, rule 6e of the Federal Rules of Criminal Procedure, regarding the secrecy of Grand Jury material [11] ).
The Information Sharing Policy Group (ISPG) formed a Discretionary Access Control Team (DACT), to work on "approval of data sets" and "access control requirements" for IDW and DataMart, and responding to other Intelligence Community agencies requesting access. [12]
The EFF FOIA IDW website states "Despite the vast amount of personal information contained in the IDW, the FBI has never published a Privacy Act notice describing the system or explaining the ways in which the records might be used." [13]
There was also a 2005 email from someone on the Office of General Council (OGC) about "preliminary staff musings that maybe we should limit FBI PIA requirements to non-NS systems" (NS being National Security). [14] There was also an email from 2006 saying that 'national security systems are exempt from E-Gov', [15] apparently referring to the E-Government Act of 2002, which has a section that deals with privacy.
The IDW used many data sources. The FOIA documents from EFF are heavily redacted, but some of the sources are as follows:
There was also talk of linking the FTTTF "Data Mart" with IDW. [26]
The data in IDW is classified at the 'Secret' level or lower. Higher classifications are not allowed, and can be removed [27]
Translingual Information Detection, Extraction and Summarization (TIDES) is a technology development program funded by the U.S. Defense Advanced Research Projects Agency (DARPA), focused on the automated processing and understanding of language data. The primary goal of the program is to enable English speakers to locate and interpret required information quickly and effectively regardless of the original language.
Computer and network surveillance is the monitoring of computer activity and data stored locally on a computer or data being transferred over computer networks such as the Internet. This monitoring is often carried out covertly and may be completed by governments, corporations, criminal organizations, or individuals. It may or may not be legal and may or may not require authorization from a court or other independent government agencies. Computer and network surveillance programs are widespread today and almost all Internet traffic can be monitored.
Open source intelligence (OSINT) is the collection and analysis of data gathered from open sources to produce actionable intelligence. OSINT is primarily used in national security, law enforcement, and business intelligence functions and is of value to analysts who use non-sensitive intelligence in answering classified, unclassified, or proprietary intelligence requirements across the previous intelligence disciplines.
The Freedom of Information Act, 5 U.S.C. § 552, is the United States federal freedom of information law that requires the full or partial disclosure of previously unreleased or uncirculated information and documents controlled by the U.S. government upon request. The act defines agency records subject to disclosure, outlines mandatory disclosure procedures, and includes nine exemptions that define categories of information not subject to disclosure. The act was intended to make U.S. government agencies' functions more transparent so that the American public could more easily identify problems in government functioning and put pressure on Congress, agency officials, and the president to address them. The FOIA has been changed repeatedly by both the legislative and executive branches.
Carnivore, later renamed DCS1000, was a system implemented by the Federal Bureau of Investigation (FBI) that was designed to monitor email and electronic communications. It used a customizable packet sniffer that could monitor all of a target user's Internet traffic. Carnivore was implemented in October 1997. By 2005 it had been replaced with improved commercial software.
Mark S. Zaid is an American attorney, based in Washington, D.C., with a practice focused on national security law, freedom of speech constitutional claims, and government accountability.
Electronic discovery refers to discovery in legal proceedings such as litigation, government investigations, or Freedom of Information Act requests, where the information sought is in electronic format. Electronic discovery is subject to rules of civil procedure and agreed-upon processes, often involving review for privilege and relevance before data are turned over to the requesting party.
The Automated Targeting System (ATS) is a United States Department of Homeland Security computerized system that, for every person who crosses U.S. borders, scrutinizes a large volume of data related to that person, and then automatically assigns a rating for which the expectation is that it helps gauge whether this person may be placed within a risk group of terrorists or other criminals. Similarly ATS analyzes data related to container cargo.
A government database collects information for various reasons, including climate monitoring, securities law compliance, geological surveys, patent applications and grants, surveillance, national security, border control, law enforcement, public health, voter registration, vehicle registration, social security, and statistics.
The Digital Collection System Network (DCSNet) is the Federal Bureau of Investigation (FBI)'s point-and-click surveillance system that can perform instant wiretaps on almost any telecommunications device in the United States.
The history of the USA PATRIOT Act involved many parties who opposed and supported the Patriot Act, which was proposed, enacted and signed into law 45 days after the September 11 terrorist attacks in 2001. The legislation, though approved by large majorities in the U.S. Senate and House of Representative, was controversial, and parts of the law were invalidated or modified by successful legal challenges over constitutional infringements to civil liberties. The Act had several sunset provisions, most reauthorized by the USA PATRIOT Improvement and Reauthorization Act of 2005 and the USA PATRIOT Act Additional Reauthorizing Amendments Act. Both reauthorizations incorporated amendments to the original USA PATRIOT Act, and other federal laws.
The Electronic Frontier Foundation (EFF) is an international non-profit digital rights group based in San Francisco, California. It was founded in 1990 to promote Internet civil liberties.
MiTAP, or Mitre Text and Audio Processing, is a computer system that tries to automatically gather, translate, organize, and present information "for monitoring infectious disease outbreaks and other global events." It is also used in the FBI Investigative Data Warehouse.
The FBI Indexes, or Index List, was a system used to track American citizens and other people by the Federal Bureau of Investigation (FBI) before the adoption of computerized databases. The Index List was originally made of paper index cards, first compiled by J. Edgar Hoover at the Bureau of Investigations before he was appointed director of the FBI. The Index List was used to track U.S. citizens and others believed by the FBI to be dangerous to national security, and was subdivided into various divisions which generally were rated based on different classes of danger the subject was thought to represent.
The practice of mass surveillance in the United States dates back to wartime monitoring and censorship of international communications from, to, or which passed through the United States. After the First and Second World Wars, mass surveillance continued throughout the Cold War period, via programs such as the Black Chamber and Project SHAMROCK. The formation and growth of federal law-enforcement and intelligence agencies such as the FBI, CIA, and NSA institutionalized surveillance used to also silence political dissent, as evidenced by COINTELPRO projects which targeted various organizations and individuals. During the Civil Rights Movement era, many individuals put under surveillance orders were first labelled as integrationists, then deemed subversive, and sometimes suspected to be supportive of the communist model of the United States' rival at the time, the Soviet Union. Other targeted individuals and groups included Native American activists, African American and Chicano liberation movement activists, and anti-war protesters.
MuckRock is a United States-based 501(c)(3) non-profit organization which assists anyone in filing governmental requests for information through the Freedom of Information Act (FOIA) and other public record laws around the United States, then publishes the returned information on its website and encourages journalism around it.
The Narcotics and Dangerous Drugs Information System, or NADDIS, is a data index and collection system operated by the United States Drug Enforcement Administration (DEA). Comprising millions of DEA reports and records on individuals, NADDIS is a system by which intelligence analysts, investigators and others in law enforcement retrieve reports from the DEA's Investigative Filing and Reporting System (IFRS). NADDIS is thought to have become the most widely used, if least known, tool in drug law enforcement.
The FISA Improvements Act is a proposed act by Senator Dianne Feinstein, Chair of the Senate Intelligence Committee. Prompted by the disclosure of NSA surveillance by Edward Snowden, it would establish the surveillance program as legal, but impose some limitations on availability of the data. Opponents say the bill would codify warrantless access to many communications of American citizens for use by domestic law enforcement.
Corporate surveillance describes the practice of businesses monitoring and extracting information from their users, clients, or staff. This information may consist of online browsing history, email correspondence, phone calls, location data, and other private details. Acts of corporate surveillance frequently look to boost results, detect potential security problems, or adjust advertising strategies. These practices have been criticized for violating ethical standards and invading personal privacy. Critics and privacy activists have called for businesses to incorporate rules and transparency surrounding their monitoring methods to ensure they are not misusing their position of authority or breaching regulatory standards.
Emma Best is an American investigative reporter and whistleblower. They gained national attention for their work with WikiLeaks and activist Julian Assange. Best is known for prolific filing of Freedom of Information Act (FOIA) requests on behalf of MuckRock and co-founding the whistleblower site Distributed Denial of Secrets (DDoSecrets).