The Systemic and Limiting Flaw for All Intelligence Agencies
All intelligence that exist, have existed or will ever exist have had to contend with a peculiar and systemic flaw.
The ratio of false positive to real leads.
This issue is well understood in the field of epidemiology and testing for infectious diseases. Let me use an example-
Let us say that a news HIV test has a false positive rate of 5 per 100. Now assume that it has an accuracy of 99.9% (almost every real case is picked up). Would you recommend this test for widespread screening? If your test population had an HIV+ve rate of 1%, 4 out of 5 positive tests would be incorrect. Given that AIDS patients may not have symptoms in the early stages, you are misidentifying 4 uninfected people for every infected person.
Once the infected persons are identified and isolated- your next batch of tests will, almost exclusively, tag uninfected people. Repeated rounds of testing will only cause more and more uninfected people to be identified as infected.
The efficacy of any intelligence agency, however grand its budget and technology, is ultimately dependent on that ratio- false positives should not overwhelm real positives. While poorer countries can tolerate inferior ratios, a dynamic productive economy cannot tolerate ratios significantly worse than 3 false leads for 1 good lead. More adverse ratios make the intelligence produced by such agencies useless and their actions counterproductive.
The amount of data produced also has a bearing on this ratio. In previous eras, it was possible to read all letters, tap every telephone conversation for a small percentage of the population. Now, the sheer volume and diversity of data along with various protocols, embedding techniques and encryption technologies combined with the need to have a human make the final decision makes that effectively impossible.
So why not dispense with human review? The answer is that we do not have algorithms good enough to replace human perception, analysis and reasoning for the final step.