 | This article relies largely or entirely on a single source .(November 2009) |
The Admiralty System or NATO System is a method for evaluating collected items of intelligence. [1] The system comprises a two-character notation assessing the reliability of the source and the assessed level of confidence on the information. The Admiralty system is used by NATO member nations and members of the Five Eyes community.
Evaluation occurs in the processing stage of the intelligence cycle recognising that collected information cannot be accepted at face value. Each item of information used in the creation of an assessment is given an indication of source reliability and assessed accuracy, based on corroboration or other assessment. Each descriptor is considered in isolation to ensure that the reliability of the source does not influence the assessed accuracy of the report.
A source is assessed for reliability based on a technical assessment of its capability, or in the case of Human Intelligence sources their history. Notation uses Alpha coding, A-F:
An item is assessed for credibility based on likelihood and levels of corroboration by other sources. Notation uses a numeric code, 1-6.
An assessment will provide a candidate course of action based on the coding of the items of collected information used to compile the assessment. The hypothesis will not be coded but the wording will caveat conclusions based on the levels of confidence indicated by the code.
Psychometrics is a field of study within psychology concerned with the theory and technique of measurement. Psychometrics generally covers specialized fields within psychology and education devoted to testing, measurement, assessment, and related activities. Psychometrics is concerned with the objective measurement of latent constructs that cannot be directly observed. Examples of latent constructs include intelligence, introversion, mental disorders, and educational achievement. The levels of individuals on nonobservable latent variables are inferred through mathematical modeling based on what is observed from individuals' responses to items on tests and scales.
In statistics and psychometrics, reliability is the overall consistency of a measure. A measure is said to have a high reliability if it produces similar results under consistent conditions:
"It is the characteristic of a set of test scores that relates to the amount of random error from the measurement process that might be embedded in the scores. Scores that are highly reliable are precise, reproducible, and consistent from one testing occasion to another. That is, if the testing process were repeated with a group of test takers, essentially the same results would be obtained. Various kinds of reliability coefficients, with values ranging between 0.00 and 1.00, are usually used to indicate the amount of error in the scores."
Validity is the main extent to which a concept, conclusion, or measurement is well-founded and likely corresponds accurately to the real world. The word "valid" is derived from the Latin validus, meaning strong. The validity of a measurement tool is the degree to which the tool measures what it claims to measure. Validity is based on the strength of a collection of different types of evidence described in greater detail below.
Health On the Net Foundation (HON) is a Swiss not-for-profit organization based in Geneva which promotes a code of conduct for websites providing health information and offers certificates to those in compliance.
Educational assessment or educational evaluation is the systematic process of documenting and using empirical data on the knowledge, skill, attitudes, aptitude and beliefs to refine programs and improve student learning. Assessment data can be obtained from directly examining student work to assess the achievement of learning outcomes or can be based on data from which one can make inferences about learning. Assessment is often used interchangeably with test, but not limited to tests. Assessment can focus on the individual learner, the learning community, a course, an academic program, the institution, or the educational system as a whole. The word "assessment" came into use in an educational context after the Second World War.
Thematic apperception test (TAT) is a projective psychological test developed during the 1930s by Henry A. Murray and Christiana D. Morgan at Harvard University. Proponents of the technique assert that subjects' responses, in the narratives they make up about ambiguous pictures of people, reveal their underlying motives, concerns, and the way they see the social world. Historically, the test has been among the most widely researched, taught, and used of such techniques.
Historical method is the collection of techniques and guidelines that historians use to research and write histories of the past. Secondary sources, primary sources and material evidence such as that derived from archaeology may all be drawn on, and the historian's skill lies in identifying these sources, evaluating their relative authority, and combining their testimony appropriately in order to construct an accurate and reliable picture of past events and environments.
Source criticism is the process of evaluating an information source, i.e.: a document, a person, a speech, a fingerprint, a photo, an observation, or anything used in order to obtain knowledge. In relation to a given purpose, a given information source may be more or less valid, reliable or relevant. Broadly, "source criticism" is the interdisciplinary study of how information sources are evaluated for given tasks.
The Shedler-Westen Assessment Procedure (SWAP-200) is a psychological test for personality diagnosis and clinical case formulation, developed by psychologists Jonathan Shedler and Drew Westen. SWAP-200 is completed by a mental health professional based on their observations and knowledge of a patient, client, or assessment subject. The person being assessed does not interact with the test. Because SWAP-200 is completed by the clinician, diagnostic findings do not depend on the accuracy of information people disclose about themselves and test results can not be faked. The SWAP instruments are based on over two decades of empirical research described in more than 100 articles in peer-reviewed scientific journals. SWAP-200 has been translated into fifteen languages. Other SWAP instruments include the revised SWAP-II and the SWAP-II-A for adolescents.
The Joint Committee on Standards for Educational Evaluation is an American/Canadian based Standards Developer Organization (SDO). The Joint Committee, created in 1975, represents a coalition of major professional associations formed in 1975 to develop evaluation standards and improve the quality of standardized evaluation. The Committee has thus far published three sets of standards for evaluations. The Personnel Evaluation Standards was published in 1988 and updated in 2008, The Program Evaluation Standards was published in 1994, and The Student Evaluation Standards was published in 2003.
In statistics, inter-rater reliability is the degree of agreement among independent observers who rate, code, or assess the same phenomenon.
Intelligence collection management is the process of managing and organizing the collection of intelligence from various sources. The collection department of an intelligence organization may attempt basic validation of what it collects, but is not supposed to analyze its significance. There is debate in U.S. intelligence community on the difference between validation and analysis, where the National Security Agency may try to interpret information when such interpretation is the job of another agency.
Psychological evaluation is a method to assess an individual's behavior, personality, cognitive abilities, and several other domains. A common reason for a psychological evaluation is to identify psychological factors that may be inhibiting a person's ability to think, behave, or regulate emotion functionally or constructively. It is the mental equivalent of physical examination. Other psychological evaluations seek to better understand the individual's unique characteristics or personality to predict things like workplace performance or customer relationship management.
In qualitative research, a member check, also known as informant feedback or respondent validation, is a technique used by researchers to help improve the accuracy, credibility, validity, and transferability of a study. There are many subcategories of members checks, including: narrative accuracy checks, interpretive validity, descriptive validity, theoretical validity, and evaluative validity. In many member checks, the interpretation and report is given to members of the sample (informants) in order to check the authenticity of the work. Their comments serve as a check on the viability of the interpretation.
Analytic confidence is a rating employed by intelligence analysts to convey doubt to decision makers about a statement of estimative probability. The need for analytic confidence ratings arise from analysts' imperfect knowledge of a conceptual model. An analytic confidence rating pairs with a statement using a word of estimative probability to form a complete analytic statement. Scientific methods for determining analytic confidence remain in infancy.
Self-report sexual risk behaviors are a cornerstone of reproductive health–related research, particularly when related to assessing risk-related outcomes such as pregnancy or acquisition of sexually transmitted diseases (STDs) such as HIV. Despite their frequency of use, the utility of self-report measures to provide an accurate account of actual behavior are questioned, and methods of enhancing their accuracy should be a critical focus when administering such measures. Self-reported assessments of sexual behavior are prone to a number of measurement concerns which may affect the reliability and validity of a measure, ranging from a participant's literacy level and comprehension of behavioral terminology to recall biases and self-presentation.
WikiTrust was a software product, available as a Firefox Plugin, which aimed to assist editors in detecting vandalism and dubious edits by highlighting the "untrustworthy" text with a yellow or orange background. As of September 2017, the server is offline, but the code is still available for download.
Intelligence source and information reliability rating systems are used in intelligence analysis. This rating is used for information collected by a human intelligence collector. This type of information collection and job duty exists within many government agencies around the world.
The Child Mania Rating Scales (CMRS) is a 21-item diagnostic screening measure designed to identify symptoms of mania in children and adolescents aged 9–17 using diagnostic criteria from the DSM-IV, developed by Pavuluri and colleagues. There is also a 10-item short form. The measure assesses the child's mood and behavior symptoms, asking parents or teachers to rate how often the symptoms have caused a problem for the youth in the past month. Clinical studies have found the CMRS to be reliable and valid when completed by parents in the assessment of children's bipolar symptoms. The CMRS also can differentiate cases of pediatric bipolar disorder from those with ADHD or no disorder, as well as delineating bipolar subtypes. A meta-analysis comparing the different rating scales available found that the CMRS was one of the best performing scales in terms of telling cases with bipolar disorder apart from other clinical diagnoses. The CMRS has also been found to provide a reliable and valid assessment of symptoms longitudinally over the course of treatment. The combination of showing good reliability and validity across multiple samples and clinical settings, along with being free and brief to score, make the CMRS a promising tool, especially since most other checklists available for youths do not assess manic symptoms.
Competence assessment is a process in which evidence is gathered by the assessor and evaluated against agreed criteria in order to make a judgement of competence. Skill assessment is the comparison of actual performance of a skill with the specified standard for performance of that skill under the circumstances specified by the standard, and evaluation of whether the performance meets or exceed the requirements. Assessment of a skill should comply with the four principles of validity, reliability, fairness and flexibility.