Active learning (machine learning)

Last updated February 23, 2024

Active learning is a special case of machine learning in which a learning algorithm can interactively query a human user (or some other information source), to label new data points with the desired outputs. The human user must possess knowledge/expertise in the problem domain, including the ability to consult/research authoritative sources when necessary. ^[1]^[2]^[3] In statistics literature, it is sometimes also called optimal experimental design.^[4] The information source is also called teacher or oracle.

There are situations in which unlabeled data is abundant but manual labeling is expensive. In such a scenario, learning algorithms can actively query the user/teacher for labels. This type of iterative supervised learning is called active learning. Since the learner chooses the examples, the number of examples to learn a concept can often be much lower than the number required in normal supervised learning. With this approach, there is a risk that the algorithm is overwhelmed by uninformative examples. Recent developments are dedicated to multi-label active learning,^[5] hybrid active learning^[6] and active learning in a single-pass (on-line) context,^[7] combining concepts from the field of machine learning (e.g. conflict and ignorance) with adaptive, incremental learning policies in the field of online machine learning. Using active learning allows for faster development of a machine learning algorithm, when comparative updates would require a quantum or super computer.^[8]

Large-scale active learning projects may benefit from crowdsourcing frameworks such as Amazon Mechanical Turk that include many humans in the active learning loop.

Definitions

Let $T$ be the total set of all data under consideration. For example, in a protein engineering problem, $T$ would include all proteins that are known to have a certain interesting activity and all additional proteins that one might want to test for that activity.

During each iteration, $i$ , $T$ is broken up into three subsets

$\mathbf {T} _{K,i}$ : Data points where the label is known.
$\mathbf {T} _{U,i}$ : Data points where the label is unknown.
$\mathbf {T} _{C,i}$ : A subset of $T U,i$ that is chosen to be labeled.

Most of the current research in active learning involves the best method to choose the data points for $T C,i$ .

Scenarios

Pool-Based Sampling: In this approach, which is the most well known scenario,^[9] the learning algorithm attempts to evaluate the entire dataset before selecting data points (instances) for labeling. It is often initially trained on a fully labeled subset of the data using a machine-learning method such as logistic regression or SVM that yields class-membership probabilities for individual data instances. The candidate instances are those for which the prediction is most ambiguous.instances are drawn from the entire data pool and assigned a confidence score, a measurement of how well the learner "understands" the data. The system then selects the instances for which it is the least confident and queries the teacher for the labels.
The theoretical drawback of pool-based samplilng is that it is memory-intensive and is therefore limited in its capacity to handle enormous datasets, but in practice, the rate-limiting factor is that the teacher is typically a (fatiguable) human expert who must be paid for their effort, rather than computer memory.
Stream-Based Selective Sampling: Here, each consective unlabeled instance is examined one at a time with the machine evaluating the informativeness of each item against its query parameters. The learner decides for itself whether to assign a label or query the teacher for each datapoint. As contrasted with Pool-based sampling, the obvious drawback of stream-based methods is that the learning algorithm does not have sufficient information, early in the process, to make a sound assign-label-vs ask-teacher decision, and it does not capitalize as efficiently on the presence of already labeled data. Therefore, the teacher is likely to spend more effort in supplying labels than with the pool-based approach.
Membership Query Synthesis: This is where the learner generates synthetic data from an underlying natural distribution. For example, if the dataset are pictures of humans and animals, the learner could send a clipped image of a leg to the teacher and query if this appendage belongs to an animal or human. This is particularly useful if the dataset is small.^[10]
The challenge here, as with all synthetic-data-generation efforts, is in ensuring that the synthetic data is consistent in terms of meeting the constraints on real data. As the number of variables/features in the input data increase, and strong dependencies between variables exist, it becomes increasingly difficult to generate synthetic data with sufficient fidelity.
For example, to create a synthetic data set for human laboratory-test values, the sum of the various white blood cell (WBC) components in a White Blood Cell differential must equal 100, since the component numbers are really percentages. Similarly, the enzymes Alanine Transaminase (ALT) and Aspartate Transaminase (AST) measure liver function (though AST is also produced by other tissues, e.g., lung, pancreas) A synthetic data point with AST at the lower limit of normal range (8-33 Units/L) with an ALT several times above normal range (4-35 Units/L) in a simulated chronically ill patient would be physiologically impossible.

Query strategies

Algorithms for determining which data points should be labeled can be organized into a number of different categories, based upon their purpose:^[1]

Balance exploration and exploitation: the choice of examples to label is seen as a dilemma between the exploration and the exploitation over the data space representation. This strategy manages this compromise by modelling the active learning problem as a contextual bandit problem. For example, Bouneffouf et al.^[11] propose a sequential algorithm named Active Thompson Sampling (ATS), which, in each round, assigns a sampling distribution on the pool, samples one point from this distribution, and queries the oracle for this sample point label.
Expected model change: label those points that would most change the current model.
Expected error reduction: label those points that would most reduce the model's generalization error.
Exponentiated Gradient Exploration for Active Learning:^[12] In this paper, the author proposes a sequential algorithm named exponentiated gradient (EG)-active that can improve any active learning algorithm by an optimal random exploration.
Random Sampling: a sample is randomly selected.^[13]
Uncertainty sampling: label those points for which the current model is least certain as to what the correct output should be.
- Entropy Sampling: The entropy formula is used on each sample, and the sample with the highest entropy is considered to be the least certain.^[13]
- Margin Sampling: The sample with the smallest difference between the two highest class probabilities is considered to be the most uncertain.^[13]
- Least Confident Sampling: The sample with the smallest best probability is considered to be the most uncertain.^[13]
Query by committee: a variety of models are trained on the current labeled data, and vote on the output for unlabeled data; label those points for which the "committee" disagrees the most
Querying from diverse subspaces or partitions:^[14] When the underlying model is a forest of trees, the leaf nodes might represent (overlapping) partitions of the original feature space. This offers the possibility of selecting instances from non-overlapping or minimally overlapping partitions for labeling.
Variance reduction: label those points that would minimize output variance, which is one of the components of error.
Conformal prediction : predicts that a new data point will have a label similar to old data points in some specified way and degree of the similarity within the old examples is used to estimate the confidence in the prediction.^[15]
Mismatch-first farthest-traversal: The primary selection criterion is the prediction mismatch between the current model and nearest-neighbour prediction. It targets on wrongly predicted data points. The second selection criterion is the distance to previously selected data, the farthest first. It aims at optimizing the diversity of selected data.^[16]
User Centered Labeling Strategies: Learning is accomplished by applying dimensionality reduction to graphs and figures like scatter plots. Then the user is asked to label the compiled data (categorical, numerical, relevance scores, relation between two instances.^[17]

A wide variety of algorithms have been studied that fall into these categories.^[1]^[4] While the traditional AL strategies can achieve remarkable performance, it is often challenging to predict in advance which strategy is the most suitable in aparticular situation. In recent years, meta-learning algorithms have been gaining in popularity. Some of them have been proposed to tackle the problem of learning AL strategies instead of relying on manually designed strategies. A benchmark which compares 'meta-learning approaches to active learning' to 'traditional heuristic-based Active Learning' may give intuitions if 'Learning active learning' is at the crossroads ^[18]

Minimum marginal hyperplane

Some active learning algorithms are built upon support-vector machines (SVMs) and exploit the structure of the SVM to determine which data points to label. Such methods usually calculate the margin, $W$ , of each unlabeled datum in $T U,i$ and treat $W$ as an $n$ -dimensional distance from that datum to the separating hyperplane.

Minimum Marginal Hyperplane methods assume that the data with the smallest $W$ are those that the SVM is most uncertain about and therefore should be placed in $T C,i$ to be labeled. Other similar methods, such as Maximum Marginal Hyperplane, choose data with the largest $W$ . Tradeoff methods choose a mix of the smallest and largest $W$ s.

Literature

Balcan, Maria-Florina & Hanneke, Steve & Wortman, Jennifer. (2008). The True Sample Complexity of Active Learning.. 45-56. https://link.springer.com/article/10.1007/s10994-010-5174-y
Active Learning and Bayesian Optimization: a Unified Perspective to Learn with a Goal, Francesco Di Fiore, Michela Nardelli, Laura Mainini, https://arxiv.org/abs/2303.01560v2
Learning how to Active Learn: A Deep Reinforcement Learning Approach, Meng Fang, Yuan Li, Trevor Cohn, https://arxiv.org/abs/1708.02383v1

Related Research Articles

<span class="mw-page-title-main">Supervised learning</span> A paradigm in machine learning

Supervised learning (SL) is a paradigm in machine learning where input objects and a desired output value train a model. The training data is processed, building a function that maps new data on expected output values. An optimal scenario will allow for the algorithm to correctly determine output values for unseen instances. This requires the learning algorithm to generalize from the training data to unseen situations in a "reasonable" way. This statistical quality of an algorithm is measured through the so-called generalization error.

In machine learning, support vector machines are supervised max-margin models with associated learning algorithms that analyze data for classification and regression analysis. Developed at AT&T Bell Laboratories by Vladimir Vapnik with colleagues SVMs are one of the most studied models, being based on statistical learning frameworks or VC theory proposed by Vapnik and Chervonenkis (1974).

Machine learning (ML) is a field of study in artificial intelligence concerned with the development and study of statistical algorithms that can learn from data and generalize to unseen data, and thus perform tasks without explicit instructions. Recently, generative artificial neural networks have been able to surpass many previous approaches in performance.

Learning classifier systems, or LCS, are a paradigm of rule-based machine learning methods that combine a discovery component with a learning component. Learning classifier systems seek to identify a set of context-dependent rules that collectively store and apply knowledge in a piecewise manner in order to make predictions. This approach allows complex solution spaces to be broken up into smaller, simpler parts.

In logic, statistical inference, and supervised learning, transduction or transductive inference is reasoning from observed, specific (training) cases to specific (test) cases. In contrast, induction is reasoning from observed training cases to general rules, which are then applied to the test cases. The distinction is most interesting in cases where the predictions of the transductive model are not achievable by any inductive model. Note that this is caused by transductive inference on different test sets producing mutually inconsistent predictions.

Bootstrap aggregating, also called bagging, is a machine learning ensemble meta-algorithm designed to improve the stability and accuracy of machine learning algorithms used in statistical classification and regression. It also reduces variance and helps to avoid overfitting. Although it is usually applied to decision tree methods, it can be used with any type of method. Bagging is a special case of the model averaging approach.

In machine learning, kernel machines are a class of algorithms for pattern analysis, whose best known member is the support-vector machine (SVM). These methods involve using linear classifiers to solve nonlinear problems. The general task of pattern analysis is to find and study general types of relations in datasets. For many algorithms that solve these tasks, the data in raw representation have to be explicitly transformed into feature vector representations via a user-specified feature map: in contrast, kernel methods require only a user-specified kernel, i.e., a similarity function over all pairs of data points computed using inner products. The feature map in kernel machines is infinite dimensional but only requires a finite dimensional matrix from user-input according to the Representer theorem. Kernel machines are slow to compute for datasets larger than a couple of thousand examples without parallel processing.

In machine learning, multi-label classification or multi-output classification is a variant of the classification problem where multiple nonexclusive labels may be assigned to each instance. Multi-label classification is a generalization of multiclass classification, which is the single-label problem of categorizing instances into precisely one of several classes. In the multi-label problem the labels are nonexclusive and there is no constraint on how many of the classes the instance can be assigned to.

In machine learning, one-class classification (OCC), also known as unary classification or class-modelling, tries to identify objects of a specific class amongst all objects, by primarily learning from a training set containing only the objects of that class, although there exist variants of one-class classifiers where counter-examples are used to further refine the classification boundary. This is different from and more difficult than the traditional classification problem, which tries to distinguish between two or more classes with the training set containing objects from all the classes. Examples include the monitoring of helicopter gearboxes, motor failure prediction, or the operational status of a nuclear plant as 'normal': In this scenario, there are few, if any, examples of catastrophic system states; only the statistics of normal operation are known.

Within statistics, oversampling and undersampling in data analysis are techniques used to adjust the class distribution of a data set. These terms are used both in statistical sampling, survey design methodology and in machine learning.

In machine learning and statistical classification, multiclass classification or multinomial classification is the problem of classifying instances into one of three or more classes.

In machine learning, feature learning or representation learning is a set of techniques that allows a system to automatically discover the representations needed for feature detection or classification from raw data. This replaces manual feature engineering and allows a machine to both learn the features and use them to perform a specific task.

Quantum machine learning is the integration of quantum algorithms within machine learning programs.

Adversarial machine learning is the study of the attacks on machine learning algorithms, and of the defenses against such attacks. A survey from May 2020 exposes the fact that practitioners report a dire need for better protecting machine learning systems in industrial applications.

Domain adaptation is a field associated with machine learning and transfer learning. This scenario arises when we aim at learning a model from a source data distribution and applying that model on a different target data distribution. For instance, one of the tasks of the common spam filtering problem consists in adapting a model from one user to a new user who receives significantly different emails. Domain adaptation has also been shown to be beneficial for learning unrelated sources. Note that, when more than one source distribution is available the problem is referred to as multi-source domain adaptation.

In machine learning, Manifold regularization is a technique for using the shape of a dataset to constrain the functions that should be learned on that dataset. In many machine learning problems, the data to be learned do not cover the entire input space. For example, a facial recognition system may not need to classify any possible image, but only the subset of images that contain faces. The technique of manifold learning assumes that the relevant subset of data comes from a manifold, a mathematical structure with useful properties. The technique also assumes that the function to be learned is smooth: data with different labels are not likely to be close together, and so the labeling function should not change quickly in areas where there are likely to be many data points. Because of this assumption, a manifold regularization algorithm can use unlabeled data to inform where the learned function is allowed to change quickly and where it is not, using an extension of the technique of Tikhonov regularization. Manifold regularization algorithms can extend supervised learning algorithms in semi-supervised learning and transductive learning settings, where unlabeled data are available. The technique has been used for applications including medical imaging, geographical imaging, and object recognition.

Data augmentation is a technique in machine learning used to reduce overfitting when training a machine learning model, achieved by training models on several slightly-modified copies of existing data.

The following outline is provided as an overview of and topical guide to machine learning:

Weak supervision is a paradigm in machine learning, the relevance and notability of which increased with the advent of large language models due to large amount of data required to train them. It is characterized by using a combination of a small amount of human-labeled data, followed by a large amount of unlabeled data. In other words, the desired output values are provided only for a subset of the training data. The remaining data is unlabeled or imprecisely labeled. Intuitively, it can be seen as an exam and labeled data as sample problems that the teacher solves for the class as an aid in solving another set of problems. In the transductive setting, these unsolved problems act as exam questions. In the inductive setting, they become practice problems of the sort that will make up the exam. Technically, it could be viewed as performing clustering and then labeling the clusters with the labeled data, pushing the decision boundary away from high-density regions, or learning an underlying one-dimensional manifold where the data reside.

Federated learning is a sub-field of machine learning focusing on settings in which multiple entities collaboratively train a model while ensuring that their data remains decentralized. This stands in contrast to machine learning settings in which data is centrally stored. One of the primary defining characteristics of federated learning is data heterogeneity. Due to the decentralized nature of the clients' data, there is no guarantee that data samples held by each client are independently and identically distributed.

References

1 2 3 Settles, Burr (2010). "Active Learning Literature Survey" (PDF). Computer Sciences Technical Report 1648. University of Wisconsin–Madison. Retrieved 2014-11-18.
↑ Rubens, Neil; Elahi, Mehdi; Sugiyama, Masashi; Kaplan, Dain (2016). "Active Learning in Recommender Systems". In Ricci, Francesco; Rokach, Lior; Shapira, Bracha (eds.). Recommender Systems Handbook (PDF) (2 ed.). Springer US. doi:10.1007/978-1-4899-7637-6. hdl:11311/1006123. ISBN 978-1-4899-7637-6. S2CID 11569603.
↑ Das, Shubhomoy; Wong, Weng-Keen; Dietterich, Thomas; Fern, Alan; Emmott, Andrew (2016). "Incorporating Expert Feedback into Active Anomaly Discovery". In Bonchi, Francesco; Domingo-Ferrer, Josep; Baeza-Yates, Ricardo; Zhou, Zhi-Hua; Wu, Xindong (eds.). IEEE 16th International Conference on Data Mining. IEEE. pp. 853–858. doi:10.1109/ICDM.2016.0102. ISBN 978-1-5090-5473-2. S2CID 15285595.
1 2 Olsson, Fredrik (April 2009). "A literature survey of active machine learning in the context of natural language processing". SICS Technical Report T2009:06.
↑ Yang, Bishan; Sun, Jian-Tao; Wang, Tengjiao; Chen, Zheng (2009). "Effective multi-label active learning for text classification" (PDF). Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '09. p. 917. CiteSeerX 10.1.1.546.9358 . doi:10.1145/1557019.1557119. ISBN 978-1-60558-495-9. S2CID 1979173.
↑ Lughofer, Edwin (February 2012). "Hybrid active learning for reducing the annotation effort of operators in classification systems". Pattern Recognition. 45 (2): 884–896. Bibcode:2012PatRe..45..884L. doi:10.1016/j.patcog.2011.08.009.
↑ Lughofer, Edwin (2012). "Single-pass active learning with conflict and ignorance". Evolving Systems. 3 (4): 251–271. doi:10.1007/s12530-012-9060-7. S2CID 43844282.
↑ Novikov, Ivan (2021). "The MLIP package: moment tensor potentials with MPI and active learning". IOP Publishing. 2 (2): 3, 4. arXiv: 2007.08555 . doi: 10.1088/2632-2153/abc9fe – via IOP science.
↑ DataRobot. "Active learning machine learning: What it is and how it works". DataRobot Blog. DataRobot Inc. Retrieved 30 January 2024.
↑ Wang, Liantao; Hu, Xuelei; Yuan, Bo; Lu, Jianfeng (2015-01-05). "Active learning via query synthesis and nearest neighbour search" (PDF). Neurocomputing. 147: 426–434. doi:10.1016/j.neucom.2014.06.042. S2CID 3027214.
↑ Bouneffouf, Djallel; Laroche, Romain; Urvoy, Tanguy; Féraud, Raphael; Allesiardo, Robin (2014). "Contextual Bandit for Active Learning: Active Thompson". In Loo, C. K.; Yap, K. S.; Wong, K. W.; Teoh, A.; Huang, K. (eds.). Neural Information Processing (PDF). Lecture Notes in Computer Science. Vol. 8834. pp. 405–412. doi:10.1007/978-3-319-12637-1_51. ISBN 978-3-319-12636-4. S2CID 1701357. HAL Id: hal-01069802.
↑ Bouneffouf, Djallel (8 January 2016). "Exponentiated Gradient Exploration for Active Learning". Computers. 5 (1): 1. arXiv: 1408.2196 . doi: 10.3390/computers5010001 . S2CID 14313852.
1 2 3 4 Faria, Bruno; Perdigão, Dylan; Brás, Joana; Macedo, Luis (2022). The Joint Role of Batch Size and Query Strategy in Active Learning-Based Prediction - A Case Study in the Heart Attack Domain. Lecture Notes in Computer Science. Vol. 13566. pp. 464–475. doi:10.1007/978-3-031-16474-3_38. ISBN 978-3-031-16473-6.{{cite book}}: |journal= ignored (help)
↑ "shubhomoydas/ad_examples". GitHub. Retrieved 2018-12-04.
↑ Makili, Lázaro Emílio; Sánchez, Jesús A. Vega; Dormido-Canto, Sebastián (2012-10-01). "Active Learning Using Conformal Predictors: Application to Image Classification". Fusion Science and Technology. 62 (2): 347–355. doi:10.13182/FST12-A14626. ISSN 1536-1055. S2CID 115384000.
↑ Zhao, Shuyang; Heittola, Toni; Virtanen, Tuomas (2020). "Active learning for sound event detection". IEEE/ACM Transactions on Audio, Speech, and Language Processing. arXiv: 2002.05033 .
↑ Bernard, Jürgen; Zeppelzauer, Matthias; Lehmann, Markus; Müller, Martin; Sedlmair, Michael (June 2018). "Towards User-Centered Active Learning Algorithms". Computer Graphics Forum. 37 (3): 121–132. doi:10.1111/cgf.13406. ISSN 0167-7055. S2CID 51875861.
↑ Desreumaux, Louis; Lemaire, Vincent (2020). "Learning Active Learning at the Crossroads? Evaluation and Discussion" (Proceedings of the Workshop on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases {(ECML} {PKDD} 2020), Ghent, Belgium, 2020). S2CID 221794570.{{cite journal}}: Cite journal requires |journal= (help)

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[settles-1] 1 2 3 Settles, Burr (2010). "Active Learning Literature Survey" (PDF). Computer Sciences Technical Report 1648. University of Wisconsin–Madison. Retrieved 2014-11-18.

[rubens2016-2] Rubens, Neil; Elahi, Mehdi; Sugiyama, Masashi; Kaplan, Dain (2016). "Active Learning in Recommender Systems". In Ricci, Francesco; Rokach, Lior; Shapira, Bracha (eds.). Recommender Systems Handbook (PDF) (2 ed.). Springer US. doi:10.1007/978-1-4899-7637-6. hdl:11311/1006123. ISBN 978-1-4899-7637-6. S2CID 11569603.

[das2016-3] Das, Shubhomoy; Wong, Weng-Keen; Dietterich, Thomas; Fern, Alan; Emmott, Andrew (2016). "Incorporating Expert Feedback into Active Anomaly Discovery". In Bonchi, Francesco; Domingo-Ferrer, Josep; Baeza-Yates, Ricardo; Zhou, Zhi-Hua; Wu, Xindong (eds.). IEEE 16th International Conference on Data Mining. IEEE. pp. 853–858. doi:10.1109/ICDM.2016.0102. ISBN 978-1-5090-5473-2. S2CID 15285595.

[olsson-4] 1 2 Olsson, Fredrik (April 2009). "A literature survey of active machine learning in the context of natural language processing". SICS Technical Report T2009:06.

[multi-5] Yang, Bishan; Sun, Jian-Tao; Wang, Tengjiao; Chen, Zheng (2009). "Effective multi-label active learning for text classification" (PDF). Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '09. p. 917. CiteSeerX 10.1.1.546.9358 . doi:10.1145/1557019.1557119. ISBN 978-1-60558-495-9. S2CID 1979173.

[hybrid-6] Lughofer, Edwin (February 2012). "Hybrid active learning for reducing the annotation effort of operators in classification systems". Pattern Recognition. 45 (2): 884–896. Bibcode:2012PatRe..45..884L. doi:10.1016/j.patcog.2011.08.009.

[single-pass-7] Lughofer, Edwin (2012). "Single-pass active learning with conflict and ignorance". Evolving Systems. 3 (4): 251–271. doi:10.1007/s12530-012-9060-7. S2CID 43844282.

[8] Novikov, Ivan (2021). "The MLIP package: moment tensor potentials with MPI and active learning". IOP Publishing. 2 (2): 3, 4. arXiv: 2007.08555 . doi: 10.1088/2632-2153/abc9fe – via IOP science.

[9] DataRobot. "Active learning machine learning: What it is and how it works". DataRobot Blog. DataRobot Inc. Retrieved 30 January 2024.

[10] Wang, Liantao; Hu, Xuelei; Yuan, Bo; Lu, Jianfeng (2015-01-05). "Active learning via query synthesis and nearest neighbour search" (PDF). Neurocomputing. 147: 426–434. doi:10.1016/j.neucom.2014.06.042. S2CID 3027214.

[Bouneffouf(2014)-11] Bouneffouf, Djallel; Laroche, Romain; Urvoy, Tanguy; Féraud, Raphael; Allesiardo, Robin (2014). "Contextual Bandit for Active Learning: Active Thompson". In Loo, C. K.; Yap, K. S.; Wong, K. W.; Teoh, A.; Huang, K. (eds.). Neural Information Processing (PDF). Lecture Notes in Computer Science. Vol. 8834. pp. 405–412. doi:10.1007/978-3-319-12637-1_51. ISBN 978-3-319-12636-4. S2CID 1701357. HAL Id: hal-01069802.

[Bouneffouf(2016)-12] Bouneffouf, Djallel (8 January 2016). "Exponentiated Gradient Exploration for Active Learning". Computers. 5 (1): 1. arXiv: 1408.2196 . doi: 10.3390/computers5010001 . S2CID 14313852.

[joint_role-13] 1 2 3 4 Faria, Bruno; Perdigão, Dylan; Brás, Joana; Macedo, Luis (2022). The Joint Role of Batch Size and Query Strategy in Active Learning-Based Prediction - A Case Study in the Heart Attack Domain. Lecture Notes in Computer Science. Vol. 13566. pp. 464–475. doi:10.1007/978-3-031-16474-3_38. ISBN 978-3-031-16473-6.{{cite book}}: |journal= ignored (help)

[shubhomoydas_github-14] "shubhomoydas/ad_examples". GitHub. Retrieved 2018-12-04.

[15] Makili, Lázaro Emílio; Sánchez, Jesús A. Vega; Dormido-Canto, Sebastián (2012-10-01). "Active Learning Using Conformal Predictors: Application to Image Classification". Fusion Science and Technology. 62 (2): 347–355. doi:10.13182/FST12-A14626. ISSN 1536-1055. S2CID 115384000.

[zhaos-16] Zhao, Shuyang; Heittola, Toni; Virtanen, Tuomas (2020). "Active learning for sound event detection". IEEE/ACM Transactions on Audio, Speech, and Language Processing. arXiv: 2002.05033 .

[:3-17] Bernard, Jürgen; Zeppelzauer, Matthias; Lehmann, Markus; Müller, Martin; Sedlmair, Michael (June 2018). "Towards User-Centered Active Learning Algorithms". Computer Graphics Forum. 37 (3): 121–132. doi:10.1111/cgf.13406. ISSN 0167-7055. S2CID 51875861.

[18] Desreumaux, Louis; Lemaire, Vincent (2020). "Learning Active Learning at the Crossroads? Evaluation and Discussion" (Proceedings of the Workshop on Interactive Adaptive Learning co-located with European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases {(ECML} {PKDD} 2020), Ghent, Belgium, 2020). S2CID 221794570.{{cite journal}}: Cite journal requires |journal= (help)

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]