Interactive machine learning for health informatics: when do we need the human-in-the-loop?

4.1 Example: subspace clustering

Grouping data sets into clusters based on their similarity is of enormous importance, and the similarity measure is the key aspect of the clustering process. Clustering is usually studied in unsupervised learning settings, but there is a huge problem with real-world data, because such data rarely result from the so-called well-behaved probabilistic models. Consequently, the study of interactive clustering algorithms is a growing area of research: Awasthi et al. [33] studied the problem of designing local algorithms for interactive clustering and proposed an interactive model and provided strong experimental evidence supporting the practical applicability of it. Their model starts with an initial clustering of the data, then the user can directly interact with the algorithm step-wise. In each step, the user provides limited feedback on the current clustering in the form of split-and-merge requests. The algorithm then makes a local edit to the clustering that is consistent with the user feedback. Such edits are aimed at improving the problematic part of the clustering pointed out by the human-in-the-loop. The goal of the algorithm is to quickly converge (using as few requests as possible) to a clustering that the user is happy with, which is called target clustering. More theoretical foundations of clustering with interactive feedback can be found in [34].

4.2 Example: protein folding

In protein structure prediction, there is still much interest in using amino acid interaction preferences to align (thread) a protein sequence to a known structural motif. The protein alignment decision problem (does there exist an alignment (threading) with a score less than or equal to K?) is NP-complete, and the related problem of finding the globally optimal protein threading is NP-hard. Therefore, no polynomial time algorithm is possible (unless P = NP). Consequently, the protein folding problem is NP-complete [35]. Health informatics is faced with many problems that (still) require the human-in-the-loop, e.g., genome annotation, image analysis, knowledge-base population, and protein structure. In some cases, humans are needed in vast quantities (e.g., in cancer research), whereas in others, we need just a few very specialized experts in certain fields (e.g., in the case of rare diseases). Crowdsourcing encompasses an emerging collection of approaches for harnessing such distributed human intelligence. Recently, the bioinformatics community has begun to apply crowdsourcing in a variety of contexts, yet few resources are available that describe how these human-powered systems work and how to use them effectively in scientific domains. Generally, there are large-volume micro-tasks and highly difficult mega-tasks [36]. A good example of such an approach is foldit, an experimental game which takes advantage of crowdsourcing for category discovery of new protein structures [37]. Crowdsourcing and collective intelligence (putting many experts-into-the-loop) would generally offer much potential to foster translational medicine (bridging biomedical sciences and clinical applications) by providing platforms upon which interdisciplinary workforces can communicate and collaborate [38].

4.3 Example: k-anonymization of patient data

Privacy preserving machine learning is an important issue, fostered by anonymization, in which a record is released only if it is indistinguishable from k other entities in the data. k-anonymity is highly dependent on spatial locality in order to effectively implement the technique in a statistically robust way, and in high dimensionalities data become sparse, hence, the concept of spatial locality is not easy to define. Consequently, it becomes difficult to anonymize the data without an unacceptably high amount of information loss [39]. Consequently, the problem of k-anonymization is on the one hand NP-hard, on the other hand the quality of the result obtained can be measured at the given factors: k-anonymity means that attributes are suppressed or generalized until each row in a database is identical with at least \(k-1\) other rows [40, 41]; l-diversity as extension of the k-anonymity model reduces the granularity of data representation by generalization and suppression so that any given record maps onto at least k other records in the data [42]; t-closeness is a refinement of l-diversity by reducing the granularity of a data representation, and treating the values of an attribute distinctly by taking into account the distribution of data values for that attribute [43]; and delta-presence, which links the quality of anonymization to the risk posed by inadequate anonymization [44]), but not with regard to the actual security of the data, i.e., the re-identification through an attacker. For this purpose, certain assumptions about the background knowledge of the hypothetical enemy must be made. With regard to the particular demographic and cultural clinical environment this is best done by a human agent. Thus, the problem of (k-)anonymization represents a natural application domain for iML.

4.4 Interim summary

Humans are very capable in the explorative learning of patterns from relatively few samples, while classic supervised ML needs large sets of data and long processing time. In the biomedical domain, often large sets of training data are missing, e.g., with rare diseases or with malfunctions of humans or machines. Moreover, in clinical medicine time is a crucial factor—where a medical doctor needs the results quasi in real-time, or at least in a very short time (less than 5 minutes), for example, in emergency medicine or intensive care. Rare diseases are often life threatening and require a rapid intervention—the lack of much data makes aML-approaches nearly impossible. An example for such a rare disease with only few available data sets is CADASIL (Cerebral Autosomal Dominant Arteriopathy with Subcortical Infarcts and Leukoencephalopathy), a disease, which is prevalent in 5 per 100,000 persons and is therefore the most frequent monogenic inherited apoplectic stroke in Germany.

Particularly in the patient admission, human agents have the advantage to perceive the total situation at a glance. This aptitude results from the ability of transfer learning, where knowledge can be transferred from one situation to another situation, in which model parameters, i.e., learned features or contextual knowledge are transferred.

The examples mentioned so far demonstrate that the application of iML-approaches in “real-world” situations sometimes can be advantageous. These examples demonstrate that human experience can help to reduce a search space of exponential possibilities drastically by heuristic selection of samples, thereby helping to solve NP-hard problems efficiently—or at least optimize them acceptably for a human end-user.

5.1 Reinforcement learning (RL)

Reinforcement learning (RL) was discussed by Turing [45] and is to date the most studied approach in ML. The theory behind RL is rooted in neuropsychological issues on behavior of how agents may optimize their control of an complex environment. Consequently, RL is a branch of ML concerned with using experience gained through interacting with the world and evaluative feedback to improve the ability of a system to generate behavioral decisions. This has been called the artificial intelligence problem in a microcosm because learning agents must act autonomously to perform well and to achieve their goals. Driven by the increasing availability of rich data, RL has achieved great results, including developments in fundamental ML-relevant areas, such as generalization, planning, exploration, and empirical methodology, leading to better applicability to real-world problems [46].

RL is different from supervised learning, where learning happens from examples provided by an external (human) supervisor. This is an important kind of learning; however, it alone is not sufficient for learning from interaction. In interactive problems, it is often impractical to obtain examples of desired behavior that are both correct and representative for all the situations in which the agent has to act. In unknown territory (which we want to explore), where one would expect learning to be most beneficial—an agent must be able to learn from its own experience. Despite all limitations, RL is the first field to seriously address the computational issues that arise when learning from interaction with an environment in order to achieve long-term goals, because it makes use of a formal framework defining the interaction between a learning agent and its environment in terms of states, actions, and rewards. This framework is intended to be a simple way of representing essential features of general AI problems and features including a sense of cause and effect, a sense of uncertainty and non-determinism, and the existence of explicit goals [47]. For an overview on some RL-algorithms please refer to [48].

In the typical RL-model, an agent is connected to its environment via perception and action, and in each step of interaction the agent receives as input i, some indication of the current state s, of the environment, and then the agent chooses an action a, which changes the state of the environment, and the value of this state transition is communicated to the agent through a scalar reinforcement signal r. The agent’s behavior B should now choose actions that tend to increase the long-run sum of values of the RL signal. It can learn to do this over time by systematic trial and error, guided by a wide variety of algorithms. Formally, the model consists of a discrete set of environment states \({\mathcal {S}}\), a discrete set of agent actions \({\mathcal {A}}\), and a set of scalar reinforcement signals, typically [0, 1] or \({\mathbb {R}}\).

An input function I determines how the agent views the environment state [49].

A current approach is multi-agent reinforcement learning (MARL), because the complexity of many tasks arising in complex domains makes it difficult to solve them with pre-programmed (single) agent behaviors, instead, agents can discover a solution on their own, using learning. A significant part of the research on multi-agent learning concerns RL-techniques [50].

However, to successfully apply RL in complex real-world situations, agents are confronted with a hard task: they must derive efficient representations of the environment from high-dimensional sensory inputs, and use these to generalize past experience to new situations. It is amazing that humans solve this problem through a harmonious combination of RL and hierarchical sensory processing systems. A big disadvantage is that such approaches have been applied in domains in which useful features can be handcrafted, or in domains with fully observable, low-dimensional state spaces. A very recent awesome work on Atari games demonstrated that a deep Q-network agent, receiving only the pixels and the game score as inputs, was able to surpass the performance of all previous algorithms and achieve a level comparable to that of a professional human games tester, thereby bridging the divide between high-dimensional sensory inputs and actions, resulting in the first artificial agent that is capable of learning to excel at a diverse array of challenging tasks [51], which opens many avenues for further research.

Interactive RL is not yet established in the health informatics domain, although there is some previous work from other domains, particularly in training of human-centric robots. The general question is how human interaction can influence the ML process and how natural human feedback can improve standard RL-algorithms [52], [53]. Another example includes a multi-agent-based reinforcement learning algorithm, in which the interactions between travelers and the environment are considered to simulate temporal–spatial characteristics of activity-travel patterns within a town [54]. It remains open for future research to transfer these insights into the health informatics domain.

5.2 Preference learning (PL)

Preference Learning (PL) is aiming to learn a predictive preference model from observations (empirical data) that reveal, explicitly or implicitly, information about the specific preferences of a user or a group of users. This can be supported by methods for preference mining, e.g., to gain knowledge about users likes and dislikes to provide personal and custom-tailored recommendations [55]. PL can be seen as a natural link between ML and decision support, and was primarily applied in information retrieval with the central task of learning to rank [56, 57]. Characteristic learning tasks include label ranking (see example below), instance ranking, and object ranking. PL is meanwhile an established subfield in ML [58] and a comprehensive and modern overview can be found in [59]. The underlying theoretical basis for PL can be found in human concept learning, the ability of humans to think in terms of abstractions: humans are able to order their experience into coherent categories by classifying a new situation as a member of a collection of previous situations for which responses may be appropriate [60].

One of the unsolved problems in the field of (human) concept learning, and which are highly relevant for ML research, concerns the factors that determine the subjective difficulty of concepts: why are some concepts psychologically extremely simple and easy to learn, while others seem to be extremely difficult, complex, or even incoherent? These questions have been studied since the 1960s by cognitive scientists, but are still unanswered. Also more recent characterizations of concepts as prototypes rather than logical rules leave it unsolved [61, 62].

The preference learning (PL) model is described by a seminal work of [63]: Let us show to the end-user M pairs of items. In each case, the user has to chose which item to be preferred. Consequently, the data set consists of the ranked pairs \({\mathcal {D}} = \{{\mathbf {r}}_{k} \succ {\mathbf {c}}_{k}; k = 1, ..., M\}.\) The symbol \(\succ\) indicates the preference of \({\mathbf r _{k}}\) over \({\mathbf c _{k}}\).

The elements \({\mathbf r _{k}}\) and \({\mathbf c _{k}}\) of the ranked pairs \({\mathbf r _{k}} \succ {\mathbf c _{k}}\) are taken from a set of training data with N elements.

The goal is to compute the item \({\mathbf x }\) with the highest user valuation in the least comparisons as possible. The valuation functions for \({\mathbf r }\) and \({\mathbf c }\) can be modeled as follows: \(u({\mathbf r _{k}}) = f({\mathbf r _{k}}) + e_rk\) and \(u({\mathbf c _{k}}) = f({\mathbf c _{k}}) + e_ck.\)

In these functions the noise terms are Gaussian, hence, a non-parametric Gaussian process prior to the unknown mean valuation can be assigned [64].

Such random utility models have a long tradition in psychology and economy and go back to Thurstone [65], and Bush and Mosteller [66].

Much more work has been done in robot learning, e.g., Knox et al. [69] presented a case study of applying a framework for learning from human feedback to a physically embodied robot. They also provided a demonstration of the ability to train multiple behaviors by such feedback without algorithmic modifications, from free-form human-generated feedback without any further guidance or evaluative feedback. Wilson et al. [70] considered the problem of learning control policies via trajectory preference queries to an expert (trajectory planning is a major challenge in robotics and autonomous vehicles). In particular, the agent confronts a human expert with pairs of policies, and the expert indicates which trajectory is preferred. The agent's goal is to elicit a latent target policy from the human expert with as few queries as possible. To tackle this problem, the authors proposed a Bayesian model of the querying process and introduced two methods that exploit this model to actively select expert queries. Jain et al. [71] considered the problem of learning good trajectories for manipulation tasks, which is challenging because the criteria defining good trajectories varies with users, tasks, and environments. For this purpose, they proposed a co-active online learning framework for teaching robots the preferences of its users for object manipulation tasks. The novelty of their approach was in the type of feedback expected from the user: the human user was not required to demonstrate optimal trajectories as training data, but merely needs to iteratively provide trajectories that slightly improve over the trajectory currently proposed by the system.

In the medical domain—as in many other domains—we have many situations where the creative input of a human is still required, further examples are the fields of active learning, active preference learning, and interactive learning and optimization which we will very briefly discuss next.

5.3 Active learning (AL)

The idea behind active learning (AL) is that a ML-algorithm can achieve greater accuracy with fewer training labels, if it is allowed to choose the data from which it learns. An active learner may pose queries, usually in the form of unlabeled data instances to be labeled by an oracle (e.g., a human annotator, for example, see [72]). Active learning is well-motivated in many modern machine learning problems, where unlabeled data may be abundant or easily obtained, but labels are difficult, time-consuming, or expensive to obtain [73].

For many types of ML-algorithms, one can compute the statistically optimal way to select training data. While the techniques for neural networks are computationally expensive and approximate, the techniques for mixtures of Gaussians and locally weighted regression are both efficient and accurate [74]. A good example was presented by Warmuth et al. [75] wherein they investigated the following data mining problem from computer-aided drug design: From a large collection of compounds, find those that bind to a target molecule in as few iterations of biochemical testing as possible. In each iteration, a comparatively small batch of compounds is screened for binding activity toward this target. They employed AL for selecting the successive batches and the main selection strategy was based on the maximum margin hyperplanes generated by Support Vector Machines. The hyperplane separates the current set of active compounds from the inactive compounds and has the largest possible distance from any labeled compound.

A further approach is interactive Learning and Optimization (ILO), which started with the paper by Brochu et al. [63]—already discussed in the Sect. 5.2. They proposed an AL algorithm to learn a continuous valuation model from discrete preferences. Their algorithm automatically decided what items are best presented to a human in order to find the item that they value highly in as few trials as possible, and exploit so-called quirks; peculiarities of human psychology to minimize time and cognitive burden. For this purpose, their algorithm maximized the expected improvement at each query without accurately modeling the entire valuation surface, which would otherwise be computationally expensive. This problem is hard, due to the fact that the space of choices is infinite. Meanwhile ILO has been applied mostly to information retrieval, for example Yue and Joachims [76] presented an online learning framework, tailored toward real-time learning from observed user behavior in search engines. They only required pairwise comparisons which were shown to be reliably inferred from implicit feedback. In their work, they applied the dueling bandits problem [77]. Progress in this area can lead to cost-effective systems for a variety of application domains such as personalized search.

Bayesian approaches to utility elicitation typically adopt (myopic) expected value of information (EVOI) as a natural criterion for selecting queries. However, EVOI-optimization is usually computationally prohibitive.

Viappiani and Boutilier [78] examined the expected value of information (EVOI) optimization using choice queries, i.e., queries in which a user is asked to select his/her most preferred product from a set. They showed that under very general assumptions, the optimal choice query with regard to EVOI coincides with the optimal recommendation set, i.e., a set maximizing the expected utility of the user selection. Since recommendation set optimization is a simpler, sub-modular problem, this can greatly reduce the complexity of both exact and approximate (greedy) computation of optimal choice queries. They also examined the case where user responses to choice queries are error-prone (using both constant and mixed multinomial logit noise models) and provide worst-case guarantees. Finally, they presented a local search technique for query optimization that worked well with large outcome spaces.

An approach can benefit from the ILO scheme in order to gradually shape, for example, the interestingness function (typically not all patterns with same statistical properties are equally interesting) and the methodology (sequences of decisions to process the vast amount of heterogeneous databases) most appropriate to discover such interesting patterns. The question “What is interesting?” is one of the most pressing questions in our fields.

6.1 Example: multi-task learning

Multi-task learning (MTL) aims to improve the prediction performance by learning a problem together with multiple, different but related other problems through shared parameters or a shared representation. The underlying principle is bias learning based on probable approximately correct learning (PAC learning) [79]. To find such a bias is still the hardest problem in any ML task and essential for the initial choice of an appropriate hypothesis space, which must be large enough to contain a solution, and small enough to ensure a good generalization from a small number of data sets. Existing methods of bias generally require the input of a human-expert-in-the-loop in the form of heuristics and domain knowledge to ensure the selection of an appropriate set of features, as such features are the key to learning and understanding. However, such methods are limited by the accuracy and reliability of the experts knowledge (robustness of the human) and also by the extent to which that knowledge can be transferred to new tasks (see next subsection). Baxter [80] introduced a model of bias learning which builds on the PAC learning model which concluded that learning multiple related tasks reduces the sampling burden required for good generalization and that the bias learnt on sufficiently many training tasks is likely to be good for learning novel tasks drawn from the same environment (the problem of transfer learning to new environments is discussed in the next subsection). A practical example is regularized MTL [81], which is based on the minimization of regularization functionals similar to Support Vector Machines (SVMs), which have been successfully used in the past for singletask learning. The regularized MTL approach allows to model the relation between tasks in terms of a novel kernel function that uses a taskcoupling parameter and largely outperforms singletask learning using SVMs. However, multi-task SVMs are inherently restricted by the fact that SVMs require each class to be addressed explicitly with its own weight vector. In a multi-task setting this requires the different learning tasks to share the same set of classes. An alternative formulation for MTL is an extension of the large margin nearest neighbor algorithm (LMNN) [82]. Instead of relying on separating hyperplanes, its decision function is based on the nearest neighbor rule which inherently extends to many classes and becomes a natural fit for MTL. This approach outperforms state-of-the-art MTL classifiers, however, many open research challenges remain open in this area [83].

6.2 Example: transfer learning (generalization)

A huge problem in ML is the phenomenon of catastrophic forgetting, i.e., when having learned one task and being transferred to another task the ML-algorithm “forgets” how to perform the learned task. This is a well-known problem which affects ML-systems and was first described in the context of connectionist networks [84]; natural cognitive systems rarely completely disrupt or erase previously learned information, i.e., natural cognitive systems do not forget “catastrophically” [85]. Consequently, the challenge is to discover how to avoid the problem of catastrophic forgetting, which is a current hot topic [86].

According to Pan and Yang [87], a major assumption in many ML-algorithms is that both the training data and future (unknown) data must be in the same feature space and are required to have the same distribution. In many real-world applications, particularly in the health domain, this is not the case: Sometimes we have a classification task in one domain of interest, but we only have sufficient training data in another domain of interest, where the latter data may be in a completely different feature space or follow a different data distribution.

In such cases transfer learning would greatly improve the performance of learning by avoiding much expensive data-labeling efforts, however, many open questions remain for future research [88].

6.3 Example: multi-agent hybrid systems

Multi-Agent Systems (MAS) are collections of many agents interacting with each other. They can either share a common goal (for example an ant colony, bird flock, or fish swarm), or they can pursue their own interests (for example as in an open-market economy).

MAS can be traditionally characterized by the facts that (a) each agent has incomplete information and/or capabilities for solving a problem, (b) agents are autonomous, so there is no global system control; (c) data are decentralized; and (d) computation is asynchronous [89]. For the health domain, of particular interest is the consensus problem, which formed the foundation for distributed computing [90].

The roots are in the study of (human) experts in group consensus problems: Consider a group of humans who must act together as a team and each individual has a subjective probability distribution for the unknown value of some parameter; a model which describes how the group reaches agreement by pooling their individual opinions was described by DeGroot [91] and was used decades later for the aggregation of information with uncertainty obtained from multiple sensors [92] and medical experts [93].

On this basis, Olfati-Saber et al. [94] presented a theoretical framework for analysis of consensus algorithms for networked multi-agent systems with fixed or dynamic topology and directed information flow. In complex real-world problems, e.g., for the epidemiological and ecological analysis of infectious diseases, standard models based on differential equations very rapidly become unmanageable due to too many parameters, and here MAS can also be very helpful [95]. Moreover, collaborative multi-agent reinforcement learning has a lot of research potential for machine learning [96].

Obviously, there would be a lot of opportunities and research challenges in integrating a human agent to form hybrid systems [97], and making use of Multi-Agent Hybrid Systems (MAHS): [98].