Recent Research Projects


    Reinforcement Learning with Qualitative Feedback

    Reinforcement learning (RL) is an established paradigm for autonomous learning from interaction with an environment in order to achieve long-term goals. Informally speaking, the task of an RL agent is to successively react to changes of the environment, by properly choosing actions, in order to maximize its accumulated reward over time. Conventional RL methods are confined to deal with numerical rewards. However, in many applications, only qualitative reward signals are readily available. Moreover, a restriction to numerical reward functions also hampers the exploitation of other conceivable sources of feedback, such as external advice. Our goal in this project is to generalize the standard RL framework so as to allow for more general types of feedback, notably non-numerical rewards and qualitative advice. Building on novel methods for preference learning, the basic idea is to provide the RL agent with qualitative policy models, such as ranking functions that allow for sorting its available actions from most to least promising, as well as algorithms for learning such models. While the focus of the project is on the development of theoretical and methodological foundations of a “preference-based reinforcement learning”, we also envision two case studies putting our ideas into practice, one in the field of game playing and another one in a medical context.

    Modeling, Learning and Processing of Know-How in CBR using Preference-Based Methods

    Case-based reasoning (CBR) is a well-established problem solving paradigm that has been used in a wide range of real-world applications. Despite its great practical success, work on the theoretical foundations of CBR is still under way, and a coherent and universally applicable methodological
    framework is yet missing. Drawing on recent research on preference handling in Artificial Intelligence and related fields, the vision and ambition of this project is to develop such a framework on the basis of formal concepts and methods for knowledge representation and reasoning with preferences. A preference-based approach to CBR appears to be appealing for several reasons, notably because case-based experiences naturally lend themselves to representations in terms of preference relations, even when not dealing with preference information in an explicit way. Moreover, the flexibility and expressiveness of a preference-based formalism well accommodate the uncertain and approximate nature of case-based problem solving. Special attention will be payed to the common subject of the project cluster, namely the use of CBR for exploiting knowledge from Internet communities. In this context, preference information occurs quite naturally and needs to be handled in an adequate way. Jointly with the project partners, a concrete application in the domain of cooking shall be realized.

    Funding: DFG (since 2011)
    Contact: Eyke Hüllermeier
    Cooperation: Business Information Systems II, University of Trier, Germany

    Learning by Pairwise Comparison for Problems with Structured Output Spaces

    Learning by pairwise comparison (LPC) is a well-established technique in the field of machine learning, where it allows for reducing the problem of polychotomous to the one of dichotomous classification. Recently, some successful attempts to apply this technique also in more complex learning scenarios have been made. In particular, there is a current trend in machine learning to study supervised learning problems involving structured output spaces, such as multi-label, ordered, and hierarchical classification as well as label ranking. Motivated by first promising though solitary results, this project aims at exploring the potential and broadening the scope of LPC in a more systematic way. The ultimate goal of the project is a general and coherent framework in which problems with structured output spaces can be solved by LPC in a principled manner. Apart from corresponding methodological contributions, an important aspect of the project concerns the investigation of theoretical properties of LPC, especially a characterization of the class of problems it can solve and the computational complexity these solutions bring about. Finally, a thorough empirical evaluation and comparison with alternative algorithms shall be conducted for a variety of concrete and practically relevant learning problems.

    Funding: DFG (2007-2013)
    Contact: Eyke Hüllermeier, Weiwei Cheng
    Cooperation: AG Knowledge Engineering, Fachbereich Informatik, TU Darmstadt


    Mining Graph-Structured Biological Data

    Recent applications from the natural and life sciences are particularly interesting from a data mining point of view and involve new challenges as to both data modeling and algorithmic solutions. In particular, these fields are often concerned with the study of data that possess a complex internal structure, and that cannot be mapped onto “flat” feature vectors of a fixed length without an inherent loss of essential information. On the other hand, graph-based models appear to be especially suitable for this type of data. This project therefore aims at developing graph-based modeling and data mining methods, with a special emphasis on so-called multiple graph alignment as a novel tool for discovering approximately conserved patterns in graph-structured data. Regarding applications, we are especially interested in analyzing protein binding sites in the context of structure-based drug design.

    Funding: DFG (2010-2014)
    Contact: Eyke Hüllermeier, Thomas Fober, Marco Mernberger
    Cooperation: Institut für Pharmazeutische Chemie, Philipps-Universität Marburg


    Preference Learning: Methods and Applications in Personalized Information Systems

    The increasing trend toward personalization of products and services in e-commerce and various other fields requires computerized methods for discovering the preferences of individuals. And indeed, methods for learning and predicting preferences in an automatic way are among the very recent research topics in disciplines such as machine learning and recommendation systems. The project's principal objective is to develop methods for the automatic acquisition of fuzzy preference models. From an application point of view, such models are especially appealing as they are more expressive and flexible than classical models. Moreover, the underlying theory of fuzzy sets provides a coherent framework for handling different types of imprecise, uncertain and incomplete information, a point of critical importance in learning of and reasoning with preferences. A second goal of the project is to exploit fuzzy preference learning in the context of recommendation systems, thereby complementing existing methods such as collaborative filtering.

    Funding: DFG (2005-2011)
    Contact: Eyke Hüllermeier, Robin Senge
    Cooperation: IRIT, Toulouse, Siemens Corporate Research, Princeton


    Data‐Driven Design of Evolving Fuzzy Systems: Enhancing Interpretability, Reliability, and User‐Interaction

    An evolving fuzzy system (EFS) is a system that permanently adapts itself to changing environmental conditions. This is done by adjusting its structure and parameters on the basis of observed data. Research in this emerging field has so far mainly focused on learning models with a high (predictive) accuracy. Despite its importance, this criterion is not sufficient, since overly complex models that cannot be understood will likely be refused in practical applications. Without any doubt, fuzzy systems do have the potential to offer both, accuracy and transparency, and the goal of this project is to exploit this high potential. More concretely, the goal is to produce concepts, methods, and algorithms for making EFS more user‐friendly. First of all, this will be achieved by developing methods for reducing the complexity of fuzzy models, thereby making them more transparent and possibly amenable to interpretable linguistic representations. Another important user requirement is reliability. In this regard, different types of uncertainty concerning the model itself and its predictions have to be captured and represented; ideally, a model is “self‐aware” in the sense of being able to judge its own reliability. Finally, novel visualization techniques and methods shall be developed that allow a human user to interact with the learning system in a dynamical way. Jointly, these contributions will greatly increase the practical usefulness and applicability of evolving fuzzy systems.

    Funding: DFG (2010-2014)
    Contact: Eyke Hüllermeier, Ammar Shaker
    Cooperation: Department of Knowledge-Based Mathematical System, University of Linz, Austria


    Quality Engineering

    Quality control and quality engineering are concerned with developing systems to ensure that products or services are designed and produced to meet or exceed customer requirements. In this regard, the goal of this project is to develop an intelligent assessment system that imitates as closely as possible the product evaluation by a human expert. In fact, even though the collection of measurement values and curves characterizing technical products has already been automated to a large extent by means of robot-based measurement techniques, the evaluation of a product based on these measurements remains a difficult issue. In this respect, aspects such as gradual transitions between satisfaction and non-satisfaction of a criterion as well as logical and partially compensatory aggregations of individual evaluations into an overall assessment are of central importance. Therefore, the project is founded on fuzzy set theory and fuzzy logic as an underlying theoretical framework.

    Funding: BMWI (2006-2009)
    Contact: Eyke Hüllermeier, Yu Yi
    Partner: Battenberg Robotic, Marburg