Maria Teresa Llano
Research Associate
About me
I’m a Research Associate in the Computational Creativity Group at Goldsmiths, University of London. I was awarded my PhD at Heriot-Watt University where I worked on the Automated Discovery of Invariants and Refinement Plans for Formal Modelling in Event-B. Currently I’m part of the WHIM Project which aims at building the What-If Machine, a software system able to invent, evaluate and present fictional ideas with cultural value. [CV]
My research interests are in the areas of Artificial Intelligence and Formal Methods for Software Engineering. In particular:
- Computational Creativity
- Natural Language Processing
- Machine Learning
- Formal Approaches to Software Engineering
- Refinement Techniques
- Automated Theory Formation
You can contact me at .[at]...
Publications
Llano, Maria Teresa; Colton, Simon; Hepworth, Rose; Gow, Jeremy Automated Fictional Ideation via Knowledge Base Manipulation Journal Article In: Cognitive Computation, 8 (2), pp. 153-174, 2016. @article{Llano2011usingb, title = {Automated Fictional Ideation via Knowledge Base Manipulation}, author = {Maria Teresa Llano and Simon Colton and Rose Hepworth and Jeremy Gow}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/10/llano_jcc2015.pdf}, year = {2016}, date = {2016-10-01}, journal = {Cognitive Computation}, volume = {8}, number = {2}, pages = {153-174}, abstract = {The invention of fictional ideas (ideation) is often a central process in the creative production of arte- facts such as poems, music and paintings, but has barely been studied in the computational creativity community. We present here a general approach to automated fictional ideation that works by manipulating facts specified in knowledge bases. More specifically, we specify a number of constructions which, by altering and combining facts from a knowledge base, result in the generation of fictions. Moreover, we present an instantiation of these construc- tions through the use of ConceptNet, a database of common sense knowledge. In order to evaluate the success of these constructions, we present a curation analysis that calculates the proportion of ideas which pass a typicality judgement. We further evaluate the output of this approach through a crowd-sourcing experiment in which participants were asked to rank ideas. We found a positive correlation between the participant’s rankings and a chaining inference technique that automatically assesses the value of the fic- tions generated through our approach. We believe that these results show that this approach constitutes a firm basis for automated fictional ideation with evaluative capacity.}, keywords = {}, pubstate = {published}, tppubtype = {article} } The invention of fictional ideas (ideation) is often a central process in the creative production of arte- facts such as poems, music and paintings, but has barely been studied in the computational creativity community. We present here a general approach to automated fictional ideation that works by manipulating facts specified in knowledge bases. More specifically, we specify a number of constructions which, by altering and combining facts from a knowledge base, result in the generation of fictions. Moreover, we present an instantiation of these construc- tions through the use of ConceptNet, a database of common sense knowledge. In order to evaluate the success of these constructions, we present a curation analysis that calculates the proportion of ideas which pass a typicality judgement. We further evaluate the output of this approach through a crowd-sourcing experiment in which participants were asked to rank ideas. We found a positive correlation between the participant’s rankings and a chaining inference technique that automatically assesses the value of the fic- tions generated through our approach. We believe that these results show that this approach constitutes a firm basis for automated fictional ideation with evaluative capacity. |
Llano, Maria Teresa; Guckelsberger, Christian; Hepworth, Rose; Gow, Jeremy; Corneli, Joseph; Colton, Simon What If A Fish Got Drunk? Exploring the Plausibility of Machine-Generated Fictions Inproceedings In: Proc. 7th Int. Conf. Computational Creativity, 2016. @inproceedings{Llano2016, title = {What If A Fish Got Drunk? Exploring the Plausibility of Machine-Generated Fictions}, author = {Maria Teresa Llano and Christian Guckelsberger and Rose Hepworth and Jeremy Gow and Joseph Corneli and Simon Colton}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/06/ICCC_Plausibility_camera_ready-1.pdf}, year = {2016}, date = {2016-07-01}, booktitle = {Proc. 7th Int. Conf. Computational Creativity}, abstract = {Within the WHIM project, we study fictional ideation: processes for automatically inventing, assessing and presenting fictional ideas. Here we examine the foundational notion of the plausibility of fictional ideas, by performing an empirical study to surface the factors that affect judgements of plausibility. Our long term aim is to formalise a computational method which captures some intuitive notions of plausibility and can predict how certain types of people will assess the plausibility of certain types of fictional ideas. This paper constitutes a first firm step towards this aim.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Within the WHIM project, we study fictional ideation: processes for automatically inventing, assessing and presenting fictional ideas. Here we examine the foundational notion of the plausibility of fictional ideas, by performing an empirical study to surface the factors that affect judgements of plausibility. Our long term aim is to formalise a computational method which captures some intuitive notions of plausibility and can predict how certain types of people will assess the plausibility of certain types of fictional ideas. This paper constitutes a first firm step towards this aim. |
Corneli, Joseph; Jordanous, Anna; Shepperd, Rosie; Llano, Maria Teresa; Misztal, Joanna; Colton, Simon; Guckelsberger, Christian Computational Poetry Workshop: Making Sense of Work in Progress Inproceedings In: Proc. 6th Int. Conf. Computational Creativity, 2015. @inproceedings{Corneli2015, title = {Computational Poetry Workshop: Making Sense of Work in Progress}, author = {Joseph Corneli and Anna Jordanous and Rosie Shepperd and Maria Teresa Llano and Joanna Misztal and Simon Colton and Christian Guckelsberger}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/03/corneli_iccc15_poetry-1.pdf}, year = {2015}, date = {2015-01-01}, booktitle = {Proc. 6th Int. Conf. Computational Creativity}, abstract = {Creativity cannot exist in a vacuum; it develops through feedback, learning, reflection and social interaction with others. However, this perspective has been relat- ively under-investigated in computational creativity re- search, which typically examines systems that operate individually. We develop a thought experiment showing how structured dialogues can help develop the creative aspects of computer poetry. Centrally in this approach, we ask questions of a poem, inviting it to tell us in what way it may be considered a “creative making.”}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Creativity cannot exist in a vacuum; it develops through feedback, learning, reflection and social interaction with others. However, this perspective has been relat- ively under-investigated in computational creativity re- search, which typically examines systems that operate individually. We develop a thought experiment showing how structured dialogues can help develop the creative aspects of computer poetry. Centrally in this approach, we ask questions of a poem, inviting it to tell us in what way it may be considered a “creative making.” |
Llano, Maria Teresa; Cook, Michael; Guckelsberger, Christian; Colton, Simon; Hepworth, Rose Towards the Automatic Generation of Fictional Ideas for Games Inproceedings In: Experimental AI in Games (EXAG’14), a workshop collocated with the tenth annual AAAI conference on artificial intelligence and interactive digital entertainment (AIIDE’14). AAAI Publications, 2014. @inproceedings{llano2014towards, title = {Towards the Automatic Generation of Fictional Ideas for Games}, author = { Maria Teresa Llano and Michael Cook and Christian Guckelsberger and Simon Colton and Rose Hepworth}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/09/llano_exag14.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {Experimental AI in Games (EXAG’14), a workshop collocated with the tenth annual AAAI conference on artificial intelligence and interactive digital entertainment (AIIDE’14). AAAI Publications}, abstract = {The invention of fictional ideas is often a central pro- cess in the creative production of artefacts such as po- ems, music, paintings and games. Currently, fictional ideation is being studied by the Computational Creativ- ity community within the WHIM European project. The aim of WHIM is to develop the What-If Machine, a soft- ware system capable of inventing, evaluating and pre- senting fictional ideas with cultural value. In this pa- per we explore the potential applications of the What-If Machine in the context of games. Specifically, we pro- pose ways in which the What-If Machine can be used as an assistant for the design of games, by providing ideas about characters, the environment, etc., as well as a creative system during gameplay, through interesting interactions with the player.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } The invention of fictional ideas is often a central pro- cess in the creative production of artefacts such as po- ems, music, paintings and games. Currently, fictional ideation is being studied by the Computational Creativ- ity community within the WHIM European project. The aim of WHIM is to develop the What-If Machine, a soft- ware system capable of inventing, evaluating and pre- senting fictional ideas with cultural value. In this pa- per we explore the potential applications of the What-If Machine in the context of games. Specifically, we pro- pose ways in which the What-If Machine can be used as an assistant for the design of games, by providing ideas about characters, the environment, etc., as well as a creative system during gameplay, through interesting interactions with the player. |
Colton, Simon; Pease, Alison; Corneli, Joseph; Cook, Michael; Llano, Maria Teresa Assessing Progress in Building Autonomously Creative Systems Inproceedings In: Proceedings of the Fifth International Conference on Computational Creativity, pp. 137–145, 2014. @inproceedings{colton2014assessing, title = {Assessing Progress in Building Autonomously Creative Systems}, author = { Simon Colton and Alison Pease and Joseph Corneli and Michael Cook and Maria Teresa Llano}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/10/colton_iccc2014.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {Proceedings of the Fifth International Conference on Computational Creativity}, pages = {137--145}, abstract = {Determining conclusively whether a new version of software creatively exceeds a previous version or a third party system is dicult, yet very important for scientific approaches in Computational Creativity research. We argue that software product and process need to be assessed simultaneously in assessing progress, and we introduce a diagrammatic formalism which exposes various timelines of creative acts in the construction and execution of successive versions of artefactgenerating software. The formalism enables estimations of progress or regress from system to system by comparing their diagrams and assessing changes in quality, quantity and variety of creative acts undertaken; audience perception of behaviours; and the quality of artefacts produced. We present a case study in the building of evolutionary art systems, and we use the formalism to highlight various issues in measuring progress in the building of creative systems.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Determining conclusively whether a new version of software creatively exceeds a previous version or a third party system is dicult, yet very important for scientific approaches in Computational Creativity research. We argue that software product and process need to be assessed simultaneously in assessing progress, and we introduce a diagrammatic formalism which exposes various timelines of creative acts in the construction and execution of successive versions of artefactgenerating software. The formalism enables estimations of progress or regress from system to system by comparing their diagrams and assessing changes in quality, quantity and variety of creative acts undertaken; audience perception of behaviours; and the quality of artefacts produced. We present a case study in the building of evolutionary art systems, and we use the formalism to highlight various issues in measuring progress in the building of creative systems. |
Colton, Simon; Ramezani, Ramin; Llano, Maria Teresa The HR3 discovery system: Design decisions and implementation details Inproceedings In: Proceedings of the AISB symposium on Computational Scientific Discovery, 2014. @inproceedings{colton2014hr3, title = {The HR3 discovery system: Design decisions and implementation details}, author = { Simon Colton and Ramin Ramezani and Maria Teresa Llano}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/07/colton_aisb14b.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {Proceedings of the AISB symposium on Computational Scientific Discovery}, abstract = {Automated Theory Formation is a hybrid AI technique which has been implemented in two scientific discovery systems, HR1 and HR2, both of which have been used successfully in vari- ous applications. We describe here the latest iteration in the HR se- ries, in terms of the lessons learned from the successes and failures of the previous versions, and how these lessons have informed our design choices and the implementation details of the new version. We also present two case studies: a synthetic domain mirroring an aspect of medical diagnosis, and invariant discovery in formal meth- ods. In each case, we compare HR3 with HR2 to highlight various improvements in the new version.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Automated Theory Formation is a hybrid AI technique which has been implemented in two scientific discovery systems, HR1 and HR2, both of which have been used successfully in vari- ous applications. We describe here the latest iteration in the HR se- ries, in terms of the lessons learned from the successes and failures of the previous versions, and how these lessons have informed our design choices and the implementation details of the new version. We also present two case studies: a synthetic domain mirroring an aspect of medical diagnosis, and invariant discovery in formal meth- ods. In each case, we compare HR3 with HR2 to highlight various improvements in the new version. |
Llano, Maria Teresa; Hepworth, Rose; Colton, Simon; Gow, Jeremy; Charnley, John; Lavrač, Nada; Žnidaršič, Martin; Perovšek, Matic; Granroth-Wilding, Mark; Clark, Stephen Baseline Methods For Automated Fictional Ideation Inproceedings In: Proceedings of the 5th international conference on computational creativity, 2014. @inproceedings{llano2014baseline, title = {Baseline Methods For Automated Fictional Ideation}, author = { Maria Teresa Llano and Rose Hepworth and Simon Colton and Jeremy Gow and John Charnley and Nada Lavrač and Martin Žnidaršič and Matic Perovšek and Mark Granroth-Wilding and Stephen Clark}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/09/llano_iccc2014.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {Proceedings of the 5th international conference on computational creativity}, abstract = {The invention of fictional ideas (ideation) is often a central process in the creative production of artefacts such as po- ems, music and paintings, but has barely been studied in the Computational Creativity community. We present here three baseline approaches for automated fictional ideation, using methods which invert and alter facts from the ConceptNet and ReVerb databases, and perform bisociative discovery. For each method, we present a curation analysis, by calculating the proportion of ideas which pass a typicality evaluation. We further evaluate one ideation approach through a crowd- sourcing experiment in which participants were asked to rank ideas. The results from this study, and the baseline meth- ods and methodologies presented here, constitute a firm basis on which to build more sophisticated models for automated ideation with evaluative capacity.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } The invention of fictional ideas (ideation) is often a central process in the creative production of artefacts such as po- ems, music and paintings, but has barely been studied in the Computational Creativity community. We present here three baseline approaches for automated fictional ideation, using methods which invert and alter facts from the ConceptNet and ReVerb databases, and perform bisociative discovery. For each method, we present a curation analysis, by calculating the proportion of ideas which pass a typicality evaluation. We further evaluate one ideation approach through a crowd- sourcing experiment in which participants were asked to rank ideas. The results from this study, and the baseline meth- ods and methodologies presented here, constitute a firm basis on which to build more sophisticated models for automated ideation with evaluative capacity. |
Llano, Maria Teresa; Hepworth, Rose; Colton, Simon; Charnley, John; Gow, Jeremy Automating Fictional Ideation Using ConceptNet Inproceedings In: Proceedings of the AISB14 symposium on computational creativity, 2014. @inproceedings{llano2014automating, title = {Automating Fictional Ideation Using ConceptNet}, author = {Maria Teresa Llano and Rose Hepworth and Simon Colton and John Charnley and Jeremy Gow}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/09/llano_aisb14.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {Proceedings of the AISB14 symposium on computational creativity}, abstract = {The invention of fictional ideas (ideation) is often a cen- tral process in producing artefacts such as poems, music and paint- ings in a creative way. Automated fictional ideation should, there- fore, be of much interest in the study of Computational Creativity, but only a few approaches have been explored. We describe here the preliminary results of a new method for automated generation and evaluation of fictional ideas which uses ConceptNet, a semantic net- work. We evaluate the results obtained through a small study that involves participants scoring ideas via an online survey. We believe this approach constitutes a firm basis on which a more sophisticated model for automated creative ideation can be built.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } The invention of fictional ideas (ideation) is often a cen- tral process in producing artefacts such as poems, music and paint- ings in a creative way. Automated fictional ideation should, there- fore, be of much interest in the study of Computational Creativity, but only a few approaches have been explored. We describe here the preliminary results of a new method for automated generation and evaluation of fictional ideas which uses ConceptNet, a semantic net- work. We evaluate the results obtained through a small study that involves participants scoring ideas via an online survey. We believe this approach constitutes a firm basis on which a more sophisticated model for automated creative ideation can be built. |
Charnley, John; Colton, Simon; Llano, Maria Teresa The FloWr Framework: Automated Flowchart Construction, Optimisation and Alteration for Creative Systems Inproceedings In: Proceedings of the 5th international conference on computational creativity, 2014. @inproceedings{charnley2014flowr, title = {The FloWr Framework: Automated Flowchart Construction, Optimisation and Alteration for Creative Systems}, author = { John Charnley and Simon Colton and Maria Teresa Llano}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/08/charnley_iccc2014.pdf}, year = {2014}, date = {2014-01-01}, booktitle = {Proceedings of the 5th international conference on computational creativity}, volume = {9}, abstract = {We describe the FloWr framework for implementing creative systems as scripts over processes and manipulated visually as flowcharts. FloWr has been specifically developed to be able to automatically optimise, alter and ultimately generate novel flowcharts, thus innovating at process level. We describe the fundamental architecture of the framework and provide ex- amples of creative systems which have been implemented in FloWr. Via some preliminary experimentation, we demon- strate how FloWr can optimise a given system for efficiency and yield, alter input parameters to increase unexpectedness, and build novel generative systems automatically.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } We describe the FloWr framework for implementing creative systems as scripts over processes and manipulated visually as flowcharts. FloWr has been specifically developed to be able to automatically optimise, alter and ultimately generate novel flowcharts, thus innovating at process level. We describe the fundamental architecture of the framework and provide ex- amples of creative systems which have been implemented in FloWr. Via some preliminary experimentation, we demon- strate how FloWr can optimise a given system for efficiency and yield, alter input parameters to increase unexpectedness, and build novel generative systems automatically. |
Pease, Alison; Smaill, Alan; Colton, Simon; Ireland, Andrew; Llano, Maria Teresa; Ramezani, Ramin; Grov, Gudmund; Guhe, Markus Applying Lakatos-style reasoning to AI problems Book Chapter In: Thinking Machines and the philosophy of computer science: Concepts and principles., Chapter 10, pp. 149–174, Information Science Reference, 2010, ISBN: 9781616920142. @inbook{pease2010applying, title = {Applying Lakatos-style reasoning to AI problems}, author = {Alison Pease and Alan Smaill and Simon Colton and Andrew Ireland and Maria Teresa Llano and Ramin Ramezani and Gudmund Grov and Markus Guhe}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/08/pease_tm10-1.pdf}, doi = {10.4018/978-1-61692-014-2.ch010}, isbn = {9781616920142}, year = {2010}, date = {2010-01-01}, booktitle = {Thinking Machines and the philosophy of computer science: Concepts and principles.}, journal = {Thinking Machines and the philosophy of computer science: Concepts and principles}, pages = {149--174}, publisher = {Information Science Reference}, chapter = {10}, abstract = {One current direction in AI research is to focus on combining different reasoning styles such as deduction, induction, abduction, analogical reasoning, non-monotonic reasoning, vague and uncertain reasoning. The philosopher Imre Lakatos produced one such theory of how people with different reasoning styles collaborate to develop mathematical ideas. Lakatos argued that mathematics is a quasi-empirical, flexible, fallible, human endeavour, involving negotiations, mistakes, vague concept definitions and disagreements, and he outlined a heuristic approach towards the subject. In this chapter we apply these heuristics to the AI domains of evolving requirement specifi- cations, planning and constraint satisfaction problems. In drawing analogies between Lakatos’s theory and these three domains we identify areas of work which correspond to each heuristic, and suggest extensions and further ways in which Lakatos’s philoso- phy can inform AI problem solving. Thus, we show how we might begin to produce a philosophically-inspired AI theory of combined reasoning.}, keywords = {}, pubstate = {published}, tppubtype = {inbook} } One current direction in AI research is to focus on combining different reasoning styles such as deduction, induction, abduction, analogical reasoning, non-monotonic reasoning, vague and uncertain reasoning. The philosopher Imre Lakatos produced one such theory of how people with different reasoning styles collaborate to develop mathematical ideas. Lakatos argued that mathematics is a quasi-empirical, flexible, fallible, human endeavour, involving negotiations, mistakes, vague concept definitions and disagreements, and he outlined a heuristic approach towards the subject. In this chapter we apply these heuristics to the AI domains of evolving requirement specifi- cations, planning and constraint satisfaction problems. In drawing analogies between Lakatos’s theory and these three domains we identify areas of work which correspond to each heuristic, and suggest extensions and further ways in which Lakatos’s philoso- phy can inform AI problem solving. Thus, we show how we might begin to produce a philosophically-inspired AI theory of combined reasoning. |
Llano, Maria Teresa; Ireland, Andrew; Pease, Alison; Colton, Simon; Charnley, John Using Automated Theory Formation to Discover Invariants of Event-B Models Inproceedings In: In Proceedings of the Rodin User and Developer Workshop, 2010. @inproceedings{llano2011using, title = {Using Automated Theory Formation to Discover Invariants of Event-B Models}, author = { Maria Teresa Llano and Andrew Ireland and Alison Pease and Simon Colton and John Charnley}, url = {http://ccg.doc.gold.ac.uk/wp-content/uploads/2016/08/llano_rodin10.pdf}, year = {2010}, date = {2010-01-01}, booktitle = {In Proceedings of the Rodin User and Developer Workshop}, abstract = {Formal methods have been successfully used for the development of safety-critical systems; however, the need for skilled knowledge when writing formal models and reasoning about them represents a major barrier in the adoption of formal methodologies for the development of non-critical applications. A key aspect in the verification of formal models and in the development of reliable systems is the identification of invariants. However, finding correct and meaningful invariants for a model represents a significant challenge. We have used automated theory formation (ATF) techniques to automatically discover invariants of Event-B models. In particular, we use Colton’s HR system [2] to explore the domain of Event-B models and suggest potential invariants.}, keywords = {}, pubstate = {published}, tppubtype = {inproceedings} } Formal methods have been successfully used for the development of safety-critical systems; however, the need for skilled knowledge when writing formal models and reasoning about them represents a major barrier in the adoption of formal methodologies for the development of non-critical applications. A key aspect in the verification of formal models and in the development of reliable systems is the identification of invariants. However, finding correct and meaningful invariants for a model represents a significant challenge. We have used automated theory formation (ATF) techniques to automatically discover invariants of Event-B models. In particular, we use Colton’s HR system [2] to explore the domain of Event-B models and suggest potential invariants. |