Translational Research at the Intersection of AI, Immersive Technology and Creative Practice
I am an interdisciplinary researcher, strategic leader and hands-on creator, working across academia and industry.
Research & Innovation
I am a Professor in Digital Creativity, at University of York's School of Arts and Creative Technologies, as well as hold an appointment as R&D Lead, AI and Immersive at dock10, the UK's leading TV facility operator. My research blends elements from AI, Data Science, Human Computer Interaction (HCI), Design and Storytelling to create new interactive experiences with application areas ranging from games and immersive live performances to science education and informal learning. I have been central contribute to various large research initiatives, including being Principle Investigator and R&D Director for Weavr.tv, a £7.2m flagship project of the UK Government's Industrial Strategy Challenge Fund for the Creative Industries. I am committed to impact-driven research. I have worked closely with a range of organisations, from broadcasters, to game publishers and museums, creating new experiences that have been used by over 40 million people world-wide.
I do not publish large quantities, but I pride myself on the quality of my work. My award-winning research has been published across leading academic venues, including ACM CHI, ACM UIST, IEEE Transaction on Visualization and Computer Graphics, as well as IEEE Transactions on Games. Please see my Google Scholar profile for a full list of publications.
Find me on
Selected Awards
BEST PAPER
ACM CHI '15
BEST DESIGN
PAPER
CSCL '15
HONORABLE
MENTION
ACM TVX '19
BEST PAPER
ACM IMX'20
Showcase
Central to my research is the making of new technology and experiences. Over the course of my career as a researcher, I have created new input devices, museum exhibits, apps, VR experiences and games. I am committed to go beyond just prototypes, and have produced research outputs that are robust enough to be applied publicly. My research outputs have been used by over 75m people.
Weavr
How will audiences of the future experience live sports entertainment? Weavr addresses the challenge set by the UK’s Industrial Strategy Challenge Fund, being one of four flagship projects seeking to explore the use of immersive technology to develop new audience experiences. Weavr (30 months project, concluded in June 2021), was led by ESL - the world’s largest esports company - and the University of York, alongside leading companies from the UK creative industries. I was the Primary Investigator and R&D director, conceiving the project’s vision, and steering the innovation structure across a 45 people strong team and 6 leading organisations.
Esports are video games that are played competitively by professional players and watched by over 580 million people each year. Esports are broadcast much like traditional sports, involving pre-game panels to introduce the teams, the actual match which blends “virtual camera” footage of the digital game environments with audio commentary, and in post-match segments pundits break down and analyse the match. However, compared to traditional sports, gameplay in digital games is often fast, complex and involves multiple simultaneous points of actions that cannot be easily captured all at once. At the same time, esports collect detailed tracking data about gameplay, capturing every action that takes place in the virtual arena, such as the way players move to the arena, which actions they take and what evens occur. Weavr leverages the rich data available in esports in combination with real-time analytics to break down the complexity of esports, create entirely new narratives and enable novel data-driven experiences for the esports audience.
Weavr Experience Engine
My research team developed a novel experience engine, using data analytics, machine learning and artificial intelligence to analyse thousands of historic esports matches, and use this knowledge to identify key highlights and extraordinary performances in an ongoing live match. The experience engine then translates these data points into novel experiences, stories and insights for audiences. The examples to the right and below illustrate the suite of experience powered by the experience engine, produced by Weavr’s consortium partners.
The Weavr DOTA 2 Companion is a mobile companion app produced by Magnopus UK (formerly Rewind) that gives viewers in-depth statistics and personalised narratives. The mobile app, available on both iOS and Android displays the full narrative space generated by the experience engine in a highly interactive visual format. A live map shows movement of all players across the virtual arena, as well as a story feed that adopts to each users favourite team and player. Additionally, WEAVR Dota 2 Companion allows viewers to perform custom head-to-head comparisons between any two players, generating their own stories that can be shared through the built-in social media integration. Weavr Companion and the Twitch extension were together used by over 212,000 unique users over the course of three major esports tournaments in 2020.
Weavr VR developed by Future Visual creates a virtual viewing space that lets fans from all over the world get together, watch games, analyse strategy and playfully interact with each other. The map in the centre of the virtual space features all of Echo NE’s narratives features. Weavr VR was used by a leading US influencer alongside some of his fans, and the interaction in the virtual environment was streamed via Twitch.tv – reaching over 222,000 viewers.
Weavr’s Virtual Studio System developed by Dock 10 combines cutting edge virtual studio technology with Echo NE’s powerful data-driven narrative engine. This unique new combination enables commentators to be immersed in the virtual world, and weave data-driven stories with their tactical analysis.
Life on Earth
How closely are YOU related to a banana? As a Research Fellow at Harvard, I have designed, implemented and helped evaluate the Deep Tree Exhibit, an interactive visualisation of how more than 90,000 species are related over 3.5 billion years of evolutionary history. The project leveraged large touch displays to create an engaging, hands-on experience for museum visitors, who can fly through the Tree of Life, discover evolutionary traits, and explore how all life on Earth is related. The DeepTree and surrounding mini-games have been on display in 15 museums, 6 countries in German, French, Spanish and English, reaching millions of people. I have also made a version of the DeepTree and associated mini-game for the web, which is hosted by PBS Nova’s Evolution Labs (America’s most trusted source of popular science), where it has attracted over 2 million returning visitors.
PhD Research
My PhD research revolved around creating novel input device and studying how they could improve expressiveness, efficiency and enjoyment during everyday digital tasks. This included a novel Touch-Display Keyboard (a keyboard with dynamic displays and capacitive touch-sensors) and Voodoo Sketch, a pen-based system that allows users to draw ad-hoc interfaces and map them to graphical user interface components.
Touch Display Keyboards
The TOUCH-DISPLAY KEYBOARD (TDK) retains the physical properties of a conventional keyboard, but adds dynamnic graphical output and touch-sensing. It seamlessly extends the primary display, enabling the efficient and dynamic bindings of commands to hotkeys. It aims to improve the learning of keyboard hotkeys, maximize the utility of the keyboard and improve efficiency and productivity of users.
Voodoo Sketch
VOODOO SKETCH is an experimental interface paradign that allows users to dynamically compose and bind physical and paper-based controls that seamlessly integrate with the graphical user interface on large touch tables. This explorative research studied how physical user interface components can be created ad-hoc to be tailored to a specific tasks, while improving efficiency of input and creating opportunities for collaboration
Consulting
I am a Co-Founder and Managing Director of FAST/FORWARD, an innovation consultancy specialising on the space where technology meets creative practice. We help our clients envision ground-breaking entertainment and interactive experiences and architect cutting edge digital platforms and systems for their delivery.
