A unique event took place yesterday at The Metropolitan Museum of Art in New York City. Museum curators, engineers, designers, and researchers gathered in The Met’s iconic Great Hall to explore and share new visions about how artificial intelligence (AI) might drive stronger connection between people and art. A highlight from Monday’s festivities was the “reveal” of a series of artificial intelligence prototypes and design concepts, developed in collaboration across three institutions: The Met, Microsoft, and MIT.

Birth of a collaboration

For MIT, the collaboration began when Loic Tallon, The Met’s chief digital officer, visited the MIT campus to deliver an MIT Open Learning xTalk on the role of open access in empowering audiences and learners to experience art worldwide. Tallon views the collaboration as part of The Met’s initiative to drive global access to the museum’s collection through digital media: “We’re continuing to think differently about how a museum works, in this case how we leverage powerful technologies such as artificial intelligence. This collaboration among The Met, with our collection expertise, MIT with all these creative technologists and their incredible thinking about meeting tough challenges, and Microsoft with its AI platform has incredible synergy.”

MIT Open Learning and the MIT Knowledge Futures Group , two Institute organizations focused on the power of open data to create new knowledge, thus began a collaboration with The Met and Microsoft to spark global connections to art through AI.

The hackathon

On Dec. 12 and 13, the three collaborators came together to develop scalable new ways to engage the world through art and artificial intelligence. Curators from The Met joined MIT students and researchers, as well as expert technologists from Microsoft for a hackathon at Microsoft’s New England Research and Development Center. The ongoing projects from the hackathon, which were “revealed” Monday night, are:

• Artwork of the Day - Using Microsoft AI to analyze open data sets, including location, weather, news, and historical data, it finds and delivers artwork from The Met collection that will resonate with users.

• Tag, That’s It - Using crowdsourcing to fine-tuning subject keyword results generated by an AI model by adding keywords from The Met’s archive into Wikidata and using Microsoft AI to generate more accurate keywords, Tag, That’s It enriches The Met collection with the global Wiki community.

• Storyteller - Built with the help of MIT faculty participants Azra Akšamija and Lara Baladi, Storyteller uses Microsoft voice recognition AI to choose artworks in The Met collection that illustrate any story or any conversation.

• My Life, My Met - Using Microsoft AI to analyze posts from Instagram, My Life, My Met substitutes one's images with the closest matching Open Access artworks from The Met collection, enabling individuals to bring art into their everyday interactions.

• Gen Studio - Empowered by Microsoft AI, Gen Studio allows anyone to visually and creatively navigate the shared features and dimensions underlying The Met’s Open Access collection. Within the Gen Studio is a tapestry of experiences based on sophisticated generative adversarial networks (GANs) which invite users to explore, search, and be immersed within the latent space underlying The Met’s encyclopedic collection. It’s being built with the help of MIT visiting artist Matthew Ritchie, the Dasha Zhukova Distinguished Visiting Artist at the MIT Center for Art, Science and Technology, and Sarah Schwettmann of the MIT Knowledge Futures Group and graduate student in Brain and Cognitive Sciences.

The Met, as part of its Open Access program (which celebrated its second anniversary on Monday), has just released a newly developed “Subject Keywords” dataset of its collection. As Tallon explains, “We want to remove this idea that there’s only one way to engage with our collection. There are so many different ways of experiencing art, and many of those ways are being explored through the hackathon and beyond.”

Reasons to collaborate: synergies among art, AI, and developers

SJ Klein of MIT’s Knowledge Futures Group views the collaboration as building “a beautiful mosaic of a solution” that blends technology and people. “We're exploring how people can find new meaning and develop understanding of the world through large-scale collaborations with these increasingly iterative cycles of people and interpreting machines and networks all trying to make sense of the space,” he says.

For Ryan Gaspar, director of strategic partnerships at Microsoft’s Brand Studio, working with MIT and The Met means combining art, storytelling, and technology to create something unique. “The richness of the art and stories helps inform the technologists here from MIT and Microsoft. And then building on top of that our AI capabilities. We're already seeing some interesting concepts and ideas that neither MIT, Microsoft, nor The Met would have ever come up with on our own.”

A case study in AI for impact

Klein adds that the role of AI is to elevate the existing openness of The Met’s collection, promoting deeper audience engagement: “In terms of making the museum a platform for connection, open access alone isn’t enough. There's an entire discipline we're figuring out regarding what tools might support access and engagement. We’re building some of those tools now.”

Microsoft sees its role as empowering developers with AI tools and showing how AI can bring positive impact to the world. “We take a very optimistic view around how AI can actually drive empathy, foster connections and productivity, as well as support progress for society, humanity, and business. This collaboration is important for us to show the power and tangibility of what AI can do,” explains Gaspar.

The experience for the MIT community

The hackathon and its projects brought together students and faculty from across the Institute ranging from brain and cognitive sciences, the Media Lab, humanities, arts, and social sciences, engineering, computer science, and more. Such interdisciplinary exchange and hands-on collaboration, enabled by open access to data, knowledge, and tools, is at the root of MIT Open Learning’s approach to transforming teaching and learning.

For MIT students accustomed to tackling tough technical problems, the focus on problem-solving in the arts was a major plus. “It’s been fun working in the arts space, and thinking about cultural impact of the technology being built,” said MIT first-year undergraduate Isaac Lau. What MIT graduate student Sarah Schwettmann took away most was “the enjoyment of collaborating with The Met’s best curators and the top experts from Microsoft in finding innovative ways to engage people around art.”

Noelle LaCharite, who leads developer experience for cognitive services and AI at Microsoft, took the long-view about what MIT students learned: “These hackers are building important skills around identifying their own strengths and tapping into the strengths of others. They’re learning not to wait for permission, to take initiative, advocate for a vision, push it forward, and ask for help when needed. Those are classic life and work skills.”

Open development will continue on the tools built during the hackathon. As Sanjay Sarma , MIT vice president for open learning, explains: “MIT supports The Met’s commitment to open access, paired with the power of Microsoft AI, in order to empower people globally to create new knowledge and ways of experiencing art and culture that are so vital to our humanity.” Monday night’s event at The Met was a celebration of that openness and collaborative spirit.