You, Me, and DALL-E: On the Relationship Between Architecture, Data, and Artificial Intelligence (2023)

Are pictures worth a thousand words?

David Holz was not motivated by money. Bowing out of the venture-backed tech startup he had built over twelve years, the self-described “serial entrepreneur” started a new company of just ten people. “It’s just about having a home for the next ten years to work on cool projects that matter,” Holz told The Verge in an August 2022 interview. “Hopefully not just to me but for the world — and to have fun.”

You likely haven’t heard of David Holz. However, you may have heard of his company and the digital tool that shares its name. Midjourney caused an earthquake among artists and designers when it launched in July 2022, triggering a tsunami of vibrant, whimsical digital artwork shared across social media sites. Holz’s tool, maintained by ten colleagues, is currently used by over 14 million users generating everything from surreal futuristic worlds to the pope donning an oversized stylish coat. If Holz’s aim was to allow the world to have fun, it's 'mission accomplished.'

AI tools such as Midjourney can help us to unlock things we cannot see. — Matias del Campo

Midjourney is an example of a generative text-to-image tool, which uses artificial intelligence to generate images responding to text prompts written by users. No drawing, no coloring, no illustrative input from humans whatsoever— just words. A marketplace that few would have imagined only two years ago, Holz’s Midjourney now competes with rival tools such as DALL-E by OpenAI, Imagen by Google, and Stable Diffusion by Stability AI in allowing anybody with an internet connection to create detailed images on whatever topic comes to mind in a matter of seconds. Despite the frenzy caused by the launch of such tools throughout 2022, the photorealistic quality of their output has continued to improve in the months since. The latest version of Midjourney, Version 5, has all but ditched the whimsical aura and distorted figures of previous versions and can now generate images indistinguishable from reality.

Text-to-image tools were only one-half of the forces that thrust AI into the public discourse in 2023. The other half came courtesy of OpenAI, the creator of Midjourney competitor DALL-E, who launched the AI-powered chatbot ChatGPT in November 2022. Where generative text-to-image tools have been trained using a dataset of millions of images, ChatGPT’s AI model was trained on millions of text documents and webpages written by humans. As a result, ChatGPT can respond to text inputs from users with coherent answers in seconds, from questions on history to advice on dinner recipes. By January 2023, ChatGPT was reported to have reached 100 million monthly active users, making it the fastest-growing consumer application in history, as users flocked to see first-hand the bot’s surreal generation of articles, jokes, essays, business ideas, and more.

While the labels ChatGPT and GPT-4 are sometimes used interchangeably in the discourse surrounding AI, there is an important distinction to be made.

Before addressing the implications of AI-based applications such as ChatGPT and Midjourney on architecture and design, it is worth reflecting on the underlying technology which drives them. ChatGPT is powered by a large language model (LLM) called GPT-4; a deep machine learning algorithm that can recognize, summarize, translate, predict, and generate text and other content such as software code, based on knowledge derived from massive datasets. While the labels ChatGPT and GPT-4 are somtimes used interchangeably in the discourse surrounding AI, there is an important distinction to be made. ChatGPT is an application; an online tool. GPT-4 is the brain behind such applications. ChatGPT relies exclusively on GPT-4 to perform its tasks. However, a machine learning model such as GPT-4 is not exclusive to ChatGPT and can be deployed for uses far beyond the chatbot.

As Zapier succinctly described in March 2023: “If we think of ChatGPT as a Dell computer, then GPT is the Intel processor that powers it. After all, different computers can run on Intel processors in much the same way other AI applications can run on GPT-3 or GPT-4.” Incidentally, OpenAI’s text-to-image tool DALL-E used a version of GPT-4’s predecessor GPT-3 when it was first unveiled in January 2021.

Seeing the forest for the trees

For a closer analogy of the potential applications of machine learning models beyond headline grabbers such as ChatGPT and DALL-E, we can look to nature; specifically to forests, trees, branches, and offshoots. Machine learning models such as GPT-4 sit at the base of an extensive tree with many branches and offshoots. Despite its outsized media attention, ChatGPT is but one branch of the GPT-4 tree. As noted above, the text-to-image application DALL-E represents another branch, having first emerged from an ‘ancestral’ GPT-3 tree in 2021. Other applications, or branches, of the GPT-4 tree can create functional websites from rough sketches, code online games from scratch, or in the case of the mobile app Be My Eyes, analyze and describe real-world objects for visually-impaired people to the “same level of context and understanding as a human volunteer,” according to OpenAI.

From within the ChatGPT and DALL-E branches, families of useful, lucrative offshoots can emerge depending on how a user exploits each application. Offshoots from the ChatGPT branch have so far included media companies using ChatGPT to prepare for interviews, simplify subjects matters, and generate ideas for popular content, as well as healthcare companies using ChatGPT to flag potential drug interactions, suggest treatment options for specific conditions, or provide relevant clinical guidelines. Meanwhile, architecture-specific offshoots from the DALL-E branch can include generating precedent images for design concepts, exploring artistic styles, and performing realistic edits to existing images using text instructions. The GPT-4 tree does not depend on any one of these branches or offshoots to survive, but each branch, whether the ChatGPT branch, the DALL-E branch, or the Be My Eyes branch, and all their dependent offshoots, rely on the GPT-4 tree for life.

Even designers with years of engagement in the field, let alone the majority of architects who have little or none, are battling to simply see the forest for the trees.

The GPT-4 tree is also one tree in a wider ‘species’ of LLM trees, each of which uses a rival LLM to support its branches. Stable Diffusion creators Stability AI, for example, recently developed and publicized its own LLM named StableLM. By releasing their LLM open-source, the StableLM tree will soon bloom with branches and offshoots grown by third-party programmers and companies eager to augment and adapt the model’s text and code generation capabilities. Zooming out even further, these various LLM trees belong to just one species of tree in the AI forest, with other species supported by alternative machine learning algorithms differentiated by their methods of learning and the extent of human supervision.

Architects and designers can engage with this forestry ecosystem across several scales. Like highly-specialized botanists, some may find themselves occupied fully with one specific branch of a particular tree, as have the many designers today who are exclusively exploiting Midjourney as a design companion. Others will see these branches as low-hanging fruit and instead become dendrologists of a whole GPT-4 or StableLM tree, seeking to find new ways of utilizing the machine learning model’s billions of variable parameters beyond more common image and text generation uses. Other architectural naturalists may go even further and, like modern-day Darwins, embark on a search for new orchards beyond our prevailing theories on the relationship between humans and artificial intelligence. With the accelerated pace of AI advances through the early part of the 2020s, even designers with years of engagement in the field, let alone the majority of architects who have little or none, are battling to simply see the forest for the trees.

Artificial intelligence and the architect

“There is a whole plethora of areas that the architecture discipline can benefit from by using machine learning and artificial intelligence,” architect and educator Dr. Matias del Campo told me in a recent conversation about text-to-image tools. Del Campo is an associate professor of architecture at the University of Michigan’s Taubman College of Architecture and Urban Planning and director of the school’s Architecture and Artificial Intelligence Laboratory. Del Campo's lab is an interdisciplinary group encompassing architecture, robotics, computer science, and data science, whose mission is to uncover ideas, concepts, and technologies with respect to using artificial intelligence in architectural design. Beyond his academic commitments, del Campo is also a director of the architecture practice SPAN alongside Sandra Manninger; a firm which in del Campo’s words “oscillates between speculating about possibilities regarding AI and architectural design, and the implementation of those possibilities in the form of projects, books, articles, papers, lectures, and exhibitions.”

Architects are wonderfully outfitted to create interesting results from Midjourney.— Matias del Campo

Del Campo belongs to a community of architectural figures whose interests in computation and artificial intelligence began over twenty years ago. Today, he is joined by a new cohort of designers who, like millions of others beyond the architecture profession, were introduced to the power of AI tools through chatbots such as ChatGPT and generative image tools such as Midjourney. “Architects are wonderfully outfitted to create interesting results from Midjourney,” del Campo told me. “Through our architectural education, we have been introduced to a large range of topics, from art and painting styles to literature, photography, and fashion. We have a unique ability to combine these varying subject matters into a comprehensive sentence that can be wonderfully visualized by artificial intelligence. All of those bits and pieces of disparate knowledge came together to create this AI explosion that was seen in the architecture discipline in 2022.”

Much of my conversation with del Campo revolved around text-to-image tools such as Midjourney, but as the architect explains, the applications of artificial intelligence in architecture go much further. To return to our earlier analogy, del Campo sees promise in stepping back from the Midjourney branch to appreciate the broader algorithmic tree. “This explosion of tools in the architecture discipline might not be something that everybody adopts forever, but it has introduced a whole new generation of architects to the world of AI,” del Campo told me. “We can now start to ask wider questions such as: How can we use AI to elevate the living standards of millions of people? How can we reduce the consumption of materials using machine learning algorithms? How can we use AI to generate structures for buildings that are more efficient and sustainable than those of today? The more people working on these questions, the more likely we are to succeed.”

“Prediction is very difficult, especially if it’s about the future,” said the physicist and Nobel laureate Niels Bohr. Nevertheless, the question of how artificial intelligence will impact the architectural profession of the near future is one which Archinect’s editorial has wrestled with before and will continue to, amid the ever-expanding forest of AI tools, applications, and models. “A year is a long time in computer science terms, and predictions are always difficult,” del Campo explained. “AI will very soon have a big impact on mundane processes such as cost calculation optimization, checking plans for codes, ensuring plans are drawn correctly, and perhaps negotiating with another AI that is operated by local planning authorities you need to submit plans to. So in the best case, it will hopefully free up time for architects to design.”

I believe [AI] is going to play the key role of augmenting the possibilities within our minds as humans.— Matias del Campo

“In the design process, too, AI will play a role,” del Campo continued. “But we shouldn’t make the mistake of thinking about these machines in the same way that we think about humans. AI tools such as Midjourney can help us to unlock things we cannot see. It can provoke us towards architectural solutions we didn’t previously think about. But it is fundamentally still an algorithm based on an enormous database, that is able to go through this database at an enormous speed. I believe it is going to play the key role of augmenting the possibilities within our minds as humans. But it is not about being replaced by AI. I don’t think that will happen any time soon.”

As my conversation with del Campo concluded, I asked if he, like many others, was surprised by the sudden acceleration in machine learning applications such as Midjourney and DALL-E in the early 2020s. “Sandra Manninger and I began conversations with computer scientists on AI in the late 1990s,” he told me. “When this started, they were able to simulate interactions between only two neurons, which is nothing. Even in the early 2010s, the algorithms and computing power we see today didn’t exist. What we are experiencing right now, I wouldn’t even have expected one year ago. Such developments were perhaps not as unexpected in computer science circles, but in the architectural world, they certainly were.” Today, there is evidence that the ever-advancing capabilities of machine learning models such as GPT-4 are even outpacing the comprehension of those responsible for their creation. In a March 2023 conversation, OpenAI CEO Sam Altman remarked about the company’s latest model GPT-4: “Do we understand everything about why the model does one thing and not one other thing? Certainly not. Not always. But I would say we are pushing back, like, the fog of war more and more.”

Brave new world?

Through this fog of war, an AI arms race is underway between tech companies large and small, all of whom fear being left behind by their competitors. It’s a dangerous recipe from an embattled industry. The red-hot crypto-craze of the late 2010s has cooled. The decade-long supply of venture capital spurred by low-interest loans has been drowned by recent interest rate hikes, exemplified by the collapse of Silicon Valley Bank in March 2023. For many tech companies, foraging the vast, unregulated, unexplored AI forest for money trees is the only game in town.

Fearing the fallout of this arms race, over one thousand notable tech figures signed an open letter in March 2023 calling for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.” The authors cited concerns such as malicious actors “flooding information channels with propaganda and untruth,” and job-hunting “nonhuman minds that might eventually outnumber, outsmart, obsolete, and replace us.” The letter instead calls for a “level of planning and management” that is currently absent in a space occupied only by AI labs “locked in an out-of-control race to develop and deploy ever more powerful digital minds that no one — not even their creators — can understand, predict, or reliably control.”

Just as real-life forests are fueled by nutrients such as nitrogen and phosphorus, our AI forest is fueled by data.

Practicing architects are not direct participants in the AI arms race, but neither are they immune from its effects. Beyond wider fears of job replacement expressed in the open letter, the deployment of AI tools in the architectural design process brings with it a series of hazards. Just as real-life forests are fueled by nutrients such as nitrogen and phosphorus, our AI forest is fueled by data, farmed from the physical and digital worlds, from city streets to social media sites. In this arms race, where the speed of deployment has so far taken precedent over questions of ethics, bias, and safety, such data is inevitablycontaminated by the many biases embedded in the built environment, be it biases against women, people of color, the elderly, or those with varying degrees of physical abilities and motion. Are architects prepared to place an increased responsibility for the design and operation of the built environment on AI tools, at a time when the decision-making processes powering such tools, to the extent we even understand them, are closely guarded secrets?

In tandem with the question of data bias is the question of data collection, ownership, and liability. Throughout 2023, the creators of AI image tools including Midjourney and Stable Diffusion have been slammed with copyright suits from artists who claim their work was taken from the internet and used in AI data training sets without their consent. In a future scenario where a tech-savvy architecture graduate acquires a machine learning algorithm from a computer science friend, trains a machine learning model exclusively on the architectural drawings, models, and imagery of projects by Zaha Hadid Architects, and generates a complete architectural project to be sold to a client, who can claim license over the scheme? The graduate? Zaha Hadid Architects? The friend who wrote the algorithm? The machine learning model itself? If the resulting scheme exhibits defects derived from the original Zaha Hadid Architects projects, resulting in a fatal accident during construction, who is liable for prosecution? With AI-specific legislation currently being developed across Europe and the United States, such legal considerations will inevitably find their place in the architectural education and licensure curricula of the near future.

The AI forest is growing faster than humanity could ever hope to cut it down.

As architects, educators, legislators, and tech leaders ponder such questions and concerns, the AI forest continues to relentlessly sprout new trees, branches, and offshoots. In human literature, forests have served as the setting for fairytales and horror stories alike. Similarly, storytellers on how artificial intelligence will alter the future course of human civilization range from ecstatic digital utopians to fearful techno-luddites. Concerned open letters aside, the AI forest is growing faster than humanity could ever hope to cut it down. The challenge, therefore, moves to how we view the forest, how we condition the data soil that sustains it, and who, if anybody, we appoint to manage it.

Like its natural counterparts, the AI forest can be either a celebration of intricacy, cooperation, and humane responsibility, or a mismanaged well of exploitation, illegality, and greed.