The emergence of Covid-19 in 2019 sparked a pandemic that has posed a significant threat to human health, but it has also led to an acceleration in digital transformation across industries. In the face of the ever-changing nature of the pandemic, the ability to bring together the virtual and real worlds with technological tools has been vital to maintaining the operation of society.

More real than reality

The strict lockdown of Taiwan’s border gave the nation’s people a chance to enjoy a long stretch of “Covid zero” life, but in May 2021 the disease suddenly broke through and the national alert level was raised to three, leading to many activities being suspended and long-term preparations coming to naught. The loss to civil society was substantial.

At this time, Tsai Tsun-hung, an assistant professor of communications design at Shih Chien University with a background in technology, put a backup plan he had already developed into action, bringing students into the free online virtual reality (VR) platform VRChat to hold an online graduation exhibition entitled “­Conation.” As past years’ exhibitions had been physical and no off-campus ones had been held, from the start Tsai made participation in the exhibition optional. However, as Taiwan’s Covid outbreak in mid-2021 led to that year’s physical event being called off, the number of students who chose to take part online began to grow, increasing from about two dozen to more than 40. In addition to hurrying to translate their works into digital form in response to the situation, the students also joined in creating the virtual exhibition halls.

The all-day, free exhibition halls were linked together like a maze, offering enough room for every piece to be accommodated. Guests who logged in could wander around and chat freely, write in the air with virtual pens, and even interact with works with no fear of damaging them. It was an experience unlike any past exhibition and a novel one for both the students and the public.

In response to the pandemic, Professor Tsai Tsun-hung of Shih Chien University used Internet resources to stage an online graduation exhibition for his students, entitled “Conation.”

A blueprint for future music venues

“Isn’t this whole ‘metaverse’ idea that everyone’s talking about now basically like the MMORPGs we used to play?” Tsai jokes. Of course, things change with time, and with today’s higher bandwidth and better resolution, along with the real-time trans­mission of virtual images, digital experiences are now more realistic than ever.

Although things like performing together remotely and attending virtual performances have not yet become particularly widespread, online concerts, born of the need to adapt to the pandemic, have become popular and have changed the ecosystem of the pop music industry.

You may wonder, what makes online concerts different from “offline” ones? Once the pandemic eases, will these online concerts vanish back into the ether? B’In Live, which has a wealth of experience in event management, is well aware of the major differences between the two. “An offline concert is a social event—being in a space with other people, the feeling of the sound, the smell, the atmosphere, it all comes together to create a shared experience of pleasure. This kind of thing can’t easily be replicated online,” says B’In Live producer Lola Wu straightforwardly. “Online concerts involve a whole different way of thinking, because they’re free of the limitations of space and geo­graphy,” adds Gas Tan, B’In Live’s technical director.

The most striking example is from the end of last year, when there were very few Covid cases in Taiwan. Even though New Year’s Eve activities went ahead as usual, the band Mayday still held both online and offline concerts. To reinforce the difference between the two, the production team used frequent transitions and gorgeous camera shots, coupled with a strong spatial atmosphere, to highlight how online concerts are not limited by physical venues, hoping to ensure that both online and offline audiences were satisfied.

Unconstrained by time and space, the virtual exhibition halls of “Conation” were open 24 hours a day, and were able to accommodate and screen all participating works at the same time. (courtesy of Tsai Tsun-hung)

5G: High-speed mobile Internet

In 2020, Chunghwa Telecom acquired Taiwan’s first fifth-generation (5G) mobile telecommunications license at the high price of NT$45.6 billion, herald­ing the arrival of the 5G era. Taiwan has had 4G for a ­decade, and the bandwidth has been sufficient to meet the daily needs of the public, from sending text messages to streaming videos. What are the key changes that 5G brings?

“5G’s biggest value comes when you need a high-speed connection on the move,” explains Tsai Tsun-hung succinctly. At present, Wi-Fi 6 can provide stable, high-speed, low-latency multi-device wireless network environments for indoor fixed locations. “With Wi-Fi indoors and 5G outdoors, the two complement one another,” he adds.

Facing an era of segmented audiences, the team at B’In Live has begun actively exploring new possibilities. The photo shows technical director Gas Tan (left) and creative director Lola Wu (right).

The robot Hanji can calculate physical distances and move around autonomously to serve as an exhibition guide. (courtesy of ET@T, © Taipei Fine Arts Museum)

Virtual influencers in the real world

So what are the applications of 5G for art and culture? The Digital Art Foundation, which is dedi­cated to developing and promoting digital art, is attempting to answer this question. Tsai Tsun-hung, who is also technical director for the DFA, gives an example. In 2018, the foundation invested in the development of a livestreaming panoramic-VR robot guide called Hanji. Built on a combination of AI and VR, Hanji uses ultra-wideband positioning technology to calculate the direction and distance of tour parti­cipants and follow them by itself. It can also stream panoramic images live and interact with online audiences at the same time.

Hanji was born in the 4G era but was upgraded to version 2.0 with the introduction of 5G. Building on its original foundation, Hanji 2.0 incorporates motion capture and simultaneous localization and mapping (SLAM), evolving into an “AR streaming robot.” “This requires rapid, high-volume data throughput, and this is where 5G shines,” says Tsai. This means that performers can use motion capture to operate Hanji, playing a role in virtual space and moving the robot around remotely to interact with others in a performing space, and even broadcasting the resulting performance through livestreaming. In other words, through a system like this, even virtual influencers like ­VTubers can go “on location.”

The immersive backdrops and technological props of Telling Tent’s performances have proven especially popular with families with children. (courtesy of Telling Tent)

Actors perform remotely wearing motion capture gear. (courtesy of Telling Tent)

 

Virtual interaction

Telling Tent, established in 2016, is another bene­ficiary of the advent of the 5G era. Formed by a group of digital artists specializing in “motion-sensing ­theater” for children, Telling Tent travels Taiwan with its dome tent and uses a panoramic screen, original animations, and technological props to create special effects like smoke, flashes, and vibrations while telling engaging stories derived from traditional folklore.

Daniel Chang, head of Telling Tent, explains that the group does two types of shows: the first is just anima­tion, while the second is built around a combina­tion of animation and real actors. Dressed in their costumes, the actors perform live in sync with the projected animations. They can also use motion capture equipment to create virtual characters, combining both real and virtual in their various ­programs.

“5G is a real boon for motion capture,” says Chang. Because Telling Tent is a touring troupe rather than performing at a fixed venue, using a private 5G network enables the audio and video of the performances to be transmitted back to the remote actors in real time. Virtual characters played by actors can thus interact with the audience based on the situation at the scene, “and so the audience knows that what they’re seeing isn’t prerecorded.” Not only that, the virtual characters integrated into the animation can even achieve effects that would be almost impossible to do in an in-­person performance. “For example, they can fly through the air or have their eyes turn into hearts when they smile,” adding a little more drama to the proceedings.

As it tours the country with its performance tent, Telling Tent has found 5G an invaluable tool. (courtesy of Telling Tent)

Telling Tent is a troupe composed of digital artists.

A new type of performance

The immersive theatrical show Lunatic Town, put on by the National Theater and Concert Hall (NCTH), is representative of how Taiwan’s art world has embraced technology in recent years. This show can be broken down into three performance elements, ­presented from multiple perspectives: the livestream, the physical performance, and the virtual world. Audi­ence members can experience the immersive performance in the outdoor plaza between the two halls and even join in the interactions during the show. At the same time, they need to also be watching the livestream to be able to piece together the whole story from its various fragments presented in these different dimensions. 

The NTCH invited B’In Live, with their rich experience in commercial performances, to help coordinate and execute the show. Lola Wu emphasizes that the show’s complex, multidimensional structure justifies the use of 5G technology to make livestreaming and remote performance feasible. “The application of such technologies needs to be reasonable and impactful,” she says. Only if performances are more than just demonstrations of technological capability can digital art progress into the future.

As the 5G era has only just begun, there is still a long way to go before any mature business models emerge. As B’In Live’s technical director Gas Tan says, “We [the production team] need to consider the characteristics and goals of the show rather than just reaching for cutting-edge technologies.” Furthermore, he adds, creating a dialogue between two such seemingly unrelated fields simply requires some crucial link.

With Taiwan enjoying a reputation for science and technology, it can thus provide a dynamic creative environ­ment for art workers to explore the inter­section of art and technology and the fireworks that can create. For example, renowned dancer Huang Yi was able, after much lobbying, to get the nod from the Taiwanese manufacturer of the robotic arm Kuka to have the six-axis arm dance with him on stage, making a wild flight of the imagination into reality. There are low-key companies with outstanding technologies and potential all across Taiwan. How might they help each other in the future as the boundaries between technology and art disappear? With high-speed Internet a driving force behind this evolution, the results will surely be worth waiting for. 

Lunatic Town, organized by the National Theater and Concert Hall, tells a story set across parallel universes, exploring new possibilities in performing arts through a variety of technologies. (courtesy of NTCH)

Unconstrained by time and space, the virtual exhibition halls of “Conation” were open 24 hours a day, and were able to accommodate and screen all participating works at the same time. (courtesy of Tsai Tsun-hung)