Record, replay, and review: UI automation with Xcode

Hey, I’m Max, and I’m an engineer on the Xcode team. There are so many awesome features and experiences inside of Xcode, it can be overwhelming. For example, did you know it’s possible to watch your app run on dozens of devices, languages, and configurations with one click? And on top of that, you can get a full quality video recording of every single run. Well, that’s totally possible with the power of UI automation in Xcode. Let's take a look at how that works. First, we’ll cover an overview of UI automation. Then we’ll prepare your app, record your interactions as automation code, replay the automation across several devices and languages, and finally, watch video recordings of the results and see a report about whether each run passed or failed.

Let’s start with an overview of how UI automation works.

Inside Xcode, we have two testing frameworks: Swift Testing and XCTest. Both frameworks are capable of quickly testing your app and your source code in many configurations.

When you import XCTest, a framework called XCUIAutomation is automatically included. XCUIAutomation can be used to automate your app and interact with it like a person does. These frameworks work together to provide a complete app testing suite, and if you ask me, I think that’s pretty sweet.

A complete app testing suite is usually comprised of both unit and UI automation tests. Unit tests test your app’s logic and models, and with Swift testing, it’s possible to run tests on frameworks and Swift packages that don’t even have a user interface.

Meanwhile, UI automation tests validate your app’s user experience, as well as its integration with Apple hardware and the behavior of common workflows.

Generally, your test suite will have more unit tests than UI tests, and you want to aim for full coverage of your code. But UI automation tests let you see how your app looks, behaves, and integrates with the rest of the operating system. There are so many benefits to doing this.

For example, you can test your app as a person would with gestures like taps, swipes and clicks. You can understand how your app is perceived by people who use assistive technologies like VoiceOver, Voice Control, and Dynamic Type.

You can view your app running on all of the languages and regions that it supports and focus on languages that have large effects on the look of your app, like languages with longer strings and languages with a right-to-left layout. You can test your app’s integration with Apple hardware features, like the Action button, camera button, Apple TV remote, and Apple Watch’s Digital Crown. And finally, you can test your app’s launch performance, which is a key metric in understanding how quickly people can get started using your app. To set up a UI automation workflow, there are three key phases: record, replay, and review.

First, record your interactions like taps, swipes, and hardware button presses, then watch Xcode automatically write those as code. Then, replay your interactions across multiple devices, languages, regions, and device orientations, both on your devices and in Xcode Cloud. Finally, review videos and results of your app running in all of those configurations and see which ones passed and which ones failed.

UI automation is supported on all Apple platforms: iOS, iPadOS, macOS, watchOS, tvOS, and visionOS (Designed for iPad). The same automation can even run on multiple platforms. So it’s possible to build an automation once and run it on all your supported devices. That means you can see how your app runs on Mac, iPhone and Vision Pro with one click and no code changes. Let’s briefly go over how UI automation works.

UI automation interacts with your app as a person does using gestures and hardware events. Automation runs completely independently from your app, so your app models and data are not directly accessible.

UI automation tells the operating system which gestures to perform, then synchronously waits for them to complete one at a time.

These actions include launching your app, interacting with buttons and navigation, setting system state like Dark Mode, and even setting a simulated location for the device if you wanted to. At Apple, accessibility is one of our core values. Apple’s assistive technologies make sure that everyone can use your app regardless of any physical, visual, audible or motor impairment. We work really hard to ensure that most of these technologies work with your app by default, with no work needed from you as a developer. That said, adding additional support can enrich your app’s experience and make it even easier to automate. The thing is, accessibility is the underlying framework that powers UI automation. Having a great accessibility experience means you get a great UI automation experience too.

Accessibility provides information directly to UI automation, such as element types, labels, values and frames but what accessibility sees is not necessarily one-to-one with what you see as a person.

Let’s take a look at an example. On this screen, the Great Barrier Reef button is visible in the UI but accessibility can see even more than that. Sure, accessibility can see the elements type and label and expose them to UI automation but the identifier property is also exposed. The accessibility identifier can be used to describe any element on screen uniquely, relative to all the elements around it. The identifier is not designed to be localized, so you can use it to refer to the same UI element in any language or device.

For checkboxes and other stateful elements, a value property can be used. This reveals the element’s current state to both accessibility and UI automation.

To learn way more about accessibility, check out “Build accessible apps with SwiftUI and UIKit” from WWDC23, or “SwiftUI Accessibility: Beyond the basics” from WWDC21.

Now that we’ve explored how a UI automation works with accessibility, let’s prepare your app automation.

First, we’ll add accessibility identifiers. Then we’ll do a quick review of your app’s accessibility. And finally, we’ll add a new UI testing target to get ready to record our interactions as code. Before we dive in, it's worth noting that your app already fully supports UI automation and UI recording out of the box with no work needed from you as a developer. The steps we’re about to cover are not required but they can lead to better and higher-quality results. Adding accessibility identifiers can be done in your view code written in SwiftUI, UIKit or AppKit. Accessibility identifiers are the best way to uniquely identify any element in your app for automation. It’s a good idea to add accessibility identifiers to elements with localized strings or dynamic content. That includes content found in any of your data models or content that’s downloaded from the internet.

Good identifiers are unique within your entire app, descriptive enough to detail the element that they are on, and static, not reacting to changes in content.

While titles and descriptions might change, good identifiers always describe the contents of the element they are attached to. That way, we can be sure my lovely landmarks stick around forever.

In SwiftUI, you can add the accessibilityIdentifier modifier to any UI element. It will be recognized as long as your view and its parent views are not hidden from accessibility.

It’s a good idea to make identifiers specific to an instance of a view, especially for views that are used many times in your app. In this example, we use the landmark’s id property to make the identifier unique for each one.

In UIKit, the accessibilityIdentifier property can be set on any UIView as long as the view is an accessibility element. Most UI views, like controls, text and images, are accessibility elements by default so usually no work is needed to do this.

Properties like accessibilityLabel, accessibilityTraits and accessibilityValue are useful to assistive technologies like VoiceOver and are also useful to UI automation.

However, the accessibilityIdentifier property is not read out loud by VoiceOver and is not exposed to anyone using your app. This makes it a useful way to provide information that is only useful to automation, like the index of a table cell or the symbol name of an image. I can even have the coding assistant in Xcode add accessibility identifiers for me. For example, I can write, “Add accessibility identifiers to the relevant parts of this view”, and it will just happen. The coding assistant even knows how to use the id property on a landmark to make each identifier completely unique. Pretty cool. Reviewing your app’s overall accessibility behavior is a good idea before starting UI recording. It’s like flossing before going to the dentist. You’re going to know exactly how well you’ve been doing soon. This will just give you a good preview of it. Xcode ships with an app called Accessibility Inspector, which lets you find, diagnose and fix accessibility issues. The Inspector can be launched from the Xcode top-level menu under Open Developer Tool. It can also be launched from Spotlight.

Accessibility Inspector can list the accessibility values for any view in your app on any platform. Simply select the platform you wish to inspect, then click the Element inspector and interact with the UI element you want to learn about on that platform. A list of element properties will be displayed. Some of them, like the ones in the Basic section, are really useful to UI automation. For views that are lacking information, you may want to change your app’s source code and add some. You can find out details about each property by clicking a property’s name. The popover will tell you exactly which property to set, provide documentation about it and more.

You can learn even more about accessibility features with our sample code project, “Delivering an exceptional accessibility experience.” This project provides some great code examples for an app that uses many accessibility features and is really friendly to UI automation. Also, check out the article, “Performing accessibility testing for your app” to learn about ways to check your app’s accessibility using a bunch of different assistive technologies. Once we’re ready to start automating, we need to add a new UI testing target to have somewhere to put our automation code.

In the project settings view in Xcode, add a new target with the “plus” button below the targets list.

Then select UI Testing Bundle from the popover.

Once you click Finish, a new UI test folder and template will be added to your project.

The template has some simple tests to help you get started. Okay, now we’re ready to magically record all of our interactions as Swift code. Let’s use iOS Simulator and Xcode for this. A few years ago, my mom and sister went on a month-long trip to Australia without me, and I’m pretty bummed I didn’t get to go with them. Lucky for me, there's an app for that. I can use the Landmarks sample project from this year’s WWDC to plan a vacation of my own. I’ll record some interactions that plan my trip so I can ensure that workflow doesn’t break in future versions of the app. When I open the UI test source file for the first time, a popover will appear telling me how I can start a UI recording. Now, I’ll start a UI recording using the button in the sidebar, and Xcode will automatically build and relaunch my app in Simulator.

Now that my app is launched, I'll go to the collections view.

As I interact with the app, the code representing my interactions is recorded in the source editor. I’ll tap the Plus button to add a new collection to start planning my own trip to Australia. Now I’ll tap the Edit button to rename the trip, and I can rename the trip “Max’s Australian Adventure”.

Xcode will keep my test up-to-date as I keep typing.

Now I'll edit the collection of landmarks.

Just add some Australian landmarks like the Great Barrier Reef and Uluru and tap the check mark.

Once I go back to the collections view, I can see that my collection has been added with some Australian landmarks included.

You can stop UI recording with the Stop Run button in Xcode. After you finish recording, there are a few things you might want to do to make sure you got the automation you wanted. First, review the recorded code. Then add validations using XCTest APIs to ensure your app is behaving as you expected it to. And finally, explore other automation APIs that can make your test even more powerful.

Let’s review the UI queries that were recorded and see if I want to make any adjustments.

Every line of recorded code will have multiple options to address each UI element, and which one you choose depends on your goals. You can click the dropdown on each line of source code to see the choices.

Quick hint, choosing the right one will help you board your plane to Australia faster.

We have a few recommendations to help you select from the options.

For views that have localized strings, like text elements or buttons, we recommend choosing an accessibility identifier if there is one. UI recording tries to use the identifier by default if one exists.

For views that are deeply nested, like text in scroll views, we recommend choosing the shortest possible query. This will help your automation stay resilient as your app changes.

Lastly, for dynamic content that is downloaded from the internet or content that changes frequently, like timestamps or the temperature, we recommend using a more generic query or an accessibility identifier if one is present.

In this example, we don’t even use an identifier or any string at all, and we always just refer to the first piece of text.

Okay, now it's time to make a selection. I’ll click the line of source code I want to edit to see the options. All of these queries uniquely identify the element you interacted with, so there's really no wrong choice. It’s just about choosing how you want to store a reference to this piece of UI for the future. I’ll select the textFields.firstMatch option to make sure the text field is always tapped in my test no matter what it’s called. Double click any of the dropdowns to store this result in your source code. Now let's quickly rerun my automation to see if it recorded my actions correctly. I'll click the test diamond to run it. While we might be testing your app, we aren't trying to test your patience. The automation replay runs really fast.

The collection gets quickly created with the correct name, the locations get added, and the automation passes. Awesome. That was way faster than a 19-hour flight. Now we can add validations to the code to check the expected behavior. In this example, I’ll validate that the Great Barrier landmark was added to my collection.

I can call methods like waitForExistence to have my automation wait for an element to appear before moving on. I can also call the more generic method, wait(for:toEqual:) to validate that any property on an XCUIElement matches the expected result.

I can pair both of these methods with XCTAssert statements from XCTest to fail the test if these methods return false.

Now, let me go back to my code and quickly add a waitForExistence on the name of my collection to make sure it’s always there in future runs.

Now is a good time to explore other automation APIs to make your code even more powerful.

It can be useful to use the setup instance method of an XCTestCase to make sure the device is in the same state in future runs. I can call APIs like orientation, appearance or even simulate a location to get my device into the correct state before a run starts.

Before launching my app, I can use properties like launchArguments and launchEnvironment to have my app use those parameters when the launch method is called.

If your app supports a custom URL scheme, you can open it to a matching URL directly using the XCUIApplication open method.

There’s even a global version, which opens a URL using the device’s default app for it.

Lastly, it’s possible to perform an accessibility audit of your app inside a UI test. There’s a great session on that called “Perform accessibility audits for your app” from WWDC23. Now that we’ve recorded our interactions and set up our automation, let’s configure the tests to replay in multiple configurations, both at-desk and in the cloud. It's really useful to add your test to a new or existing test plan. Test plans let you include or exclude individual tests, set system settings for where and how your tests run, and manage test properties like timeouts, repetitions, parallelization, execution order and more.

Test plans are also associated with a scheme, which lets you pair a test plan with specific build settings.

You can learn a lot more about this with the article, “Improving code assessment by organizing tests into test plans” in our developer documentation.

In your test plan, you can add or remove tests on the first screen or switch to the Configurations tab to make changes to how the test will run.

I can set up multiple configurations to run my app in multiple languages.

Typically, each locale exists as a separate configuration in your test plan.

You can have settings that are focused for a specific locale configuration or other settings that are shared across all of them.

It can be helpful to include configurations for languages with longer strings, like German, or right-to-left languages, like Arabic and Hebrew.

There are even UI automation-focused settings in the Configurations tab.

These include whether to capture a video or screenshots during the run and whether any media will be kept afterwards.

By default, videos and screenshots are only kept for failing runs, to let you review any issues. If you want to keep them for all runs, even runs that pass, select "On, and keep all". This setting would let you keep video recordings for other purposes, like documentation, tutorials or marketing. There are so many other great settings to explore in the Configurations tab. To learn more about them, check out “Author fast and reliable tests for Xcode Cloud” from WWDC22.

Xcode Cloud is a service built into Xcode that’s also available in App Store Connect. It can help you build your app, run tests, upload to the App Store and so much more. All of that happens in the cloud without you using any of you or your team’s devices. I think you’ll find that when it comes to Xcode Cloud, it’s all sun and games. For the Landmarks app, we’ve configured an Xcode Cloud workflow that runs all of the UI automations I just wrote, using the test plan I just created. This plan will run the same way in the cloud as it was run on my simulator, on any number of devices and configurations like English, Arabic, Hebrew, and German on iPhone and iPad.

You can view a history of your Xcode Cloud runs from within Xcode or in the Xcode Cloud section of App Store Connect. There, you can see an overview of build information, logs, failure descriptions and more.

Using Xcode Cloud, my entire team can see a history of my runs and download results and video recordings from them. They can do that even if I’m literally in the clouds, and by that I mean on my flight to Sydney.

There’s so much more to learn about Xcode Cloud. For more advanced configurations, check out “Create practical workflows in Xcode Cloud” from WWDC23. Now that we’ve run our recorded tests using a test plan in multiple configurations, we can review the results and the video recordings using the test report. The Xcode test report has some great tools to help you view, understand and diagnose your test results. It looks like one of our runs failed from the automation I just ran. Guess I can’t pack my bags for Australia just yet.

To navigate to my failing test, I’ll click the Test button, then I’ll double click the failing run to see a video recording and a description of what just happened.

I can see all of the runs in this test in the runs drop-down. This lets me quickly switch between video recordings of my test running in different configurations, like different languages. Also, fun fact, I can download the video by using a secondary click and choosing Save.

I’ll press Play to start video playback.

As the video plays, dots showing UI interactions are overlayed on top of the video. These actions are also represented in the timeline below as dots.

Looks like there will be a bit of time before my failure, so let’s skip ahead. I’ll jump straight to the failure moment using the failure diamond on the timeline.

I see a failure message, but it’s hard to say what went wrong. The message says we’re looking for a button called Max’s Australian Adventure. Let’s see what was actually present at the point of failure.

At the moment of failure, I see an overlay of all of the UI elements that were present right on top of the video recording.

If I click any of these, I get code recommendations for ways I can address this element in my automation code. I can even hit Show All to see alternative examples and find one that works for me.

I think I see what's wrong. We were expecting a button, but there’s no button here. It's just text. Let me fix that really quick.

I’ll select the sample I want and secondary click to copy it.

Then, I can click View Source to go directly to my tests and paste the new line of code over my existing one.

Now, I can replace the temporary XCUIApplication variable with the app variable for my UI recording, and I’m good to go.

Great, now this should run as expected. Let me click the test diamond to watch the test rerun.

This time, I’ll run the test in Arabic to see that the same automation works even when my app is running in a right-to-left layout.

The automation quickly creates my collection and renames it just as it does in English.

Pretty cool.

Looks like the automation passed. Time to finish this up and go on the trip of a lifetime. Maybe my mom and sister can come along to show me around. There’s so much more we can do with the Xcode test report. Luckily, the session “Fix failures faster with Xcode test reports” from WWDC23 really goes in depth and covers it all. It’s amazing how UI automation, accessibility, localization, Xcode Cloud and the test report all work together to increase the quality of your app and make it easier to use for everyone around the world. Bringing these technologies together into a single flow has been such a joy, and I can’t wait to see how it’s used by developers.

You can learn more about unit testing and Swift testing in the talk “Meet Swift Testing” from WWDC24. If you have additional questions or feedback, you can find us in the Developer Forums. Thanks for following along, and I’ll see you in Australia.