Java | Software Development | Testing
Mutation testing with Pitest 
Jan Vermeir 12 Feb, 2020
Share 
XCTest UI Testing
When Apple introduced the new UI automation framework in 2015 we became excited. We could finally test the UI and the behaviour of our applications using Swift (or Objective-C) and the native library.
Previously, we had to write scripts in JavaScript and execute them using Instruments. This was not the easiest way to write, maintain, and debug tests. Another option was to use the KIF framework, which allows us to write integration tests and to test the UI of the application. One major downside of this is the usage of the private API which could change every iOS / Xcode version. Thankfully, since Apple introduced the new framework things began to change…but did they really?
When you start playing with it for a longer period of time you will notice that the behaviour of some methods is different depending on the iOS and/or Xcode version. But this is something Apple has been used to for some time now. Although, the real fancy things are just starting to appear. When you finally manage to write working UI tests, and you reach the magical number of 128 test cases you will find the first Easter egg.
Pseudo Terminal Limit
At the beginning, you won’t notice anything. Run Xcode, open the project, execute tests, and everything will work as expected. But when you configure Fastlane, Travis CI, Jenkins, the Xcode Server, or any other Continuous Integration system to build your project you will notice that after around test 128 all of the tests begin to fail and the same error occurs
Error Domain=IDEPseudoTerminalDomain Code=1 "(null)"This issue is very interesting because it does not occur on Xcode. It only occurs if you are executing xcodebuild directly (or by using wrappers like gym), so debugging the issue is not straightforward.
I searched for a solution on Google, but unfortunately I did not manage to find one. I only came upon information regarding some issues reported in 2014 without any results. Resetting the iOS Simulator did not help either, so I came up with the idea to conduct some research, more specifically, to dive into Xcode and try to find my own solution.
At the beginning I had no idea how to debug this issue. Everything was working on Xcode so I was unable to use any breakpoints or a debugger. My first approach was to execute only the failing tests by using an -only-testing flag. This did not help. I managed to execute the tests without any problems. My next thought was that the execution order can have something to do with it. Disabling tests from the scheme did not help either. After reaching 128 tests, they started to fail. After a few tries, I grew tired and felt that I have had enough. We had written almost 150 tests already. Executing the whole test suite took almost an hour. Checking a few cases took me the whole day.
In the next step, I wanted to speed up the test execution. I hoped that this issue would not be not related to my project, so I created a small test project. This simple application has 200 UI tests which do not do anything:
func testExample000(){}
func testExample001(){}
func testExample002(){}
// ...
func testExample199(){}
I executed the tests and "thankfully" they failed and the same error message appeared. This was proof that this issue was not related to my project, but rather to the xcodebuild tool.
After that, I started looking for a more general solution. If the error is related to some resource, here to the pseudo terminals, maybe I reached some kind of limit? After conducting a short investigation on Google, I found that macOS has a default limit for ptys set to 127.
$ sysctl kern.tty.ptmx_max
kern.tty.ptmx_max: 127
So it seems we have found the cause! But what is the source? Which process creates so many pseudo terminals?
When we execute a test from Xcode we can use lsof to see how many terminals are already allocated.
$ lsof /dev/ptmx
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
iTerm2 1688 user 0u CHR 15,0 0t13596 572 /dev/ptmx
iTerm2 2164 user 0u CHR 15,1 0t71369123 572 /dev/ptmx
Xcode 2173 user 31u CHR 15,6 0t7887 572 /dev/ptmx
Xcode 2173 user 49u CHR 15,5 0t0 572 /dev/ptmx
Xcode 2173 user 54u CHR 15,8 0t0 572 /dev/ptmx
Xcode 2173 user 56u CHR 15,7 0t7 572 /dev/ptmx
Xcode creates about 5 pseudo terminals and keeps this number constant. When we execute a test from xcodebuild we can notice a more interesting output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
iTerm2 1688 user 0u CHR 15,0 0t13596 572 /dev/ptmx
iTerm2 2164 user 0u CHR 15,1 0t71369123 572 /dev/ptmx
xcodebuil 8567 user 21u CHR 15,6 0t22726 572 /dev/ptmx
xcodebuil 8567 user 23u CHR 15,6 0t22726 572 /dev/ptmx
xcodebuil 8567 user 29u CHR 15,5 0t0 572 /dev/ptmx
xcodebuil 8567 user 31u CHR 15,5 0t0 572 /dev/ptmx
xcodebuil 8567 user 32u CHR 15,7 0t0 572 /dev/ptmx
xcodebuil 8567 user 34u CHR 15,7 0t0 572 /dev/ptmx
...
xcodebuil 8567 user 143u CHR 15,45 0t0 572 /dev/ptmx
xcodebuil 8567 user 145u CHR 15,45 0t0 572 /dev/ptmx
xcodebuil 8567 user 146u CHR 15,46 0t0 572 /dev/ptmx
xcodebuil 8567 user 148u CHR 15,46 0t0 572 /dev/ptmx
Each test creates a new terminal which is not deallocated. Around test 128, when the last allowed pty is spawn, Xcode is not able to start a new process on the Simulator and the rest of the tests fail.
This issue takes place only on the iOS Simulator. iOS Devices are not affected by this problem.
The First Workaround
Obviously the Apple developer forgot to release something after every test and we are unable to fix this issue without their help. So the only possibility is some kind of workaround. Thankfully, we can increase the maximum number of pseudo terminals to 999.
sudo sysctl -w kern.tty.ptmx_max=999When using sysctl the change is only temporary (until the system is restarted). To permanently save the value, create or open the file /etc/sysctl.conf by using:
sudo touch /etc/sysctl.conf
sudo chown root:wheel /etc/sysctl.conf
sudo chmod 644 /etc/sysctl.conf
echo "kern.tty.ptmx_max=999" | sudo tee -a /etc/sysctl.conf
By changing ptmx_max we can now test up to 999 test cases in a single scheme, but only in theory…
App Accessibility
When we start testing once again we will find another interesting issue. Around test 170 Xcode or xcodebuild welcome us with the following error:
Waiting for accessibility to load
Wait for app to idle
App event loop idle notification not received, will attempt to continue.
App animations complete notification not received, will attempt to continue.
After this error , all of the next tests fail with another message:
Waiting for accessibility to load
Assertion Failure: testUITests.swift:21: UI Testing Failure - App accessibility isn't loaded
This bug is more interesting than the previous one. It does not only happen for xcodebuild, but also for Xcode. What is even worse, is that it happens on both the iOS Simulator and the iOS device. Unfortunately Google does not help. There are topics on the Apple Developer Forum, but no solutions.
To investigate this issue, I set up the "Test Failure" and "Exception" breakpoints to catch the problem as soon as possible. The retrieved call stack does not contain anything useful. I also used fs_usage to track the created process and lsof to check how many resources were used by the Simulator and the Xcode tools:
sudo fs_usage -f exec
sudo lsof -p
Unfortunately, without any results. "App accessibility" is not a separate process, so it is not a problem connected with launching an app. lsof has shown about 700 open files, while macOS has a limit of 10240 open files per process (sysctlkern.maxfilesperproc). Bash sets this limit to 4864 open files (ulimit -S -n). Both values are not even close to what Xcode allocates.
I was not able to find a cause or a fix.
The Second Workaround
Without a good fix the only possibility is to create a workaround. The only one we came up with at PGS Software would be to create more shared schemes in the project. I would suggest to add no more than 160 test cases into one scheme.
Update
On Xcode 8.3 beta the situation looks a little bit better. The tests fail after 206 test cases. Three more years and maybe we will be able to run 300 tests.
Update 14.06.2017
On Xcode 9.0 beta it’s even better. The tests fail after 256 test cases. Apple is on the right track to reach 300 soon.
Summary
Until Apple finally fixes those, and probably more issues, writing and executing UI tests will be problematic for all iOS developers. For now, we can only look for workarounds and keep our fingers crossed that Apple will finally create a working and stable UI automation tool.
Links
by Xebia Author
