feat: add isHittable attribute to iOS page source #1021
Conversation
| } | ||
|
|
||
| /*! Whether the element is truly hittable based on XCUIElement.hittable */ | ||
| - (BOOL)isWDNativeHittable |
There was a problem hiding this comment.
this will be a serious hit to the performance as it would try to resolve each element in the tree. Consider checking fb_filterDescendantsWithSnapshots: onlyChildren: for more details on how to match a snaphot to an element instance.
| /*! Whether the element is truly hittable based on XCUIElement.hittable */ | ||
| - (BOOL)isWDNativeHittable | ||
| { | ||
| XCUIApplication *app = [XCUIApplication fb_activeApplication]; |
There was a problem hiding this comment.
this might be suboptimal. Sometimes, the interface method might be called on XCUIElement instance, which would result in element->snapshot->element chain. It is necessary to modify fb_valueForWDAttributeName to avoid that.
| } | ||
|
|
||
| // Adding isHittable, only if not excluded | ||
| if (excludedAttributes == nil || ![excludedAttributes containsObject:FBExclusionAttributeHittable]) { |
There was a problem hiding this comment.
this attribute must be excluded by default. Having it in the tree by default would seriously slow down the tree retrieval for all users
| /*! Whether the element is considered hittable based on snapshot hit point */ | ||
| @property (nonatomic, readonly, getter = isWDHittable) BOOL wdHittable; | ||
|
|
||
| /*! Whether the element is truly hittable based on XCUIElement.hittable */ |
There was a problem hiding this comment.
I'm still not convinced that XCTest itself uses a different method to determine element hittability in comparison to the one we do:
- (BOOL)isWDHittable
{
XCUIHitPointResult *result = [self hitPoint:nil];
return nil == result ? NO : result.hittable;
}
@Dan-Maor @KazuCocoa What do you think?
There was a problem hiding this comment.
Yea, I think still isWDHittable is valid to return hittable by XCUIElement in the backtrace basis
There was a problem hiding this comment.
This is the disassembly of the -[XCUIElement isHittable] function, and the underlying logic is pretty much the same isWDHittable @mykola-mokhnach mentioned here, only difference is that the element is first explicitly resolved and a fresh snapshot is retrieved on which the hitPoint function is called. If retrieving the hittable attribute in a getAttribute fetches a new snapshot then the current logic appears to be identical.
There was a problem hiding this comment.
@mykola-mokhnach @Dan-Maor @KazuCocoa
I ran a direct comparison between isWDHittable (based on hitPoint:nil) and XCUIElement.hittable, and the results are significantly different in practice.
While isWDHittable tends to return "true" for almost every visible element in the hierarchy, regardless of accessibility or interactivity, XCUIElement.hittable produces more accurate results — for example, only elements that are actually reachable and interactable (like visible buttons or text) are marked as "true".
You can see the diff here, where I removed everything except type, label, isHittable, and rawIdentifier from the source trees and compared them:
📎 filtered_XCUIElement.hittable.json
📎 filtered_isWDHittable.json
This confirms that isWDHittable is not a reliable substitute for XCUIElement.hittable, and justifies exposing the native runtime value separately via isNativeHittable.
There was a problem hiding this comment.
Btw, the true/false in your expected hittable is similar or equal to "isAccessible" : "1" result...?
I found that https://stackoverflow.com/questions/48845519/xcuielement-exists-but-is-not-hittable, so the hittable needs to be isAccessibilityElement. So the hitPoint AND isAccessibilityElement is the XCUIElement.hittable?
There was a problem hiding this comment.
it could be that this difference is the result of snapshotting logic. E.g. when the getter is invoked from XCUIElement instance then a new element snapshot is taken via resolveOrRaiseTestFailure. In case of fb_tree invocation the snapshot is only taken once for the whole XCUIApplication instance.
You may validate this theory by retrieving the element hittability from each element as an attribute instead of checking it in the tree.
Nevertheless, the above change would require to take a snapshot from each element in the page tree, and there might be hundreds of them. We cannot let such change in, definitely not without an explicit setting that would enable such behavior.
| } | ||
|
|
||
| // Return hittable status if element is found, otherwise NO | ||
| return matchedElement.exists ? matchedElement.hittable : NO; |
There was a problem hiding this comment.
What happens when you use matchedElement.hittable only? Removing .exists handle would be more appropriate to get hittable directly. The result was also slightly different in a home screen. Also, the above's .exists usage also can be hittable only?
Based on XCTest's documentation, hittable might include the existence check
https://developer.apple.com/documentation/xctest/xcuielement/ishittable?language=objc
https://developer.apple.com/documentation/xctest/xcuielement/exists?language=objc
|
Hi, I tried everything suggested so far (snapshot flattening, resolving elements via Unfortunately, the results are still not correct — they’re nearly identical to Can you help clarify what the correct resolution flow should be, step by step? Thanks! |
| } | ||
|
|
||
| /*! Whether the element is truly hittable based on XCUIElement.hittable */ | ||
| - (BOOL)isWDNativeHittable |
There was a problem hiding this comment.
I wondered if this return value can be simplified as self.isWDAccessible && self.isWDHittable && self.isWDVisible for now.
In appium/appium-xcuitest-driver#2565 testing, XCTest's isHittable returned the same value of self.isWDAccessible && self.isWDHittable. It could be different in a complicated screen, but I wondered if it would be worth verifying the self.isWDAccessible && self.isWDHittable && self.isWDVisible is sufficient as this intention @ikharebashviliGD
I'm not sure this could be reasonable in terms of performance. Possibly getting vanilla XCTest's isHittable would help rather than this idea.
There was a problem hiding this comment.
@KazuCocoa
Thanks for the suggestion — I implemented the isWDAccessible && isWDHittable && isWDVisible approach and tested it across several UI cases.
The results match exactly what I observed earlier using .hittable on resolved XCUIElement instances, so this seems like a reliable and much more efficient solution.
I didn’t add an extra configuration flag for excluding this attribute, since the value is now computed entirely from snapshot-based data and doesn't introduce the performance cost that .hittable resolution would.
Let me know if you’d still prefer it to be gated explicitly — otherwise I think this version is ready.
There was a problem hiding this comment.
After further investigation by creating a simple (default template), I discovered that your original implementation also didn't follow vanilla XCTest's native value in some cases. -> appium/appium-xcuitest-driver#2565
For example, the top element of the app should be hittable according to the vanilla XCTest, but /source returned false for the element with this branch.
#1022 will update the isWDHittable logic for the getting hittable attribute endpoint call. This PR doesn't affect page source/XPath element search, thus I assume this PR will be applying the change for XML stuff. Then, using the isWDHittable will be more accurate than the current isWDNativeHittable.
There was a problem hiding this comment.
I’ve tried replacing the resolved XCUIElement.hittable with isWDHittable inside /source, but in my testing, it always returns "true" for almost every element — even for non-interactive ones like Application, Other, or Image.
Here’s a sample of the output where isHittable: true is present across the entire hierarchy (including non-hittable types):
{
"type": "Application",
"label": "TestApp-UIKit",
"isHittable": "1",
...
}Because of this, I don’t think isWDHittable is a meaningful replacement here.
Would it make sense to instead expose isWDResolvedHittable, which is based on isWDAccessible && isWDHittable && isWDVisible?
This value has shown good consistency with vanilla XCUIElement.hittable in practice.
Let me know how you'd prefer to proceed — happy to implement whatever direction makes the most sense. Thanks again! 🙏
There was a problem hiding this comment.
The PR itself is not sufficient to apply the change. #1023 plus like below change retrieves proper data for hittable attribute. This diff is also not sufficient, but could be a quick check. Please don't request page source in JSON format, it just corrects the http://localhost:8100/source result. The PR fixes the hittable value via Get Element Attribute endpoint. I think JSON format would require more changes in some places.
diff --git a/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m b/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m
index 1737ff7a..6cb74557 100644
--- a/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m
+++ b/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m
@@ -48,6 +48,8 @@
static NSString* const FBExclusionAttributeFocused = @"focused";
static NSString* const FBExclusionAttributePlaceholderValue = @"placeholderValue";
static NSString* const FBExclusionAttributeNativeFrame = @"nativeFrame";
+static NSString* const FBExclusionAttributeHittable = @"hittable";
_Nullable id extractIssueProperty(id issue, NSString *propertyName) {
SEL selector = NSSelectorFromString(propertyName);
@@ -265,6 +267,9 @@ + (NSDictionary *)dictionaryForElement:(id<FBXCElementSnapshot>)snapshot
},
FBExclusionAttributeFocused: ^{
return [@([wrappedSnapshot isWDFocused]) stringValue];
+ },
+ FBExclusionAttributeHittable: ^{
+ return [@([wrappedSnapshot isWDHittable]) stringValue];
}
} mutableCopy];
kazu $ git diff
diff --git a/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m b/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m
index 1737ff7a..c3539c2d 100644
--- a/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m
+++ b/WebDriverAgentLib/Categories/XCUIApplication+FBHelpers.m
@@ -48,6 +48,8 @@
static NSString* const FBExclusionAttributeFocused = @"focused";
static NSString* const FBExclusionAttributePlaceholderValue = @"placeholderValue";
static NSString* const FBExclusionAttributeNativeFrame = @"nativeFrame";
+static NSString* const FBExclusionAttributeHittable = @"hittable";
+
_Nullable id extractIssueProperty(id issue, NSString *propertyName) {
SEL selector = NSSelectorFromString(propertyName);
@@ -265,6 +267,9 @@ + (NSDictionary *)dictionaryForElement:(id<FBXCElementSnapshot>)snapshot
},
FBExclusionAttributeFocused: ^{
return [@([wrappedSnapshot isWDFocused]) stringValue];
+ },
+ FBExclusionAttributeHittable: ^{
+ return [@([wrappedSnapshot isWDHittable]) stringValue];
}
} mutableCopy];
diff --git a/WebDriverAgentLib/Utilities/FBXPath.m b/WebDriverAgentLib/Utilities/FBXPath.m
index e11b52d8..96a11793 100644
--- a/WebDriverAgentLib/Utilities/FBXPath.m
+++ b/WebDriverAgentLib/Utilities/FBXPath.m
@@ -147,7 +147,7 @@ + (nullable NSString *)xmlStringWithRootElement:(id<FBElement>)root
if (rc >= 0) {
[self waitUntilStableWithElement:root];
- rc = [self xmlRepresentationWithRootElement:[self snapshotWithRoot:root useNative:NO]
+ rc = [self xmlRepresentationWithRootElement:[self snapshotWithRoot:root useNative:YES]
writer:writer
elementStore:nil
query:nil
@@ -356,7 +356,7 @@ + (int)xmlRepresentationWithRootElement:(id<FBXCElementSnapshot>)root
includedAttributes = [NSMutableSet setWithArray:FBElementAttribute.supportedAttributes];
// The hittable attribute is expensive to calculate for each snapshot item
// thus we only include it when requested by an xPath query
- [includedAttributes removeObject:FBHittableAttribute.class];
+// [includedAttributes removeObject:FBHittableAttribute.class];
if (nil != excludedAttributes) {
for (NSString *excludedAttributeName in excludedAttributes) {
for (Class supportedAttribute in FBElementAttribute.supportedAttributes) {There was a problem hiding this comment.
Also, you could try out KazuCocoa/hittableTest@f2824b2 XCTest test code for your test app. The result of allHittableElementLabels tells you what element labels must be "hittable" in your view as vanilla XCTest framework.
So far, I have confirmed #1023 (comment) against https://github.com/KazuCocoa/hittableTest had the same result with vanilla XCTest. You could try out like @driver.find_elements :xpath, '//*[@hittable="true"]' XPath query to see if the #1023 change gets isHittable elements on your app. (Then, please try out without bundleId to retrieve elements from the top https://appium.github.io/appium-xcuitest-driver/latest/guides/troubleshooting/#interact-with-dialogs-managed-by-comapplespringboard)
There was a problem hiding this comment.
@KazuCocoa @mykola-mokhnach
Hi all,
After further investigation and testing, I’ve created a follow-up PR that provides a better approach for exposing isHittable.
In this new PR, I’ve added a new flag use_native_hittable for the /source?format=xml endpoint. When passed, it forces the use of native snapshots and includes the accurate hittable attribute in the XML output.
This approach:
- Avoids changes to the JSON representation
- Keeps performance impact under control (opt-in only)
Looking forward to your feedback on the new solution!
PR link: #1026
Replaced XCUIElement-based hittability with a lightweight estimation: isWDAccessible && isWDHittable && isWDVisible. This approach avoids performance overhead of resolving live elements, while producing results consistent with vanilla `.hittable` in testing.
|
Closing this as #1026 is a more proper way |
4 participants
Summary
This PR adds support for exposing the
isHittableattribute in the/sourcetree by introducing a new propertywdNativeHittableonFBXCElementSnapshotWrapper. The value reflects the runtimeXCUIElement.hittablestatus, which is already available in the Mac2 driver and is part of the public XCTest API.Motivation
Knowing whether an element is actually hittable at runtime helps determine if it's ready for interaction — for example, when it becomes visible after scrolling or animations. The current snapshot-based approach (
wdHittable) is often too optimistic and doesn't reflect real-world behavior, leading to flaky tests.Adding
isHittableto the page source allows clients to make better decisions during automation, improves stability, and brings iOS driver behavior closer to Mac2.Related issue: [appium/WebDriverAgent#1005](#1005)
Implementation
- (BOOL)isWDNativeHittablewas added toFBXCElementSnapshotWrapper.XCUIElementat runtime based on itsidentifierorlabel, filtered by type..hittablevalue is returned and exposed as a string in the/sourceoutput via theisNativeHittablefield.excludedAttributes.Feedback welcome
If there's a more efficient or consistent way to resolve the original
XCUIElementfrom a snapshot — especially one backed by session context — suggestions are appreciated.