Axe

Axe is a powerful, type-safe Swift wrapper around the macOS Accessibility API (AXUIElement). It provides a fluent, chainable interface for traversing the UI hierarchy, interacting with elements, and building robust UI automation and accessibility tools.
While AppleScript has traditionally been the go-to for macOS automation, its reliability has waned with recent macOS updates. Axe was created as a modern, robust, and type-safe replacement.

Features

• AppleScript-like: If you are familiar with AppleScript, you can use Axe with ease.
• Type-safe: Allows safely interacting with the low-level AXUIElement APIs when necessary.
• Useful high-level APIs: Traverse complex UI hierarchies with readable, chainable calls (e.g., .window(1)?.group(2)?.button("Submit")), or find elements by role, identifier, title, value, or custom predicates.
• Input Simulation: Simulate mouse clicks, keyboard events, and other interactions reliably.
• Event Observation: Easily subscribe to accessibility notifications (e.g., window creation, title changes) with Swift concurrency support.

Requirements

• macOS 12.0+
• Swift 5.5+

Installation

Add Axe to your Package.swift dependencies:

dependencies: [
    .package(url: "https://github.com/lnquy/axe.git", from: "0.1.0")
],
targets: [
    .target(
        name: "MyTool",
        dependencies: [
            .product(name: "Axe", package: "axe")
        ]
    )
]

Note: Your application must have "Accessibility" permissions enabled in System Settings > Privacy & Security > Accessibility to control other applications.

Usage

Getting Started

Import the library and find an application to automate:

import Axe

// Find a running application by Bundle ID
guard let messages = App.find(bundleID: "com.apple.MobileSMS") else {
    print("Messages app is not running")
    exit(1)
}

print("Found Messages: \(messages.pid)")

Finding Elements

Axe provides a hierarchical way to navigate the UI tree. You can access children by index, role, or other criteria.

// Access hierarchy: Application -> First Window -> First Group -> Button
if let confirmButton = try messages.window(1)?.group(1)?.button(1) {
    print("Found button: \(confirmButton)")
}

// Find a specific element by Identifier
let saveButton = try? messages.find(role: .button, identifier: "Save")

Interacting with Elements

Perform actions like clicking, getting/setting attributes, and more.

// Click a button
try button.perform(.press)

// Use helpers for common attributes
let title: String = try window.getTitle() ?? "Untitled"
let isEnabled: Bool = try button.isEnabled()

// Use attribute() for attributes that don't have a helper function
let size: CGSize = try window.attribute(.size) ?? .zero

// Set attributes
try window.setAttribute(.position, value: CGPoint(x: 100, y: 100))

Advanced Search

Use find and findAll with a closure when pre-defined helper chains (like .button(1) or .find(role: .button, identifier: "...")) are not enough for your custom logic.

// Find the first enabled button with a specific title
let submitBtn = try window.find(role: .button) { button in
    try button.isEnabled() && button.getTitle() == "Submit"
}

// Find all enabled text fields in the app
let enabledTextFields = try app.findAll(role: .textField) { field in
    try field.isEnabled()
}

Simulating Input

Axe provides a powerful simulation engine to interact with elements when standard accessibility actions are not enough.

// Click on an element or specific coordinates
try Simulate.click(on: button)
try Simulate.click(at: CGPoint(x: 100, y: 200))

// Move mouse to coordinates
try Simulate.move(to: CGPoint(x: 500, y: 500))

// Type text into a focused field
try Simulate.type("Hello world!")

// Press keys with modifiers (e.g., Command + S)
try Simulate.press(key: .letter("s"), modifiers: .command)

// Scroll vertically or horizontally
try Simulate.scroll(amount: 10, on: scrollArea) // Scroll down
try Simulate.scrollHorizontal(amount: -5, on: scrollArea) // Scroll left

// Smoothly scroll an element into view
try element.scrollIntoView()

Event Observation

Observe accessibility notifications using Swift's async/await.

// Observe title changes on a window
for await event in try window.observe(notification: .titleChanged) {
    print("Window title changed!")
}

Documentation

For more detailed examples and advanced usage, check out the docs/ and examples/ directories.

• examples/AxeRecorder: A tool to record UI interactions and print the details to the console.
• examples/ScrollDemo: A demo to show how to scroll an element into view.

Development

To start working on Axe, clone the repository and ensure you have Swift installed.

Building

make build

Running Tests

Axe includes both unit tests and integration tests aiming for high code coverage.
For more details on testing, see docs/TESTING.md.

License

This project is licensed under the MIT License.

Inspirations

• AXSwift: Swift wrapper for accessibility clients.
• pyautogui: A cross-platform GUI automation Python module for programmatically control the mouse & keyboard.
• MacosUseSDK: Library to traverse and control MacOS.

References:

1. https://developer.apple.com/documentation/applicationservices/
2. https://leopard-adc.pepas.com/documentation/Accessibility/Reference/AccessibilityCarbonRef/AccessibilityCarbonRef.pdf
3. https://jenkins.heirloomcomputing.com/downloads/MacOSX10.8.sdk/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/Headers/
4. https://leopard-adc.pepas.com/documentation/Carbon/Conceptual/MakingAppsAccessible/AXCarbonIntro/AXCarbonIntro.html