GitHub - microsoft/FSQ: FSQ is an evidence-first agent harness for replayable, verifiable AI UI automation across web, mobile, and desktop.

An evidence-first agent harness for replayable, verifiable AI UI automation.

Quick Start • Why FSQ • Platforms • Documentation • Contributing

Why FSQ?

Evidence-First

Every step captures screenshots, UI snapshots, and action traces. You verify through evidence, not agent self-reports.

Replayable

Successful AI runs auto-generate strict YAML. Replay deterministically without LLM — same harness, same evidence, zero flakiness.

Verifiable

Results are judged by an evidence-based verifier, not the agent claiming success. Auditable, trustworthy, CI-ready.

Other AI agents say "I'm done." FSQ shows you the proof.

See It in Action

┌──────────────────────────────────────────────────────────────────────┐
│ $ cd /path/to/workspaces/web-demo                                   │
│ $ fsq-agent run --platform web --record                             │
│     --goal "Search for FSQ on Bing"                                │
├──────────────────────────────────────────────────────────────────────┤
│  ► Planning: 3 key actions identified                                │
│  ► Step 1: startBrowser          📸 screenshot + UI snapshot         │
│  ► Step 2: navigateTo bing.com   📸 screenshot + UI snapshot         │
│  ► Step 3: typeText "FSQ"        📸 screenshot + UI snapshot         │
│  ► Step 4: pressKey Enter        📸 screenshot + UI snapshot         │
│  ► Verification: PASSED ✅ (evidence-based)                          │
│  ► Recording manifest → .fsq/runs/web/<run-id>/recording.json       │
│  ► Replayable YAML → .fsq/runs/web/<run-id>/recorded.fsq.yaml       │
└──────────────────────────────────────────────────────────────────────┘

What Can FSQ Do?

	Dynamic Mode 🤖 _{AI-driven exploration}	Strict Mode 🔁 _{Deterministic replay}
AI explores and operates your app	✅	—
Evidence captured on every step	✅	✅
Auto-generates replayable YAML	✅	—
Deterministic regression execution	—	✅
AI-powered visual assertions	✅	✅
Runs without LLM	—	✅

The Dual Loop: AI explores → evidence proves it worked → strict YAML locks it down → replay catches regressions.

Quick Start

1. Install

pip install fsq-agent[web]

Using uv (recommended for development)

curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --extra dev --extra web

2. Create a Workspace

fsq-agent control-plane

Open Workspace, create a workspace for the application under test, and give it a unique name such as web-demo. A workspace can contain any combination of Android, Web, Windows, and macOS platform configurations. Add one or more platforms during creation, then add or edit platforms independently from the workspace configuration page.

For non-interactive setup, init creates exactly one platform per invocation:

cd /path/to/workspaces

# Create and register a Web workspace platform.
fsq-agent init --name web-demo \
  --platform web \
  --browser-executable-path /path/to/google-chrome

# Add Android to the same registered workspace.
fsq-agent init --name web-demo \
  --platform android \
  --app-id com.example.app

init always creates or updates <current-directory>/<name>. Platform target options are:

Platform	`init` target options
Android	`--app-id APP_ID` (required)
Web	`--browser-executable-path FILE` (required)
Windows	`--app-path PATH` (required), plus optional `--window-title-re` and `--launch-args`
macOS	`--bundle-id` or `--app-path` (at least one required)

Repeating an equal platform configuration returns unchanged. If its target or private environment mapping differs, pass --update-existing to replace only that platform's target and --env NAME=VALUE entries. Use fsq-agent --output json init ... or --output jsonl for one-record machine output. Command-line environment values may be visible in shell history or process inspection.

run must be launched from the exact workspace root and requires an explicit configured platform. It validates the current directory directly instead of looking up a registry name or searching parent directories. report and playground keep their registered workspace-name selection and can run outside the workspace directory. Legacy .fsq/config.yaml and .fsq-agent-workspace layouts are not migrated.

3. Configure a Provider

Open Config, select Add configuration, and configure one active Provider:

Provider	Setup
GitHub Copilot GPT	Model name and GitHub device-code authentication
Azure GPT	Azure OpenAI-compatible base URL, model/deployment name, and API key

Provider configuration is stored under ~/.fsq and is used by the next complete FSQ task. Config is available only when Control Plane is bound to and accessed through a loopback address.

4. Run

cd /path/to/workspaces/web-demo

# Dynamic: AI-driven exploration with full evidence
fsq-agent run --platform web \
  --record \
  --goal "Open https://www.bing.com, search for 'FSQ automation', verify results appear."

# Strict: Deterministic replay from YAML (no LLM needed)
fsq-agent run --platform web --strict \
  --case-yaml path/to/case.fsq.yaml

# Strict batch: Run matching cases under cases/web/.
fsq-agent run --platform web --strict

Strict directory runs, including the no-input default, recursively load exact .fsq.yaml files and skip cases authored for another platform. If files are present but none match --platform, the command reports zero cases and succeeds without creating run artifacts. A single --case-yaml platform mismatch is an error.

Output: Every executed run writes evidence and a report under .fsq/runs/<platform>/<run-id>/. For a dynamic run, --record writes a recording.json manifest for the recording attempt and writes recorded.fsq.yaml when the run is eligible and contains replayable commands. Failed runs are skipped unless --record-on-failure is also supplied. Strict runs do not record new cases.

Control Plane

Launch the local browser Control Plane for multi-platform workspace management, platform readiness, target and case discovery, Explore runs, Strict Replay, and live evidence:

fsq-agent control-plane

It listens on 127.0.0.1:8879 and opens a browser by default. Use --host, --port, and --no-open-browser to override those defaults. A wheel installation includes the compiled frontend and needs no Node.js runtime. From a source checkout, run npm ci && npm run build before starting the Control Plane.

Supported Platforms

Platform	Backend	Install
🌐 Web	Playwright	`pip install fsq-agent[web]`
📱 Android	uiautomator2	`pip install fsq-agent[android]`
🖥️ Windows	pywinauto	`pip install fsq-agent[windows]`
🍎 macOS	Appium Mac2	`pip install fsq-agent[macos]`

All platforms share the same HarnessInterface, evidence model, and FSQ YAML format. A registered workspace may configure one or more platforms independently:

<workspace-root>/
  .fsq/config/config.<platform>.yaml
  cases/<platform>/
  knowledge/<platform>/
  .fsq/runs/<platform>/

The run command requires --platform PLATFORM and uses the exact current directory as its workspace root. The report and playground commands require both --workspace NAME and --platform PLATFORM. The control-plane command manages browser selection instead and accepts neither option.

Platform setup details

Web — Set the browser executable in .fsq/config/config.web.yaml:

target:
  browser_executable_path: /usr/bin/google-chrome

Android — Set the app ID in .fsq/config/config.android.yaml, then select a connected ADB device per run:

target:
  app_id: com.example.app

cd /path/to/workspaces/my-workspace
fsq-agent run --platform android \
  --android-serial emulator-5554 --goal "Open the app"

When exactly one device is online, --android-serial may be omitted.

Windows — Keep backend_kind in the repository preset config.windows.yaml; set app-specific values in .fsq/config/config.windows.yaml:

target:
  app_path: C:\Program Files\MyApp\app.exe
  window_title_re: .*MyApp.*

macOS — Keep appium_server_url in the repository preset config.macos.yaml; set the app identity in .fsq/config/config.macos.yaml:

target:
  bundle_id: com.example.app

How It Works

The Dual Loop in a nutshell:

Dynamic (AI) → LLM agent explores → evidence captured at every step → replayable YAML generated
Strict (Replay) → replays YAML deterministically (no LLM) → evidence captured → pass/fail

Compared To...

	FSQ	Browser Use	Midscene.js	Playwright	Appium
Evidence per step	✅ screenshots + UI snapshots + traces	❌	❌	❌	❌
AI → Replay YAML	✅ auto-generated strict cases	❌	❌	Codegen (manual)	❌
Verification	Evidence-based verifier	Agent self-report	Vision assert	Manual assertion	Manual assertion
Cross-platform	Web + Android + Windows + macOS	Web only	Web + Mobile	Web only	Multi (different APIs)
Runs without LLM	✅ Strict mode	❌	❌	✅	✅
Extensible harness	Protocol-based plugin system	❌	❌	❌	Driver plugins

Documentation

Resource	Description
Architecture Overview	Dual Loop design and module structure
Platform Setup	Detailed per-platform configuration
FSQ YAML Reference	DSL syntax, lifecycle hooks, replay semantics
Harness Development Guide	Build a new platform harness
Roadmap	Product direction and planned phases
Governance	Roles, decisions, and maintainer responsibilities

Contributing

We welcome contributions! FSQ is designed to be extended.

git clone https://github.com/microsoft/FSQ.git && cd FSQ
uv sync --extra dev
npm ci && npm run build
uv run python -m pytest

Ways to contribute:

Path	For whom
🐛 Report bugs / suggest features	Everyone
📝 Improve docs and examples	Beginners welcome
🧪 Add FSQ YAML test cases	QA engineers
🔌 Build a new platform harness	Platform experts
⚡ Improve agent / verification	AI engineers

See CONTRIBUTING.md for the full guide and contributor growth path. Project decisions and role progression follow GOVERNANCE.md.

License

MIT — Copyright (c) Microsoft Corporation.

_{Built with ❤️ by the FSQ team at Microsoft}

Name		Name	Last commit message	Last commit date
.github		.github
docs		docs
frontend		frontend
fsq_agent		fsq_agent
knowledge/skills		knowledge/skills
scripts		scripts
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
GOVERNANCE.md		GOVERNANCE.md
LICENSE		LICENSE
README.md		README.md
ROADMAP.md		ROADMAP.md
SECURITY.md		SECURITY.md
SPEC.md		SPEC.md
config.android.yaml		config.android.yaml
config.example.yaml		config.example.yaml
config.macos.yaml		config.macos.yaml
config.web.yaml		config.web.yaml
config.windows.yaml		config.windows.yaml
package-lock.json		package-lock.json
package.json		package.json
pyproject.toml		pyproject.toml
tsconfig.json		tsconfig.json
uv.lock		uv.lock
vite.config.js		vite.config.js
vitest.config.ts		vitest.config.ts

Navigation Menu

Uh oh!

Repository files navigation

An evidence-first agent harness for replayable, verifiable AI UI automation.

Why FSQ?

See It in Action

What Can FSQ Do?

Quick Start

1. Install

2. Create a Workspace

3. Configure a Provider

4. Run

Control Plane

Supported Platforms

How It Works

Compared To...

Documentation

Contributing

License

About

Releases

Packages

Used by

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

An evidence-first agent harness for replayable, verifiable AI UI automation.

Why FSQ?

See It in Action

What Can FSQ Do?

Quick Start

1. Install

2. Create a Workspace

3. Configure a Provider

4. Run

Control Plane

Supported Platforms

How It Works

Compared To...

Documentation

Contributing

License

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages