fix: replace CARemoteLayerServer with CAContext+CALayerHost for cross-process GUI
Sitar118
0548359
Three bugs killed the GUI:
1. CARemoteLayerServer does not capture a hidden NSWindow's content cross-process.
Replaced with CAContext (server) + CALayerHost (client) — the correct private-but-stable
macOS API for cross-process layer embedding, stable from 10.10 through Sequoia.
2. NSWindow was never orderFront: — WindowServer never composited its content.
Fixed: window positioned at (-32000,-32000) and orderFront: called.
3. NSMachBootstrapServer (deprecated on 13+) no longer needed.
contextId (uint32_t) is sent directly over the existing IPC socket.
New file: Source/Shared/PrivateCA.h — declares CAContext, CALayerHost, CGSMainConnectionID.
fix: wire onGuiCreated callback in HelperIPC constructor to send gui_ready notification
HelperPluginHost now has onGuiCreated callback. HelperIPC sets it.
When showGui() finishes creating the editor + registering the layer,
it calls onGuiCreated which triggers sendGuiReady with port name + size.
fix: use setLayerPortName() instead of direct member access
Bug 1: paths with spaces got split by tokenizer. Fix: join all args after --test.
Bug 2: some plugins need the JUCE message loop to finish loading (VST3 validation).
Fix: use a Timer to delay quit, giving the plugin time to initialize.
Root cause: commit 0632681 removed setAlwaysOnTop(true) and replaced it
with manual objc_msgSend level tricks. Those tricks don't keep the window
in the same Space as Ableton when fullscreen.
Fix: put back setAlwaysOnTop(true) + keep the X button fix (closeButtonPressed
hides window instead of quitting). Removed all objc_msgSend hacks.
revert main.cpp — remove NSApplicationActivationPolicyAccessory (this broke the fullscreen behavior)
The original code had NO activation policy override. macOS was showing
the Helper window in Ableton's Space by default. Adding Accessory policy
caused macOS to treat it differently and put it in a separate Space.
Going back to the original main.cpp that was working before the close-button fix.
STOP — revert HelperPluginHost to last known working state (GUI in Helper but with original window behavior before all my 'fixes')
I've been making this worse. The original code worked — GUI opened inside
Ableton's window. The problem started when close-button fix changed window
behavior. Need to go back to what worked and make minimal close-button fix only."
fix: show plugin GUI in-process (inside Ableton window) not from Helper
ROOT CAUSE: Moving GUI to Helper process broke it. macOS treats Helper
as separate app → opens in different Space when fullscreen.
SOLUTION: PluginBridge.vst3 (in-process) loads a GUI-only plugin instance.
This instance doesn't process audio — it's only for displaying the editor.
Ableton manages it as a normal plugin window (same as Pro-Q 4 native).
Parameter sync: GUI instance ↔ Helper via IPC (bidirectional).
- User moves knob on GUI → notify Helper → Helper updates real instance
- MCP/automation changes param on Helper → notify Plugin → update GUI instance
Helper no longer shows any GUI window.
fix: set fullscreen-compatible behavior BEFORE making window visible
Previous fix set collection behavior after setVisible(true) which is
too late — macOS has already assigned the window to a Space.
Now sets canJoinAllSpaces + fullScreenAuxiliary + stationary (1<<4)
BEFORE the window becomes visible. Also uses NSPanel-level behavior
and moves window to active Space explicitly via orderWindow:relativeTo:.
If this still fails in fullscreen, the only remaining option is to
render the plugin GUI in-process (framebuffer sharing).
fix: GUI now floats over Ableton in fullscreen mode
Root cause: NSWindow needs collection behavior flags to appear in
fullscreen Spaces. Without them, macOS puts the window in a separate
Space (different desktop), causing the 'opens in new window' behavior.
Fix adds to showAsFloatingPanel():
- canJoinAllSpaces (1 << 0) — window can appear on any Space
- fullScreenAuxiliary (1 << 8) — window can overlay fullscreen apps
- Window level raised to 25 (NSStatusWindowLevel) which is above
fullscreen app level
This means the plugin GUI will float over Ableton whether Ableton
is windowed OR fullscreen.
fix: set NSApplicationActivationPolicyAccessory at launch to prevent separate app appearance
The helper was appearing as a separate app in Dock/Cmd+Tab because
the activation policy was never being set. The showAsFloatingPanel()
only handled window level, not the application-level policy.
Now sets policy=1 (Accessory) as the FIRST thing in initialise(),
before any windows are created. This means:
- No Dock icon
- No Cmd+Tab entry
- No focus steal when helper launches
- GUI window floats over Ableton as a panel
Fix GUI focus: Remove makeAccessoryProcess entirely, use NSWindow level trick to float without activation
