One note a month on AI security, from the people building the harness
What went wrong in the world, what it means for your agents, and what we changed because of it. No product announcements.
Reading progress: 0% complete
What went wrong in the world, what it means for your agents, and what we changed because of it. No product announcements.
Lessons from local AI, and the question of permission when capturing a conversation becomes routine.
Adesh Gairola
The Open Passage. A quiet riverside towpath in pale morning mist, dark woodland on the left and a small open gate in a low boundary on the right.
Lessons from local AI, and the question of permission when capturing a conversation becomes routine.
If you ask whether you can record our conversation, I'm usually fine with it.
It's the asking that matters to me. I get a moment to decide what I'm comfortable saying. You know whether I'm comfortable with you keeping it.
What I keep wondering about is what happens after capturing a conversation becomes routine. You switch on a useful feature. It helps you remember something. You keep using it. Does asking stay part of that routine?
A device can protect the conversation it captures while changing the social expectation that capturing it requires asking first.
Apple's Audio Intelligence makes that question concrete.
Devices that help people record and remember conversations already exist. Apple's entry puts the question into a familiar product, a watch people wear through ordinary life.
Apple announced Live Rewind and Siri Recap on 9 September, with a beta planned for later this year. Live Rewind turns the preceding 15 seconds of speech into text when the wearer double-presses the Digital Crown. Siri Recap produces high-level notes from conversations. This is an announced design, so we cannot yet claim to know how people will behave with it. Apple announcement.
Apple describes meaningful restrictions on raw audio. Secure Exclave isolates audio processing from the operating system and apps. The paired iPhone processes and deletes audio. For Recap, condensed text goes to Private Cloud Compute to produce the final summary. Saved text has separate encryption protections. Apple privacy paper, pp. 4 to 9.
As a security practitioner, I want those restrictions. Local transcription and limits on what reaches the cloud give architects useful ways to reduce exposure. The technical appendix follows those decisions through the design: where models run, what each stage can read and what gets kept when processing succeeds or fails.
The wearer opts in to Recap and can choose schedules and locations. Apple says Recap makes no audible signal. Live Rewind behaves differently. It chimes on activation, even when the watch is silent. That happens after the words it can recover have already been spoken. Apple privacy paper, pp. 7, 9.
The owner has made a choice. What do the people around them understand is happening?
Phillippa Lally and colleagues followed 96 volunteers repeating everyday behaviours for 12 weeks. Repetition in a consistent context was associated with people reporting that the behaviour felt more automatic. The study examined eating, drinking and activity, so applying it to asking permission remains a hypothesis. Lally and colleagues, 2010.
There is also a reason to question whose comfort becomes the default. A 2026 camera-glasses study surveyed 525 people in China and conducted paired interviews with 20 participants. Bystanders wanted stronger protections than wearers were willing to provide, including prior consent and control over sharing. Those are reported expectations in a particular setting, not observed behaviour with Apple's audio features. Wang and colleagues, 2026.
Imagine a conversation over coffee. I expect you to remember some of it. Someone at the next table might overhear a sentence. I may still feel differently about a device routinely turning it into a retained summary.
Helen Nissenbaum's framework of contextual integrity helps explain that distinction. Privacy concerns whether information flows fit the expectations of a setting. The fact that information is available in one situation does not settle every use of it afterward. Privacy as Contextual Integrity.
A summary can also preserve something consequential without keeping a voice recording. Think of a hypothetical note about a colleague considering leaving their job. Apple describes filtering intended to omit sensitive information. Whether that sentence is sensitive depends heavily on who reads it and what they already know.
Meta glasses can retain photos and videos. Apple's described audio pipeline does not retain replayable audio.
Meta describes a capture LED for gallery photos and videos, with measures intended to prevent tampering. That is the vendor's account of a specific capture path, not a guarantee about every camera function. Meta's explanation.
Deliberately concealing capture does not tell us whether ordinary users will stop asking as a habit develops.
The useful comparison is the position of the person nearby. They have to recognise the device and work out what it is doing. The convenience belongs mainly to the wearer. Everyone else has to decide whether to interrupt and ask.
There are good reasons people want these tools. In a recent r/Blind discussion, contributors described the value of assistance that leaves their hands free for a cane. The thread had 56 net votes in the indexed snapshot. Those individual accounts cannot represent a whole community, but they are a reason to take the benefits seriously. Discussion, 10 September.
I don't think privacy is purely a social construct. The engineering matters and laws already constrain capture and use. In Australia, NSW restricts recording private conversations, with exceptions. A separate national right to sue for serious invasions of privacy can reach individuals. NSW government explanation, OAIC guidance.
US federal law generally allows participant interception, subject to exceptions; California has stricter rules for confidential communications. In the EU, the GDPR can govern personal-data processing even without a retained audio file. Consent is one lawful basis and purely personal or household activity is exempt. These frameworks have different scopes and thresholds. None gives us a simple verdict on Apple's design.
The everyday expectation is the part I'm trying to understand here. Asking creates a small pause in which both people can decide what kind of conversation they are having.
If capturing becomes easier and more familiar, does that pause survive? Or does the person who wants it become the one expected to interrupt?
I'm usually comfortable saying yes. I want to understand what happens when we stop treating it as a question.
If you are building something that handles private conversations, there is useful engineering to learn from this design. What can stay on the device? What actually needs to leave it? Who can read the result and for how long?
Those are the decisions I would take into an architecture review. To follow their consequences, keep the hypothetical colleague from the essay in mind. They say, "I might leave my job, but I haven't decided." This is an illustrative test case, not a result from Apple's system.
Apple divides the work across devices. Speech detection happens on the Watch. Transcription and condensation happen on the paired iPhone. Recap sends the reduced text to the cloud for further processing. Basic Rewind stops at local transcription.
That is a useful design to learn from even if we cannot reproduce Apple's silicon. Local models can do more than reduce latency. They can change what another system ever gets to see.
The flows below come from Apple's Audio Intelligence Privacy Overview, especially pages 4 to 9. These are published design claims, not an independent hardware audit. The implementation choices I suggest are my interpretation of what other architects can take from them.
The diagram shows logical processing stages and protected boundaries. It does not specify the processor or accelerator executing each model.
For Recap, the Watch detects nearby speech and buffers audio inside its Secure Exclave. It encrypts the audio for the paired iPhone, where on-device speech recognition produces text. A local language model then condenses the transcript to less than half its original length, followed by safety screening. Condensed text and selected context go to PCC for a title, summary and key points.
Before choosing where to run the model, decide what data each stage needs. A cloud summariser may need text, but that does not mean it needs the microphone stream. A local model can turn a rich input into a narrower representation before a network transfer.
For a meeting assistant, I would compare a full transcript with locally extracted topics or a condensed version. Our colleague's sentence could become "Considering leaving job; undecided." That is shorter, but it still carries the sensitive fact. Local transcription can keep their voice off the network while condensation sends their meaning onward.
Apple also sends context such as calendar information, Now Playing information and coarse location. That is another input to review. Reducing the transcript while attaching unnecessary contextual data can undo part of the benefit.
Dashed paths are optional user actions. The text return path is logical, not a claim about a particular transfer protocol.
Live Rewind sends the preceding 15 seconds to the paired iPhone for on-device transcription. Text returns to the Watch. A later Siri question can send text to PCC, but the basic transcription does not require that cloud step. Here, "local" spans two paired devices. It does not mean every model runs on the Watch.
I would use that distinction when defining a product's minimum useful feature. Could transcription, search or extraction complete on a device, with a separate cloud operation only when the user needs it? The architecture should make that boundary visible and testable.
There are tradeoffs to measure: recognition quality across languages and background noise, latency, battery use, memory and support for older devices. A silent fallback that uploads audio when the local model struggles changes the privacy properties. It needs an explicit product decision, not just an exception handler.
Secure Enclave uses a dedicated security processor to protect cryptographic keys and authentication operations. Secure Exclave isolates the sensor-data processing described here. Apple's Enclave specifications do not establish Exclave's internal processor or memory design.
On the Watch's S11 chip, Apple says the Exclave keeps raw audio inaccessible to watchOS and apps. The paired iPhone has a compatible protected environment. For Recap, an encrypted channel joins the paired Exclaves; audio-verified pairing supplements Bluetooth pairing.
Running a transcription model in an ordinary application process would not reproduce that protection. The operating system, debugging tools or another privileged component may still be able to access its memory.
For my own design, I would name the components that can read raw audio, including operational tooling. Process isolation, restricted permissions and avoiding raw-content logs are practical starting points. A claim that even a compromised operating system cannot read the audio would require a stronger, platform-specific boundary and evidence that it works.
Apple's failure paths are as instructive as its successful transfers.
| Data or operation | Apple's described behaviour | Design decision to carry into a review |
|---|---|---|
| Recap audio transfer | Watch deletes its copy after successful transfer. Failed transfers can retry until keys expire; the timeout is not published. | Bound retry queues by time and size. Specify what happens when the limit is reached. |
| iPhone audio processing | Raw audio is deleted after processing. | Give raw inputs a lifetime separate from derived text. Check temporary files and diagnostics too. |
| Rewind without a reachable iPhone | Transfer fails and audio is discarded immediately. | Decide whether failure discards data or retains it for later. These are different products. |
| Rewind text | Disappears 30 seconds after the display dims unless saved. | Separate temporary display from intentional retention. |
| Recap text | Normally expires after seven days unless saved. | Define defaults for outputs, not only inputs. |
| Optional Apple Intelligence Report | Authenticated local access can expose PCC request information for 15 minutes or seven days. | Include diagnostic copies in the retention model. |
In our example, deleting the raw audio would not delete a saved note about the colleague's possible departure. That note needs its own retention rule. I would test both lifetimes during disconnection, process crashes and retries. A retention promise needs to survive failure. Expiring a key can make encrypted data unusable, but it is not evidence that every physical memory location was overwritten.
PCC processes readable input inside its protected environment. Apple's design combines restrictions on operator access with software attestation and transparency mechanisms. Attestation gives a client evidence about the software at the receiving endpoint before sending a request. Apple's PCC architecture.
I would inspect the whole route to inference: where encryption terminates, who can inspect requests and whether logs or debugging systems can retain them. Transport encryption alone does not answer those questions. Using an ordinary cloud model API does not reproduce PCC's guarantees.
iCloud syncing protects a different stage. Apple describes end-to-end encrypted syncing of saved text when the account has two-factor authentication and the device has a passcode. This concerns retained outputs across devices, rather than the cloud's ability to compute over input.
For an architect, those belong in separate rows of the design: processing access, transport protection and stored-output access. A single "encrypted" label hides too much.
Local transcription can keep raw audio out of a cloud service. Local condensation can reduce how much text crosses that boundary. Neither establishes that the surviving text is harmless.
A useful evaluation would measure transcription errors, lost qualifications and sensitive details retained in summaries, alongside latency and resource use. For the colleague's sentence, check whether the summary preserves "might" and "haven't decided". A version that says "I am leaving" changes the meaning, even if every transfer was secure. Evaluation should include the languages and environments the product actually serves. Apple's paper describes sensitive-information filtering but supplies no measured false-negative rate for it.
The hardware protections are Apple-specific. The questions about model placement, transfer and retention belong in our own architecture reviews too. Following one sentence through the system makes those decisions concrete. It also brings us back to the person who said it. Did they expect a note about leaving their job to survive the conversation?
Ready to assess how advanced AI properties might impact your organization? to discuss your specific deployment context and governance needs.
