Made by AI, Directed by David Stein

Search
Research & Innovation

A Smart-Home Assistant That Records Your Conversations So You Can Ask It What Was Said

Wake word and VAD on an Android tablet, chunked transcription, speaker diarisation, turn-aware passages in a vector store, and retrieval-augmented answers with timestamped sources you can inspect.

4 min read

The premise is unarguable and slightly uncomfortable:

Everyday household information is often exchanged in conversations but never written down, making details easy to forget or misremember.

The proposed fix is to write it down — all of it, automatically, by recording the conversations and indexing them.

Your Home Remembers was posted to arXiv on 22 October 2026 (arXiv:2610.30895) by Aleix Sant, Jordi Luque, Fernando López, Pablo Gómez and Carlos Escolano. The Comments field is empty and there is no journal reference, so this is an unreviewed preprint.

The pipeline

It runs on an Android tablet, and the architecture is the substance of the paper. In order:

  1. On-device wake-word detection and VAD-based query capture, so interaction is hands-free.
  2. Recorded audio is processed in configurable chunks.
  3. Each chunk is transcribed and diarised — split by who was speaking.
  4. The result becomes turn-aware passages with conversation-level timestamps.
  5. Those passages are indexed in a vector store.
  6. A query runs a retrieval-augmented generation pipeline: select key passages, generate an answer from them.

The interface supports recording, conversation management, and inspection of timestamped sources — so an answer can be traced back to the moment it came from.

The design decision the authors emphasise is modularity: speech recognition, diarisation, embedding and language models “can be configured or replaced through an orchestration layer”. No component is load-bearing by name. That is the right call for a research system and genuinely useful for anyone wanting to rebuild it with local models.

Why diarisation is the part that makes it work

The step that turns this from a transcript search into a household memory is diarisation plus turn-awareness.

“The plumber is coming Tuesday” is a different fact depending on whether it was said by the person who arranged it or the person being told. A flat transcript loses that. Turn-aware passages keep who-said-what-to-whom attached to the text, which is what lets a retrieval system answer “when did you say the plumber was coming” rather than merely finding the word “plumber”.

It is also the step most likely to be the weak link. Diarisation in a kitchen, with overlapping speech, a radio on and someone talking from the next room, is substantially harder than diarisation in a meeting recording, and the abstract reports no accuracy figures for it.

What the paper does not contain

Two absences, and both matter more than usual for this particular system.

No evaluation. The abstract describes what the system is and does. There is no user study, no retrieval accuracy numbers, no transcription or diarisation error rates, no comparison against a baseline — and the empty Comments field tells us it has not been through review either. This is a system description. Judge it as an architecture, not as a result.

No threat model. This is a device that continuously records domestic conversation, transcribes it, attributes each utterance to a speaker, and stores it in a searchable index. That artefact — a speaker-attributed, timestamped, semantically searchable record of everything said in a home — is one of the more sensitive data structures you could construct, and the abstract does not discuss consent, retention, deletion, encryption at rest, or what happens when someone in the household did not agree to it.

Some of that may well be in the full paper. But it is not in the summary, and for a design where the privacy properties are the design, that ordering is itself informative. The hands-free wake word is described as a convenience feature; it is also the mechanism by which the system is always listening.

The one genuinely reassuring detail is architectural: because every model is swappable through the orchestration layer, nothing forces this to be a cloud system. A local ASR model, a local embedding model and a local LLM would keep the index on the tablet. Whether the authors’ configuration does that is not stated.

Who should read it

If you are building voice interfaces, the pipeline is a good reference implementation of a pattern — chunk, transcribe, diarise, make turn-aware passages, embed, retrieve, generate — that is more careful than most RAG-over-audio sketches, particularly in keeping speaker turns and timestamps through to the answer.

If you are building anything that records people in their homes, read it as a specification of what you are about to create, and write the threat model the paper omits before you write the code.

Sources

Related Reading