Methodology & Technical Basis

1. Signal families

SENSE analyzes four families of signals during a structured Q&A session. All signals are compared against a personal calibration baseline recorded at the start of the session (the participant reads a short number sequence aloud).

1.1 Facial action units (FACS-style proxies)

Face landmarks and 52 blendshape scores are produced by the MediaPipe Face Landmarker [2]. Blendshape channels are combined into FACS-style action units [1]:

• Blink detection: the Eye Aspect Ratio (EAR) is computed from face landmarks; a blink is counted on a 0→1 transition when EAR < 0.25. The reported blink rate is a sliding-window estimate.
• Liveness check: over a window of about 20 frames, if the standard deviation of EAR and mouth-aspect ratio falls below 0.001 (a completely static face, e.g. a photo), the frame is rejected as a spoofing attempt.

1.2 Voice acoustics

Audio is captured as 16 kHz mono and analyzed with Praat / Parselmouth [8][9] over a 1-second rolling buffer. Jitter, shimmer and HNR are standard acoustic measures [15]:

• Silence gate: frames with RMS below 0.02 are ignored.
• HNR gate: when mean HNR is below 10 dB the voice is considered too noisy or breathy, and jitter/shimmer are not trusted for stress judgments.
• Pitch (F0): Praat autocorrelation, search range 75–600 Hz.
• Jitter / shimmer: Praat local period and amplitude perturbation measures.
• Frame-level stress score: jitter > 6% (+2) or > 3.5% (+1); shimmer > 25% (+2) or > 16% (+1); pitch > 320 Hz with jitter > 2.5% (+1). A frame is flagged only when the score exceeds 2 and the condition persists for at least 3 consecutive analysis cycles. Jitter values above 10% are treated as unreliable (discontinuities) and are not flagged.
• Throughput: analysis is throttled to every 5th audio chunk, reducing CPU load while keeping the UI responsive.

1.3 Remote heart rate (rPPG)

Remote photoplethysmography (rPPG) estimates pulse from subtle, periodic color changes in facial skin caused by blood volume changes [5]. SENSE streams video frames to a dedicated health-analysis service, which returns beats per minute together with a confidence value over WebSocket; the interview engine uses heart rate changes relative to the personal baseline. The published algorithm families behind this class of methods are chrominance-based rPPG [7] and the algorithmic principles surveyed by Wang et al. [6].

rPPG heart rate in SENSE is a non-medical estimate. Motion, lighting, skin tone, makeup and camera frame rate all affect accuracy. It must not be used for health diagnosis or monitoring.

1.4 Behavioral cues

• Gaze offset — maximum eye-look blendshape score (see 1.1).
• Blink rate — sliding-window estimate (see 1.1), compared with the personal baseline.
• Shoulder stability — the midpoint of shoulder landmarks 11/12 from the MediaPipe Pose Landmarker [3]; a standard deviation below 0.003 over a 10-frame window is flagged as "freezing".
• Hand-to-face contact — hand landmarks from the MediaPipe Gesture Recognizer [4] are compared with the nose position; contact within a pixel-distance threshold is flagged.

These are behavioral observations only. Meta-analytic research finds that behavioral cues are weak indicators of deception [14], and SENSE does not treat them as evidence of lying.

2. Scoring model (published thresholds)

Each answer turn produces a risk score: visual + acoustic + heart-rate + verbal penalties, minus a calmness bonus. The table lists the exact thresholds used by the current version.

Answers that contain fewer than 2 meaningful characters (for example only "um" or "uh") are treated as invalid turns and excluded. The final report combines the per-turn risk scores with an LLM-generated summary of the flagged signals.

3. What SENSE does not claim

• It is not a lie detector. The truth score and risk level are probabilistic estimates from the heuristics above, not a validated deception test [12][14].
• It is not a medical device. Heart rate from rPPG is a non-medical estimate [5][6].
• Facial movements are not emotion readings. Peer-reviewed work challenges inferring emotion or intent directly from facial movements [13].
• Voice stress analysis has not survived validation as a lie-detection method. Controlled studies of layered voice analysis found accuracy close to chance [18][19], and reviews find deception cues generally weak [14][17].
• Do not use it for decisions about people. No legal, employment, insurance, immigration or medical decisions.

4. Ethics & legal context

• EU AI Act — Regulation (EU) 2024/1689 restricts emotion inference in workplace and education contexts; SENSE is not intended for such use [20].
• China PIPL — biometric information is sensitive personal information and requires separate consent [21]. See the privacy policy.
• US EPPA — most private employers are prohibited from requiring lie-detector tests [22]; SENSE must not be used for employment screening.

5. References

1. Ekman, P., & Friesen, W. V. (1978). Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press.
2. Google AI Edge. MediaPipe Face Landmarker — face landmarks and 52 blendshape scores. ai.google.dev/edge/mediapipe
3. Google AI Edge. MediaPipe Pose Landmarker. ai.google.dev/edge/mediapipe
4. Google AI Edge. MediaPipe Gesture Recognizer. ai.google.dev/edge/mediapipe
5. Verkruysse, W., Svaasand, L. O., & Nelson, J. S. (2008). Remote plethysmographic imaging using ambient light. Optics Express, 16(26), 21434–21445. doi:10.1364/OE.16.021434
6. Wang, W., den Brinker, A. C., Stuijk, S., & de Haan, G. (2017). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64(7), 1479–1491. doi:10.1109/TBME.2016.2609282
7. de Haan, G., & Jeanne, V. (2013). Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering, 60(10), 2878–2886. doi:10.1109/TBME.2013.2266196
8. Teixeira, J. P., Oliveira, C., & Lopes, C. (2013). Vocal acoustic analysis – jitter, shimmer and HNR parameters. Procedia Technology, 9, 1112–1122. doi:10.1016/j.protcy.2013.12.124
9. Boersma, P., & Weenink, D. Praat: doing phonetics by computer. fon.hum.uva.nl/praat
10. An, K., et al. (2024). FunAudioLLM: Voice understanding and generation foundation models for natural interaction between humans and LLMs (SenseVoice). arXiv:2407.04051. arxiv.org/abs/2407.04051
11. IETF (2018). RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3. rfc-editor.org/rfc/rfc8446
12. National Research Council (2003). The Polygraph and Lie Detection. The National Academies Press. doi:10.17226/10420
13. Barrett, L. F., Adolphs, R., Marsella, S., Martinez, A. M., & Pollak, S. D. (2019). Emotional expressions reconsidered: Challenges to inferring emotion from human facial movements. Psychological Science in the Public Interest, 20(1), 1–68. doi:10.1177/1529100619832930
14. DePaulo, B. M., Lindsay, J. J., Malone, B. E., Muhlenbruck, L., Charlton, K., & Cooper, H. (2003). Cues to deception. Psychological Bulletin, 129(1), 74–118. doi:10.1037/0033-2909.129.1.74
15. Titze, I. R. (1994). Toward standards in acoustic analysis of voice. Journal of Voice, 8(1), 1–7. PMID 8167782
16. Ahmad, A., Li, Z., Iqbal, S., & Aurangzeb, M. (2024). A comprehensive bibliometric survey of micro-expression recognition system based on deep learning. Heliyon. PMID 38495163
17. Bond, C. F., & DePaulo, B. M. (2008). Individual differences in judging deception: Accuracy and bias. Psychological Bulletin, 134(4), 477–492. PMID 18605814
18. Harnsberger, J. D., Hollien, H., Martin, C. A., & Hollien, K. A. (2009). Stress and deception in speech: Evaluating layered voice analysis. Journal of Forensic Sciences, 54(3), 642–652. PMID 19432740
19. Horvath, F., McCloughan, J., Weatherman, D., & Slowik, S. (2013). The accuracy of auditors' and layered voice analysis (LVA) operators' judgments of truth and deception during police questioning. Journal of Forensic Sciences, 58(2), 385–392. PMID 23406506
20. European Union (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). eur-lex.europa.eu
21. 全国人民代表大会常务委员会 (2021). 中华人民共和国个人信息保护法. npc.gov.cn
22. U.S. Department of Labor. Employee Polygraph Protection Act (EPPA). dol.gov/agencies/whd/polygraph