# Trillboards Sensing SDK — Android Integration Brief (agent-executable) > **This is NOT the ad-monetization (IMA) guide.** This is the **camera + microphone + Wi-Fi + Bluetooth audience-SENSING** SDK — the same `agent-core` engine that powers Trillboards' own retail screens. It produces audience **Moments, scenes, demographics, and custom chips** in the Trillboards Venue portal. Ad serving is a separate, optional SDK. > > **How to use this doc:** paste it into Claude Code / Cursor inside your Android app's repo. The AI agent should follow it top-to-bottom and stop only at the **Done criteria** at the end. Everything needed is here. --- ## What you're integrating You are embedding the **Trillboards Sensing SDK** (`com.trillboards:agent-core`, from GitHub Packages) into an **existing Android app** that runs on in-store screens/boxes. After integration, each box's camera/mic/Wi-Fi/BLE feed on-device AI (face, age/gender, emotion, pose, audio, speech, BLE/Wi-Fi co-viewing), and the resulting **audience intelligence attributes to that box's screen** in the Trillboards Venue portal — automatically. No ads, no kiosk takeover; the SDK runs as a library inside the host app. ## Prerequisites (you already have these) 1. **Your devices are already registered** with Trillboards (one row per box) via authenticated `POST /v1/partner/device`, each with a stable **`external_id`** you chose (e.g. a serial like `DD-H80F55`). That `external_id` is the **only per-box value the SDK needs at runtime.** If a box is not yet registered, register it first (see Appendix A). 2. **Your Trillboards Partner API key** (format `trb_partner_…` or `tb_prt_…`) is used only for provisioning actions such as registration. Do not bundle it in the device app for heartbeat, identity, or sensing startup. > The SDK passes your `external_id` to the backend, which already knows which screen+venue that box maps to (from registration). That is what makes the data attribute to the right location. --- ## Step 1 — Add the dependency In **`settings.gradle.kts`** (or root `build.gradle`), add the canonical public Trillboards Maven repository: ```kotlin dependencyResolutionManagement { repositories { google() mavenCentral() maven { url = uri("https://maven.trillboards.com") } } } ``` The repository is anonymous-read. No GitHub token or Trillboards API key belongs in Gradle credentials. In your **app module `build.gradle.kts`**: ```kotlin dependencies { implementation("com.trillboards:agent-core:1.14.0") // Moonshine ASR runtime — enables on-device speech→structured output (FunctionGemma 270M). // agent-core ships ASR compileOnly; this supplies the runtime. Omit only if you don't want speech. implementation("com.trillboards.bundled:sherpa-onnx:1.12.26") } ``` **Build floor (required):** `minSdk = 26`, Android Gradle Plugin `8.0+`, JDK 17. The AAR is ~180 MB (bundled on-device models). If your app is below these, bump them. **Release candidates.** A new version is first published to the staging repository, and it becomes the public `latest` only after that exact build passes a 48 h soak on an Ember-class tablet. To build a candidate before then, add the staging repository and pin the exact version: ```kotlin maven { url = uri("https://maven.trillboards.com/staging") } // app module: implementation("com.trillboards:agent-core:") ``` Staging has no `latest`, so a dynamic version (`1.+`) never resolves to a candidate. A staged version never changes, and the public release is the same bytes: what you tested as the candidate is what ships. The version directory's `build-info.json` lists the SHA-256 of the AAR and the POM. Remove the staging line once the version is public. ## Step 2 — Permissions + foreground service In **`AndroidManifest.xml`**, declare (inside ``): ```xml ``` Grant the runtime permissions (`CAMERA`, `RECORD_AUDIO`, `ACCESS_FINE_LOCATION`, `BLUETOOTH_SCAN`, and — on Android 13+ / API 33+ — `NEARBY_WIFI_DEVICES`) at app start (or via MDM auto-grant on managed fleets). `BLUETOOTH_SCAN` and `NEARBY_WIFI_DEVICES` are runtime-**dangerous**: declaring them (or letting the AAR merge them in) is not a grant, and skipping either request leaves that radio silently reporting nothing while the other looks healthy. WiFi scanning additionally needs `ACCESS_FINE_LOCATION` at **every** API level — on 33+ it is required alongside `NEARBY_WIFI_DEVICES`, because `WifiManager.startScan()`/`getScanResults()` are the carve-out `NEARBY_WIFI_DEVICES` does not cover — and the AAR caps its own declaration at `maxSdkVersion="30"`, so declare an uncapped one yourself on API 31+. The per-API-level `requestPermissions` snippet is in the [CTV Measurement SDK — Partner Integration Guide § Step 1b](https://api.trillboards.com/docs/integrations/ctv-measurement-partner-integration.md). **Foreground service:** the SDK binds the camera to the app process lifecycle. If your screen app is **always in the foreground** (kiosk/always-on display), no extra work is needed. If you sense while the app is **backgrounded**, declare a foreground service with `foregroundServiceType="camera|microphone"` and call `startForeground()` before sensing — otherwise Android stops the camera in the background. ## Step 3 — Start the SDK Call this once at app start (e.g. `Application.onCreate()` or your main Activity), passing **this box's `external_id`** as `deviceId`: ```kotlin import com.trillboards.sdk.SensingSdkConfig import com.trillboards.sdk.TrillboardsSensingSdk TrillboardsSensingSdk.start( context = applicationContext, config = SensingSdkConfig( deviceId = "DD-H80F55" // ← THIS box's external_id (the one you registered) // speech/ASR + on-device FunctionGemma 270M run by default (sherpa-onnx dep added in Step 1) ) ) ``` That's the whole integration. The SDK resolves this box's screen from `deviceId` (`check-screen`), handles any privileged sensing-channel credential internally, opens the camera/mic, runs on-device AI, and emits attributed observations. Heartbeat liveness and targeting-identity ingestion are unauthenticated after registration. To stop: `TrillboardsSensingSdk.stop(applicationContext)`. > **`deviceId` MUST equal the `external_id` you registered for this box.** That's the entire attribution contract. If you templatize the app across a fleet, inject each box's `external_id` at provisioning time (build config, MDM app-config, or a per-device file). --- ## Step 4 — Verify it worked (do this, don't skip) 1. Run the app on a real device with a camera in view of people (or yourself). 2. In **logcat**, confirm: `TrillboardsSensingSdk: Resolved screen binding: screenId=…` and `FrameCaptureManager: ... SENDING TO API ... screenId=`. 3. In the **Trillboards Venue portal** (`/earner`), open that box's screen → the **Moments** card populates within a few minutes (scenes, demographics, custom chips). That is success. ## What you get (in the portal + via the partner API) - **Moments / scenes** — Gemini scene descriptions per capture. - **Demographics** — age range, gender, group composition (age/gender model downloads ~178 MB on first run; until then those fields are empty, everything else works). - **Custom chips** — fitness archetype, dietary signal, attire, lifestyle, brands detected, ad-fit categories (IAB + Trillboards taxonomy). - **Audio → structured insights** — on-device **FunctionGemma 270M** turns speech (via Moonshine ASR) into structured retail-conversation signals (brand mentions, purchase-journey stage, sentiment, objections…). Needs the sherpa-onnx dep (Step 1); see "On-device speech LLM" below. - **Co-viewing / proximity (BLE + Wi-Fi)** — produced by the **co-installed Measurement SDK** (Step 5), not this SDK. The Sensing SDK is camera + audio. - Same data is readable via the partner API audience/moments endpoints. ## Step 5 — Proximity graph: co-install the Measurement SDK (BLE/Wi-Fi) The Sensing SDK is camera + audio. **Wi-Fi/Bluetooth proximity** (the co-viewing graph) is produced by the **CTV Measurement SDK**, which runs the scans on its **own scheduler** — so proximity never weighs down the camera/audio path. To capture the full sensor suite, co-install it with the **same `deviceId`** so both attribute to the same screen: ```kotlin // app/build.gradle.kts (same GitHub Packages repo as agent-core, from Step 1): dependencies { implementation("com.trillboards:ctv-measurement:1.3.2") } ``` ```kotlin import com.trillboards.measurement.MeasurementConfig import com.trillboards.measurement.TrillboardsMeasurement TrillboardsMeasurement.initialize( applicationContext, MeasurementConfig.Builder(deviceId = "DD-H80F55").build() ) TrillboardsMeasurement.setConsentStatus(true) TrillboardsMeasurement.startScheduledScans() ``` This SDK also reports the box's hardware (incl. RAM) and, when the platform exposes it, the real resettable GAID/AAID on its heartbeat. It never substitutes the registered device or screen ID for a MAID. The hardware report is what lets the backend serve the FunctionGemma 270M model to capable (≥ 2 GB) boxes. **Camera-only?** Skip this step. **Full suite (proximity + on-device speech LLM)?** Co-install it. Both SDKs expose an `ExtendedIdentityProvider` for host-owned, exact targeting identities. Supply an already-computed SHA-256 email digest as `hem_sha256` or a current opaque UID2 advertising token as `uid2_token`. HEM is lowercased once and never re-hashed; UID2 remains case-sensitive and should carry its expiry. These values stay separate from MAID and leave the backend as OpenRTB `user.eids`; MAID leaves as `device.ifa`. ## Step 6 — Report your own plays, trips and taps (optional, agent-core 1.22+) Three calls tell Trillboards what YOUR app owns, so the audience the SDK measured is attributed to it: the ads your app plays, your taxi meter, and what riders do with the screen. They do nothing unless you call them. ```kotlin import com.trillboards.sdk.HostInteractionType import com.trillboards.sdk.TrillboardsSensingSdk // An ad your app played. creativeRef is the content hash POST /v1/partner/creatives returned. TrillboardsSensingSdk.reportPlay( playId = "play-7f3a", // your unique id for this play creativeRef = "sha256:6c4b8f3e2a1d7b90c5e42f18a3d6b7e90124fa5c83b16d0e7f294a5c81b3d6e1", completedAtEpochMs = System.currentTimeMillis(), durationSeconds = 15.0, impressionId = null, // ONLY for an ad Trillboards served (see below) ) // The meter: number the changes of one trip 0, 1, 2... TrillboardsSensingSdk.setTripState(tripId = "trip-9", eventSeq = 0, meterOn = true, stateAtEpochMs = startedAt) TrillboardsSensingSdk.setTripState( tripId = "trip-9", eventSeq = 1, meterOn = false, stateAtEpochMs = endedAt, fareCents = 2450, tipCents = 400, paymentType = "card", passengerCount = 2, destination = "JFK T4", ) // What a rider did: TAP, QR_SHOWN, QR_SCANNED or DWELL_PROMPT. TrillboardsSensingSdk.reportInteraction("tap-1", HostInteractionType.TAP, tappedAt, creativeId = "cr-9", tMs = 8000) ``` From Java they are static methods with short overloads: `TrillboardsSensingSdk.reportPlay("play-7f3a", creativeRef, System.currentTimeMillis());`, `TrillboardsSensingSdk.setTripState("trip-9", 0L, true, startedAt);`, `TrillboardsSensingSdk.reportInteraction("tap-1", HostInteractionType.TAP, tappedAt);`. How they behave: - **Each call returns at once and never throws.** `true` means the event was queued for delivery; `false` means it was refused as invalid, with the reason in logcat under `TrillboardsHostHooks`. Bounds: `playId` 1–256 characters, `tripId` and `interactionId` 1–128, `creativeRef` a `sha256:<64 hex>` content hash, `durationSeconds` 0–86,400, `eventSeq`, fares and `tMs` 0–4,294,967,295, `passengerCount` 0–255. - **Delivery is in the background**, to `POST /v1/partner/device//{plays,trips,interactions}` with this device's own credential (never your partner API key). The screen is the device's binding; you never pass it. - **Offline-safe.** Events are written to a queue on the device and sent once the device is online and bound to its screen, including after an app restart. Calls made before `start()` are held in memory (up to 256) and queued when `start()` runs. The queue keeps at most 2,160 events, each for at most 72 hours; past either bound the oldest are dropped first. - **Retries.** A network error, a 5xx, a 429 (honouring `Retry-After`), a 401 or a 404 keeps the event and retries with exponential backoff from 2 s to 5 minutes. A 400 or 403 drops it (the same event can never succeed; the reason is logged). A 409 means the server already received that request. - **Idempotent.** Every retry resends the same body, and the server keeps one play per `playId`, one meter event per (`tripId`, `eventSeq`) and one interaction per `interactionId`. To retry your own call, reuse the id; for a new event, use a new id. - **`impressionId` is only for ads Trillboards served** (for example through the Trillboards VAST tag): pass that ad's Trillboards impression id so the play is counted once. Leave it `null` for every ad your app sold or scheduled itself. ## Step 7 — Seatbelt reminders (optional, agent-core 1.24+) The SDK tells your app when the rear riders are unbuckled while the cab moves, so your app can show a reminder. Set a listener; nothing is delivered without one. ```kotlin import com.trillboards.sdk.HostInteractionType import com.trillboards.sdk.TrillboardsSensingSdk TrillboardsSensingSdk.setSafetyEventListener { reminder -> showSeatbeltReminder(reminder.seatSide) // your UI TrillboardsSensingSdk.reportInteraction( reminder.eventId, HostInteractionType.SEATBELT_REMINDER_SHOWN, System.currentTimeMillis(), ) } ``` From Java: `TrillboardsSensingSdk.setSafetyEventListener(reminder -> { ... });`. Pass `null` to remove the listener. How it behaves: - **When a reminder fires.** Trillboards switches the seatbelt check and its reminders on per screen, from the server. A reminder fires once the rear riders have been unbuckled for 20 seconds while the cab moves, at most 3 times per trip and at least 5 minutes apart (server settings). - **`SeatbeltReminder`** carries `tripId` (your id while a trip you opened with `setTripState(…, meterOn = true, …)` is open; otherwise the SDK's camera trip, `cab:`), `seatSide` (the rear seat, facing forward in the vehicle: `left`, `right`, `center`, `multiple` when riders on different sides are unbuckled, or `unknown`) and `eventId` (unique per reminder, starting `:`). - **Report each reminder you show**, with its `eventId` as the interaction id. A retry with the same id is recorded once. Report only reminders that were on screen: the reported count is how the reminder's effect is measured. - **Trips.** Call `setTripState` (Step 6) with `meterOn = true` when a passenger boards and `meterOn = false` when the trip ends. Reminders then count per your trip; without it, the SDK finds trips from the camera. Keep a `tripId` to 112 characters, so the reminder's `eventId` (`:` and up to 11 more) fits the 128-character interaction id. - The callback runs on the main thread, before or after `start()`. An exception thrown in it is logged under `TrillboardsSafety` and ignored. - `SeatbeltReminder` has a public constructor: to test your UI, call your listener with `SeatbeltReminder("trip-test", "left", "trip-test:1")`. ## On-device speech LLM (FunctionGemma 270M) - ASR (Moonshine) runs by default once the sherpa-onnx dep (Step 1) is present; it transcribes **on-device**. From agent-core 1.21 the transcript text and its segment times are sent to Trillboards and stored; no audio recording leaves the box. - The transcript feeds **FunctionGemma 270M** (271 MB `.litert-lm`, CPU-only, ~216 MB resident) which emits structured conversation insights. It **OTA-downloads** on first run when the device is known to have ≥ 2 GB RAM — which the co-installed Measurement SDK reports (Step 5). On a 3.5 GB box it runs comfortably. ## Performance notes (low-/mid-tier boxes, e.g. Snapdragon 695 / 3.5 GB) - Expect ~3 fps sensing, ~175 MB native heap, stable — sufficient (the backend aggregates over 10s windows). - The on-device **vision** VLM (8 GB+) stays **off** by design; the **audio** FunctionGemma 270M (~216 MB) runs on ≥ 2 GB boxes. ## Troubleshooting - **Logcat shows `No screen bound to fingerprint yet`** → `deviceId` doesn't match a registered `external_id`. Verify the box is registered (Appendix A) and the `external_id` matches exactly. - **Data in PG but `screen_mongo_id` null / not in the portal** → you're on an SDK older than `1.1.2`. Pin `1.1.2`. - **App crashes on launch with `MobileAdsInitProvider` / "Missing application ID"** → you're on an SDK older than `1.1.2` (ads were decoupled in this line). Pin `1.1.2`. - **No demographics** → the ~178 MB age/gender model is still downloading; everything else works meanwhile. - **No audio/speech insights** → confirm the `sherpa-onnx` dep (Step 1) is present (logcat `hasSpeechModel=true`, ASR transcribing). FunctionGemma 270M also requires the device to report **≥ 2 GB RAM** — co-install the Measurement SDK (Step 5) so its heartbeat reports hardware; without a known RAM tier the model isn't served. - **No proximity / co-viewing** → that's the Measurement SDK (Step 5), not this SDK. Confirm it's co-installed with the **same `deviceId`**. --- ## Appendix A — register a box (only if not already registered) ```bash curl -X POST https://api.trillboards.com/v1/partner/device \ -H "Authorization: Bearer trb_partner_YOUR_KEY" \ -H "Content-Type: application/json" \ -d '{"device_id":"DD-H80F55","name":"Store #123 — Front Counter","location":"123 Main St, Dallas, TX"}' ``` The response binds the box to a screen+venue. Send the registration `device_id` and use that **same identifier** as `deviceId` in Step 3. The stored binding remains `external_id`. (Bulk: loop your CSV of boxes.) ## Done criteria (stop here when all true) - [ ] App builds with `com.trillboards:agent-core:1.14.0` from the anonymous public Maven repository, minSdk 26 / AGP 8+ / JDK 17. - [ ] Manifest has camera/mic/Wi-Fi/BLE perms (+ FGS if backgrounded); runtime perms granted. - [ ] `TrillboardsSensingSdk.start(...)` is called once with `deviceId = this box's external_id`. - [ ] No Partner API key is embedded in the runtime app; registered-device heartbeat and identity POSTs succeed without one. - [ ] Logcat shows `Resolved screen binding: screenId=…` and POSTs with a non-null `screenId`. - [ ] The box's screen shows Moments/chips in the Venue portal. - [ ] **(Speech)** `sherpa-onnx` dep present → logcat shows ASR transcribing + FunctionGemma 270M loaded after OTA; audio insight chips appear. - [ ] **(Proximity)** Measurement SDK (Step 5) co-installed with the **same `deviceId`** → BLE/Wi-Fi co-viewing appears for the screen.