Cloudagotchi Part 2 : Giving it a face: Sprites and touch on a 1.8" screen

In part 1, we gave our virtual pet a nervous system: the ESP32-S3 now speaks mutual-TLS MQTT with AWS IoT Core, and we can chat with it from the AWS console. Very satisfying. Also very much a black screen with a serial log, a pet only a backend developer could love.

Today we fix that. By the end of this article, a face blinks at you from the screen: two soft eyes and an orange smile that looks suspiciously familiar if you've ever received a package (or used AWS ^^). Tap it and it beams. Shake the board and it goes wide-eyed. Ignore it for two minutes and it dozes off with a little zzz. And every one of those interactions is published to the cloud, ready for the brain we'll build in part 3.

In this article, I will walk you through building a sprite-based pet face with LVGL plus touch input, a shake detector in ~30 lines of IMU code, and the very honest story of why I ended up with sprites after my first approach fell apart on real hardware.

⚠️ Reality check (sorry 😅): my first version of this pet was "elegant" a pet drawn procedurally from LVGL arcs and rounded rectangles, animated at 30 fps. It looked great in my head and became a corrupted mess of color bands on the actual panel. This article teaches the approach that survived contact with the hardware. The detours are documented in the ⚠️ boxes, because they'll save you an afternoon each.

The code for this article is on the article-2 branch:

git clone https://github.com/tagazok/cloudagotchi && cd cloudagotchi
git checkout article-2

The board does the hard part: the BSP

Getting a display, touch controller, and LVGL configured from scratch on embedded hardware is normally an afternoon of datasheets and SPI timing tantrums, even more for me who is a JavaScript developer 😅. The reason I picked ESP-IDF over Arduino in part 1 pays off right here: Waveshare publishes an official Board Support Package (BSP) for this exact board as a managed component. One line in idf_component.yml:

dependencies:
  waveshare/esp32_s3_touch_amoled_1_8: '^2.0.3'

...and one function call:

#include "bsp/esp32_s3_touch_amoled_1_8.h"

bsp_display_start();   // AMOLED on, touch on, LVGL initialized.

Behind that call, the BSP configures the CO5300 display driver over QSPI, the touch controller over I2C, and starts an LVGL task with display flushing hooked up.

Also, know this quirk: flashing leaves the panel dark. The panel keeps running during idf.py flash with nobody driving it. If you also have this problem after flashing, reset the board (Ctrl+T Ctrl+R in the monitor) and it comes up clean.

One more rule to remember, because it will bite you exactly once: LVGL isn't thread-safe. The BSP runs LVGL in its own task, so any other task touching the UI must take the lock:

bsp_display_lock(0);      // 1. take the LVGL mutex (0 = wait forever)
lv_bar_set_value(...);    // 2. mutate the UI
bsp_display_unlock();     // 3. give it back

Forget it and you get the embedded classic: a crash, or worse, a silent no-op.

Why sprites, not shapes: a war story in one paragraph

My first face was procedural: the head an lv_obj with a big corner radius, the mouth an lv_arc, everything animated by LVGL's animation engine: bobbing, blinking, squishing. Beautiful in theory. On hardware, the panel's QSPI transaction queue would overflow under load (Wi-Fi bring-up, TLS downloads) and silently drop color transfers. Animated widgets self-heal, they redraw every frame. But each dropped chunk of a static widget stays on screen as a garbage band forever. The more the UI animated, the more the queue overflowed, the worse the corruption. I tried buffer tuning, cache settings, custom DMA allocations; some attempts made it worse in spectacular new ways.

The approach that survived: make the display's job trivial. Pre-render every expression as a sprite baked into flash. The face is ONE static image widget; changing expression = swapping which sprite is shown = one clean rectangular blit. A blink, a mood change, a talking mouth, all the same cheap operation. Nothing redraws unless an event happens, so nothing can outrun the display.

// The face: ONE static image widget, centered. That's the whole pet.
s_face = lv_image_create(screen);
lv_image_set_src(s_face, &face_neutral);
lv_obj_align(s_face, LV_ALIGN_CENTER, 0, 20);

The sprite pipeline: from PNG to C array

The art is a set of PNGs, one per expression (neutral, happy, sad, sleeping, three talking mouths, plus per-mood blink frames). I generated mine with an AI image generator and composited them onto a common canvas with identical eye positions so frames don't jitter when swapped.

Each PNG becomes a C file via a small Python script (in the repo): pixels converted to RGB565A8, 16-bit color plus an 8-bit alpha plane, LVGL 9's friendliest transparent format, and wrapped in an lv_image_dsc_t:

const lv_image_dsc_t face_happy = {
  .header = { .magic = LV_IMAGE_HEADER_MAGIC, .cf = LV_COLOR_FORMAT_RGB565A8,
              .w = 210, .h = 162, .stride = 420 },
  .data_size = sizeof(happy_map),
  .data = happy_map,   // lives in flash, memory-mapped, zero RAM cost
};

Seven faces at ~100 KB each sounds heavy, but they compile into the 16 MB flash and LVGL blits them straight from there.

Blinking, by image swap

No animation engine needed, a blink is two sprite swaps on a timer:

static void blink_timer_cb(lv_timer_t *t)
{
    if (s_talking || s_dozing || s_mood == PET_MOOD_SLEEPING) return;
    // Dedicated blink frame: closed eyes, SAME mouth as the current mood —
    // a blink should only move the eyes.
    switch (s_mood) {
    case PET_MOOD_HAPPY: set_face(&face_blink_happy);  break;
    case PET_MOOD_SAD:   set_face(&face_blink_sad);    break;
    default:             set_face(&face_blink_neutral); break;
    }
    lv_timer_create(blink_close_cb, 120, NULL);  // reopen after 120 ms
}

The doze: a pet that gets bored

Leave the pet alone for two minutes and it falls asleep (the sleeping sprite, complete with a little zzz). Touch it or shake it and it wakes. One LVGL timer plus a reset-on-activity:

// 2 minutes with no interaction → the pet nods off.
static void idle_timer_cb(lv_timer_t *t)
{
    if (s_talking) return;      // reading aloud is not boredom
    s_dozing = true;
    set_face(&face_sleeping);
}

// Any interaction resets the boredom clock — and wakes the pet.
static void note_activity(void)
{
    if (s_idle_timer) lv_timer_reset(s_idle_timer);
    if (s_dozing) {
        s_dozing = false;
        set_face(mood_sprite(s_mood));
    }
}

A deliberate design choice: cloud updates don't wake it. The stat bars refresh quietly under the napping face; only human interaction wakes the pet. Nobody likes being woken by a cron job.

The HUD: three stats, three icons, one secret button

Across the top: three bars with icons : 🥑 hunger (avocado green), ⚡ energy (teal), ❤️ mood (pink). The icons are tiny 28×28 sprites drawn programmatically by the same PNG→C pipeline.

The avocado is not just a label, it's a button. Tapping it feeds the pet (we wire up what "feed" means in part 3; today it publishes the interaction):

static void food_clicked_cb(lv_event_t *e)
{
    pet_ui_react_happy();                           // nom nom, instantly
    if (s_on_interaction) s_on_interaction(PET_INTERACTION_FEED);
}

Shake detection: the IMU in 30 lines

The board has a QMI8658 6-axis IMU. You might reach for a gesture-recognition library. Don't, a shake is beautifully simple physics: at rest the accelerometer reads ~1 g of gravity (1000 mg); a shake is the magnitude repeatedly leaving that baseline.

// The Waveshare qmi8658 component hands us calibrated milli-g values
// (we asked for mg units once, at init, with qmi8658_set_accel_unit_mg).
float ax, ay, az;
qmi8658_read_accel(&s_imu, &ax, &ay, &az);

// 1. Total acceleration, all axes combined. At rest: ~1000 mg.
float mag = sqrtf(ax*ax + ay*ay + az*az);

// 2. A "jolt" = a reading well above resting gravity.
if (mag > 1800.0f) jolts++;

// 3. Three jolts within 800 ms = the human is shaking us.
if (jolts >= 3 && within_window) {
    on_shake();
    cooldown();   // 4. then ignore everything for 1.5s (debounce!)
}

The full version (in app_imu.c) polls at 50 Hz from its own FreeRTOS task, plenty for human hands.

The three constants are worth tuning to taste:

💡 Debounce is the whole game. My first version fired 14 "play" events per shake, which in part 3's economy would have turned one wiggle into a pet so overstimulated it refused dinner.

One threading gotcha that bit me: the shake callback runs on the IMU task, not the LVGL task, so the reaction functions take the display lock (it's recursive, so they're safe from touch callbacks too).

Wiring body to nervous system

Everything converges in main.c, and the flow reads like the pet's reflex arc: input, reaction on screen, report to the cloud:

static void on_shake(void)
{
    pet_ui_react_startled();  // reflex: wide eyes, instantly
    app_mqtt_publish("interaction", "{\"type\":\"play\"}");  // then tell the brain
}

void app_main(void)
{
    // Face first: the pet appears before Wi-Fi even starts.
    pet_ui_start(on_interaction);
    pet_ui_set_state(80, 80, 80, PET_MOOD_NEUTRAL);

    app_imu_start(on_shake);
    app_wifi_connect();
    app_mqtt_start(on_cloud_message);
}

That ordering is a deliberate UX decision: react locally first, sync to the cloud second. The happy face flashes the instant you tap, not a network round-trip later. The cloud is the source of truth for state (part 3), but never sits between a touch and its reaction. Pets have reflexes; only their feelings need a brain.

Flash it:

cd firmware && idf.py flash monitor

...and there it is. A face, blinking on that ridiculous little AMOLED, beaming when you tap it, nodding off when you don't. Meanwhile, in the AWS console (MQTT test client, cloudagotchi/#), your affection is now telemetry:

cloudagotchi/cloudagotchi-01/interaction  →  {"type":"pet"}
cloudagotchi/cloudagotchi-01/interaction  →  {"type":"feed"}
cloudagotchi/cloudagotchi-01/interaction  →  {"type":"play"}

Why this matters

Two takeaways travel well beyond virtual pets. First, on constrained display hardware, sprites beat shapes: pre-rendering moves all the expensive work to build time and reduces the runtime to "copy rectangle", the exact operation embedded panels are good at. The corollary: the less your UI redraws, the less can go wrong, and event-driven beats animated. Second, the local-reflex / cloud-state split is the interaction pattern for connected devices. Anything that must feel instant lives on the device; anything that must be remembered lives in the cloud. Get that boundary right and your device feels alive even on flaky hotel Wi-Fi.

Right now, though, our pet publishes its little heart out into the void. Nobody listens. Its stats are hardcoded at 80. It is, developmentally speaking, a very cute reflex with no memory.

Next article: the brain. DynamoDB remembers, Lambda decides, EventBridge Scheduler makes time pass, and the pet finally gets hungry while you sleep.

Try it yourself

• The Cloudagotchi repo : git checkout article-2
• LVGL images documentation : formats, lv_image_dsc_t, and the online converter
• Waveshare BSP on the ESP Component Registry : the one-liner display setup