Linux genpd Power Domains: Why a Rail Won’t Power Off?

A genpd power domain removes power only when every device attached to it is runtime-suspended, every subdomain is already off, and the governor agrees. When a rail stays on, the cause is one of the conditions genpd_power_off() checks before it calls the provider's ->power_off() callback. Those conditions are a device that never runtime-suspended, a subdomain still counted as on, an always-on flag, a system suspend in progress, a resume-latency constraint the governor will not accept, or the boot-time stay_on hold introduced in Linux 6.17. Start at /sys/kernel/debug/pm_genpd/pm_genpd_summary, which shows three of those conditions directly and narrows down the rest.

On most embedded SoCs, power is not gated per device. It is gated per block: one switch feeds the GPU and its MMU, another feeds the display controller together with the DSI host and the PHY. The Linux kernel models these blocks as genpd power domains. This article explains, from the code, how genpd power domains decide that a block may lose power, and how to find out which condition is holding your rail on.

It sits one level below an earlier post on the per-device layer, Runtime PM: Why Your Device Never Suspends. Runtime PM tracks whether one device is idle; genpd decides whether the shared rail feeding several devices may be switched off. They fail for different reasons.

Where the genpd code lives

The data structures are in include/linux/pm_domain.h. The core logic, the governors and the debugfs interface are in drivers/pmdomain/ — the core in drivers/pmdomain/core.c, the in-tree governors in drivers/pmdomain/governor.c. The core moved there in Linux 6.8; on 6.7 and earlier it is at drivers/base/power/domain.c. Everything quoted below is from Linux 6.17, and one mechanism described later is new in that release. The framework needs CONFIG_PM_GENERIC_DOMAINS, which platform code selects rather than you enabling it in menuconfig; CONFIG_PM_GENERIC_DOMAINS_OF and CONFIG_PM_GENERIC_DOMAINS_SLEEP then default to y when OF and PM_SLEEP are set. The debugfs interface is guarded by CONFIG_DEBUG_FS.

raghu@techveda.org:~$ zcat /proc/config.gz | grep -E 'PM_GENERIC_DOMAINS|CONFIG_DEBUG_FS='
CONFIG_PM_GENERIC_DOMAINS=y
CONFIG_PM_GENERIC_DOMAINS_SLEEP=y
CONFIG_PM_GENERIC_DOMAINS_OF=y
CONFIG_DEBUG_FS=y

Inside struct generic_pm_domain

A provider driver allocates a struct generic_pm_domain, fills in the two hardware callbacks, and registers it. These are the fields that decide power-off:

/* include/linux/pm_domain.h - fields discussed here, comments condensed */
struct generic_pm_domain {
        struct list_head        parent_links;    /* links where this domain is the parent */
        struct list_head        child_links;     /* links where this domain is the child */
        struct list_head        dev_list;        /* devices attached to this domain */
        struct dev_power_governor *gov;
        const char              *name;
        atomic_t                sd_count;        /* subdomains currently powered on */
        enum gpd_status         status;          /* GENPD_STATE_ON or GENPD_STATE_OFF */
        unsigned int            device_count;
        unsigned int            suspended_count; /* system suspend device counter */
        unsigned int            prepared_count;  /* suspend counter of prepared devices */
        bool                    stay_on;         /* stay powered-on during boot */
        int (*power_off)(struct generic_pm_domain *domain);
        int (*power_on)(struct generic_pm_domain *domain);
        unsigned int            flags;
        struct genpd_power_state *states;
        unsigned int            state_count;
        unsigned int            state_idx;       /* state the domain enters when off */
};

Registration is pm_genpd_init(genpd, gov, is_off), where the third argument states whether the hardware is already off at registration. Hierarchy is declared with pm_genpd_add_subdomain(parent, child). For Device Tree, the provider then calls of_genpd_add_provider_simple() for a single domain or of_genpd_add_provider_onecell() for several, and consumer nodes reference it with power-domains = <&pd 0> against the provider's #power-domain-cells. Attachment happens automatically during probe, but only for a device with exactly one power-domains entry: genpd_dev_pm_attach() counts the phandles and returns without attaching when the count is not one. A device listing two domains is silently left unmanaged unless its driver calls dev_pm_domain_attach_by_name(), which keys on power-domain-names, or devm_pm_domain_attach_list(). Nothing reports this, so it is worth checking early on a new board. Idle states come from a domain-idle-states property in the provider node; without it the domain gets one default state and can only ever report off-0.

Reading pm_genpd_summary

The genpd core creates a debugfs directory named pm_genpd holding a global file called pm_genpd_summary. Read it first on any board where power is higher than expected.

raghu@techveda.org:~$ sudo mount -t debugfs none /sys/kernel/debug
raghu@techveda.org:~$ sudo cat /sys/kernel/debug/pm_genpd/pm_genpd_summary

The two header lines and the separator are printed verbatim by summary_show(). An illustrative result on a board with a display domain and a GPU domain looks like this:

domain                          status          children        performance
    /device                         runtime status                  managed by
------------------------------------------------------------------------------
pd_display                      on                              0
                                                pd_dsi
    fd4a0000.display                active                      0           SW
pd_dsi                          on                              0
    fd4b0000.dsi                    active                      0           SW
pd_gpu                          off-0                           0
    fde60000.gpu                    suspended                   0           SW

The layout is not obvious on a first reading, because subdomains appear on their own indented line rather than beside the domain that owns them. The same output as a tree:

The status column shows on, or off-N where N is state_idx, the index of the idle state the domain entered — a domain with several idle states will show off-0, off-1 and so on. Child domain names print on a continuation line indented by 48 characters, so they land under the children heading rather than on the domain's own row. Device rows are indented four spaces and carry the device's Runtime PM status, then its performance state, then HW or SW for hardware- or software-controlled mode.

Read the Runtime PM status closely. active, suspended, suspending and resuming are the normal values, but unsupported means disable_depth is non-zero: the driver never called pm_runtime_enable(), or disabled it again. Such a device can never satisfy pm_runtime_suspended(), so it blocks its domain permanently. Rule that out first.

Why genpd power domains refuse to power off

The decision for all genpd power domains is made in one function, genpd_power_off(). It returns early, without touching the hardware, under any of the following conditions. The checks fall into four stages, and the domain stays on the moment any one of them fails:

The individual conditions, in the order the code tests them:

1. The domain is already off (status is not on).
2. prepared_count > 0, meaning a system suspend is in progress for devices in this domain.
3. The domain has GENPD_FLAG_ALWAYS_ON, or has GENPD_FLAG_RPM_ALWAYS_ON (on except during system suspend).
4. stay_on is still set — the boot-time hold described in the next section.
5. sd_count > 0: at least one subdomain is powered on. This is tested again after the governor call, in case a subdomain began powering on in the meantime.
6. Any child domain is not in its deepest state, that is state_idx is below state_count - 1.
7. Any attached device has rpm_always_on set, which a driver requests with dev_pm_genpd_rpm_always_on().
8. More than one attached device is not runtime-suspended. The core counts devices where pm_runtime_suspended() is false, and also counts an IRQ-safe device attached to a domain that is not IRQ-safe, that is, one without GENPD_FLAG_IRQ_SAFE. Exactly one such device is tolerated only when the call came from the domain's own runtime-suspend path, because that device's status has not been updated yet.
9. The governor refuses. The common choice is simple_qos_governor, and the QoS in the name is the point: its default_power_down_ok does not predict how long the domain will be idle. It adds the state's power_off_latency_ns and power_on_latency_ns, then refuses if any attached device's effective PM QoS resume-latency constraint, or any subdomain's max_off_time_ns, is less than or equal to that total. It refuses because a consumer could not tolerate the wake-up delay, not because the idle window looked short. The governor that does predict idle duration is pm_domain_cpu_gov, for CPU domains under CONFIG_CPU_IDLE. A provider passing NULL to pm_genpd_init() has no governor, skips this check, and always uses state index 0.

If all of that passes, the core runs the GENPD_NOTIFY_PRE_OFF notifier chain and then calls the provider's ->power_off(). If either step fails, genpd increments states[state_idx].rejected and leaves the domain on. Note also that a provider with no ->power_off() callback at all is treated as a success: the domain is marked off in software with no hardware action. On success the core sets GENPD_STATE_OFF, increments the state's usage counter, then walks up through child_links and attempts to power off each parent domain in turn.

One point is frequently misunderstood: the comparison of suspended_count against device_count is not part of this path. That comparison lives in genpd_sync_power_off(), which runs only in the noirq and syscore stages of system suspend. Runtime power-off counts devices by walking dev_list.

The boot-time hold: stay_on and sync_state

Of all the conditions that keep genpd power domains powered, this is the one most likely to be unfamiliar — and the one to check your kernel version against first. The stay_on field, GENPD_FLAG_NO_STAY_ON and of_genpd_sync_state() are new in Linux 6.17. On 6.16 and earlier this mechanism does not exist and grepping for it will find nothing. Among the current long-term branches that means only 6.18.y carries it; 6.12.y, 6.6.y, 6.1.y, 5.15.y and 5.10.y do not, so most products shipping on a long-term kernel today will not see this behaviour at all.

pm_genpd_init() always calls genpd_set_stay_on(). Built with CONFIG_PM_GENERIC_DOMAINS_OF, it sets stay_on = true for any domain already powered on, unless the provider passed GENPD_FLAG_NO_STAY_ON; built without it, a second definition hard-codes the field to false:

#ifdef CONFIG_PM_GENERIC_DOMAINS_OF
static void genpd_set_stay_on(struct generic_pm_domain *genpd, bool is_off)
{
        genpd->stay_on = !genpd_is_no_stay_on(genpd) && !is_off;
}
#endif

While stay_on is true, genpd_power_off() returns immediately. The purpose is to protect devices that firmware left running — a display that is already showing a splash screen, for example — from losing power before their driver has probed. The hold is released by the driver-model sync_state() callback, once every consumer of the provider has probed. The core clears the flag and immediately retries the power-off:

void of_genpd_sync_state(struct device_node *np)
{
        struct generic_pm_domain *genpd;

        if (!np)
                return;

        mutex_lock(&gpd_list_lock);
        list_for_each_entry(genpd, &gpd_list, gpd_list_node) {
                if (genpd->provider == of_fwnode_handle(np)) {
                        genpd_lock(genpd);
                        genpd->stay_on = false;
                        genpd_power_off(genpd, false, 0);
                        genpd_unlock(genpd);
                }
        }
        mutex_unlock(&gpd_list_lock);
}

Lifetime of the hold, on Linux 6.17 and newer:

The practical consequence: if one consumer of a genpd provider never probes — a missing driver, a failed regulator, a permanently deferred probe — sync_state() never fires, stay_on is never cleared, and the domain stays powered for the life of the boot. The default policy is the strict one — fw_devlink.sync_state=strict waits indefinitely for consumers to probe successfully — so fw_devlink.sync_state=timeout on the kernel command line is a quick way to test whether this is what you are looking at.

The symptom is a domain that reads on in pm_genpd_summary with every listed device showing suspended. Do not stop there: an always-on flag, a per-device rpm_always_on, a subdomain not in its deepest state, and a governor refusal all produce exactly the same output. The summary cannot separate them, which is why the next section and the checklist above matter.

The per-domain debugfs files

Alongside the global summary, genpd creates one directory per domain containing current_state, sub_domains, idle_states, active_time, total_idle_time and devices. A perf_state file appears only for providers that implement set_performance_state.

raghu@techveda.org:~$ ls /sys/kernel/debug/pm_genpd/pd_gpu/
active_time  current_state  devices  idle_states  sub_domains  total_idle_time
raghu@techveda.org:~$ sudo cat /sys/kernel/debug/pm_genpd/pd_gpu/idle_states
State          Time Spent(ms) Usage      Rejected   Above      Below
S0             412            37         2          0          0

Usage and Rejected are the useful pair. A rising Rejected count means the core decided to power off but the transition failed — usually the provider's ->power_off() callback returning an error, though a GENPD_NOTIFY_PRE_OFF notifier veto increments the same counter. Either way the next place to look is that callback. A Usage count of zero with Rejected also at zero means the domain was never asked to power off at all, which points back to the list above.

Above and Below record how often the chosen idle state was too deep or too shallow for the residency actually achieved, but they are only maintained by pm_domain_cpu_gov, which is the sole caller that sets reflect_residency. On a domain running simple_qos_governor or no governor they stay at zero, as above, so do not read anything into that. Unlike the summary, devices and sub_domains print one bare name per line with no header.

A practical order of investigation

Use this order when genpd power domains stay on and the cause is not yet known.

1. Read pm_genpd_summary and find the domain that is on when you expect it off.
2. Any device row that is not suspended is a Runtime PM problem in that driver, not a genpd problem. A row reading unsupported means runtime PM was never enabled for that device.
3. If every device is suspended, check the continuation line and confirm each subdomain is also off.
4. If devices and subdomains are all idle, the cause is one of the conditions the summary cannot show. Check them directly:

raghu@techveda.org:~$ cat /sys/kernel/debug/devices_deferred
raghu@techveda.org:~$ grep -rn 'GENPD_FLAG_ALWAYS_ON\|GENPD_FLAG_RPM_ALWAYS_ON' drivers/pmdomain/
raghu@techveda.org:~$ cat /sys/devices/platform/fde60000.gpu/power/pm_qos_resume_latency_us
raghu@techveda.org:~$ cat /sys/kernel/debug/regulator/regulator_summary

The first shows consumers still waiting to probe, which on Linux 6.17 is what keeps stay_on set. The second finds an always-on flag in the provider driver. The third shows a resume-latency constraint tight enough to make default_power_down_ok refuse; this attribute exists only when the driver called dev_pm_qos_expose_latency_limit(), so its absence does not rule the cause out. The fourth is the check most often skipped: off-0 means only that the core believes the domain is off. Confirm the rail actually dropped, through the regulator summary or a meter on the sense point, before concluding the software is correct.

1. If the domain does power off but not as often as expected, read idle_states. A rising Rejected count means the core did decide to power off and the transition failed, so the problem is in the provider's callback rather than in the checklist.

Working through power domains on real silicon is a large part of driver work, and it is covered in our Linux device drivers training.

Key takeaways

• Runtime PM decides whether a device is idle; genpd power domains decide whether a shared rail may be switched off. Both must agree.
• genpd_power_off() tests a fixed set of conditions in a fixed order. Match the symptom to one of them instead of guessing.
• off-N in pm_genpd_summary is not an error — N is the index of the idle state the domain entered.
• A domain that is on while all its devices are suspended has several possible causes that the summary cannot tell apart: an always-on flag, a per-device rpm_always_on, a governor refusal, or, on Linux 6.17 and newer, the stay_on hold.
• simple_qos_governor refuses on PM QoS resume-latency constraints, not on a predicted idle window. That changes what you go looking for.
• The Rejected counter in idle_states separates a core-level refusal from a transition that was attempted and failed.
• genpd reporting off-0 means the core believes the domain is off. Confirm the rail separately.
• The core moved to drivers/pmdomain/core.c in Linux 6.8; before that it is drivers/base/power/domain.c.

Frequently asked questions

What does "off-0" mean in pm_genpd_summary?
It means the domain is powered off and entered idle state index 0. The number after the hyphen is state_idx, so a domain with several idle states can show off-0, off-1 and so on. Only a powered-on domain prints a plain on.

Every device in my domain is suspended but the domain still reads "on". Why?
pm_genpd_summary cannot distinguish the remaining causes, so check them in turn: GENPD_FLAG_ALWAYS_ON or GENPD_FLAG_RPM_ALWAYS_ON in the provider driver, a device that called dev_pm_genpd_rpm_always_on(), a subdomain not in its deepest state, and a governor refusing on a resume-latency constraint. On Linux 6.17 and newer there is one more: the boot-time stay_on hold, which is released only when the provider's sync_state() callback runs after every consumer has probed.

What is the difference between GENPD_FLAG_ALWAYS_ON and GENPD_FLAG_RPM_ALWAYS_ON?
GENPD_FLAG_ALWAYS_ON keeps the domain powered at all times. GENPD_FLAG_RPM_ALWAYS_ON keeps it powered during normal operation but allows a power-off during system suspend, which runs through genpd_sync_power_off() instead. Both make genpd_power_off() return before it reaches the device and subdomain checks.

What is the difference between the Rejected and Usage counters in idle_states?
Usage counts the times the domain successfully entered that idle state. Rejected counts the times the core decided to power off but the transition failed, usually because the provider's ->power_off() callback returned an error, though a GENPD_NOTIFY_PRE_OFF notifier veto increments the same counter. A rising Rejected count points at the provider driver and its hardware sequence.

Further reading

• include/linux/pm_domain.h (Linux 6.17) — struct generic_pm_domain, the GENPD_FLAG_* definitions and the full genpd API.
• drivers/pmdomain/core.c — genpd_power_off(), genpd_set_stay_on(), of_genpd_sync_state() and the debugfs implementation.
• Documentation/devicetree/bindings/power/power-domain.yaml — the power-domains and #power-domain-cells bindings.
• Runtime Power Management Framework for I/O Devices — the per-device layer that genpd builds on.
• PM: SoC idle support using PM domains (LWN) — background on idle states and the genpd governors.