Mechanisms are coarsely grained: either you're at the console or you're not (pam_console). Either you're a member of a group or you're not (Debian). There is no easy way to specify that only a subset of privileged operations should be available for a given user (e.g. it's hard to express "it's fine to mount removable media; it's not fine to mount fixed media; it's not fine to change the timezone" in a coherent way).

The way most people use pam-console and sudo is fundamentally broken. Full-fledged GTK+ or Qt applications run as the super user which means that millions of line of code (including code such as image loaders that historically have lots of security problems) runs privileged. This is in direct violation of the well-known "least privilege" principle. In addition, often applications look out of place because settings in such programs now read per-user settings from root's home directory.

UNIX group membership have always been problematic; if a user is a member of a group once, he can always become member of the group again (copy /bin/bash to $HOME; chown to group, set the setgid bit, done).

It is difficult for upstream projects (such as GNOME or KDE) to implement features that requires administrative privileges because most downstream consumers (e.g. operating systems) have different ways of implementing access control. As a result most of these features are punted to OS distributors who have their own code for doing the same thing e.g. setting the date/timezone etc.; there is no way for file sharing applications (such as gnome-user-share, Banshee, Rhythmbox) to punch a hole in the firewall.

Without a centralized framework, access control configuration is often scattered throughout the system which makes it hard for system administrators to grasp how to configure the system. There's literally a bunch of different configuration files all with different formats and semantics.