Fix Promtail log shipping and Logs Explorer dashboard queries

Promtail was not shipping logs due to missing /var/lib/promtail directory
and insufficient permissions (not in adm or systemd-journal groups).
Logs Explorer dashboard queries failed on Loki 3.x because allValue ".*"
is rejected as an empty matcher — changed to ".+" which requires at least
one character. Removed unit filter from main Logs query since varlog
streams lack the unit label.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

incusos/deploy-observability

@@ -465,8 +465,12 @@ scrape_configs:
         target_label: unit
 PTCFG"
 
-    # Ensure data directory exists with correct ownership
+    # Ensure data directories exist with correct ownership
     container_exec monitoring bash -c "mkdir -p /var/lib/loki && chown loki:nogroup /var/lib/loki"
+    container_exec monitoring bash -c "mkdir -p /var/lib/promtail && chown promtail:nogroup /var/lib/promtail"
+
+    # Promtail needs adm group for /var/log access, systemd-journal for journal
+    container_exec monitoring bash -c "usermod -aG adm,systemd-journal promtail"
 
     container_exec monitoring systemctl enable loki
     container_exec monitoring systemctl restart loki

incusos/observability-dashboards/core/logs-explorer.json

@@ -48,7 +48,7 @@
         "label": "Job",
         "refresh": 2,
         "sort": 1,
-        "allValue": ".*"
+        "allValue": ".+"
       },
       {
         "type": "query",
@@ -63,7 +63,7 @@
         "label": "Host",
         "refresh": 2,
         "sort": 1,
-        "allValue": ".*"
+        "allValue": ".+"
       },
       {
         "type": "query",
@@ -78,7 +78,7 @@
         "label": "Unit",
         "refresh": 2,
         "sort": 1,
-        "allValue": ".*"
+        "allValue": ".+"
       },
       {
         "type": "textbox",
@@ -229,7 +229,7 @@
       },
       "targets": [
         {
-          "expr": "{job=~\"$job\", host=~\"$host\", unit=~\"$unit\"} |= \"$search\"",
+          "expr": "{job=~\"$job\", host=~\"$host\"} |= \"$search\"",
           "refId": "A"
         }
       ],

notes/observability-guide.md

@@ -229,8 +229,8 @@ Log exploration powered by Loki with job/host/unit/search variables.
 - Error rate and warning rate by systemd unit
 
 ![Logs Explorer](../screenshots/logs-explorer.png)
-*Log exploration dashboard with volume timeline, log stream, and error/warning
+*Log exploration showing systemd journal and varlog streams, live log entries
-rate panels. Requires Promtail to be shipping logs to Loki for data.*
+with labels and severity highlighting, and error/warning rates by unit.*
 
 ### Incus Operations — Prometheus Health