Skip to main content

Prototype Implementation & Evaluation

This document details the concrete implementation status, verification test suites, regression findings, and known operational characteristics of the Console V2 (Weave) process-centric architecture.

1. Subsystem Implementation Status​

Console V2 reached the V2.5 / V2.6 process-centric architecture. The implementation status across all core modules is summarized below:

SubsystemLocationStatusVerified Capabilities
Process Domainagentictools/processes/ImplementedProcess, immutable ProcessVersion, ProcessStep (20 step types), ProcessRun state machine, StepRun, and ExecutionContext.
Execution Runtimeagentictools/runtime/ImplementedExecutionRuntime, DAG planner (TaskPlan), ToolRunner, SafetyLayer, PolicyEngine, SessionManager, and ModelRouter.
Tool Busagentictools/tools/Implemented29 auto-discovered tools subclassing BaseTool with parameter schema enforcement and MEDIA_ROOT/generated/ file sandboxing.
Skill Registryagentictools/skills/Implemented~95 domain skills across finance, governance, maintenance, risk detection, sales, and newsletter automation.
Memory Substratebrainbox/Implemented (Opt-in)4-layer substrate (event store, 384-dim pgvector semantic memory, knowledge graph, inference layer), decoupled via MemoryProvider.
Governance & HITLcore/ImplementedApprovalQueue supporting manual, hybrid, and autonomous operation; PolicyRuleModel with glob-matching.
Billing & Meteringbilling/Implemented6 plan tiers (Free to Enterprise) with metered ExecutionUsage (token accounting & execution volume).
Omnichannel Supportmonica/ImplementedLive customer-facing support app, FAQ RAG, dispute intake, and embeddable chat widget with origin allow-list.
Vertical Agent Personasarchived/Retired (V2.6)Monolithic personas (marketer, businessfinder, etc.) retired; capabilities migrated to skills; legacy source tree deleted.
Marketplace Surface/api/marketplace/Decommissioned (V2.6)MarketplaceRating and TenantAgentModel authoring surfaces permanently removed in favor of process composition.

2. Test Suites & Verification Coverage​

The platform relied on automated test suites and dedicated regression verification scripts:

Test Distribution​

  • Process Domain (agentictools/processes/tests.py):
    • State machine transitions (draft → running → (waiting | paused) → completed / failed / cancelled).
    • Immutability of published ProcessVersion records (preventing step mutation after freeze).
    • Fail-closed multi-tenant query filtering.
  • Payment & Metering (billing/tests.py):
    • Credit transaction accounting, plan limits enforcement via billing/enforcement.py (PLAN_LIMITS).
  • Support & Widget (monica/tests.py):
    • FAQ retrieval accuracy, monica_public_key verification, and CORS origin validation.
  • Production Regression Suite (scratch/tests/test_sentry_regressions.py):
    • Bootstrap regression testing verifying error-free Django application loading.
    • Race condition mitigation during concurrent Tenant and UserProfile provisioning.
    • Foreign key assignment validation ensuring resolve_django_user() is invoked when assigning lightweight principal objects (JWTUser, APIKeyUser) to ForeignKey(AUTH_USER_MODEL).

3. Key Operational Findings & Production Lessons​

During prototype evaluation and live testing, several critical engineering constraints emerged:

1. Fail-Closed Tenant Filtering​

In multi-tenant SaaS environments, falling back to global model queries when tenant context is missing introduces fatal data-leakage vulnerabilities. Console V2 strictly enforced:

if not tenant:
    return MyModel.objects.none()  # NEVER .all()

2. Lightweight Principals vs. ORM Type-Checking​

The authentication layer (APIKeyOrJWTAuthentication) returned lightweight principal objects (JWTUser, APIKeyUser) rather than standard database model instances to avoid database query overhead on every HTTP request. However, assigning these lightweight objects directly to Django foreign keys caused ValueError: Cannot assign ... must be a "User" instance. The platform resolved this by introducing a mandatory resolve_django_user() resolver.

3. Decoupled Memory Resilience​

Early iterations coupled the execution loop directly to vector search. If the vector index experienced transient latency or connection timeouts, active agent tasks crashed. Decoupling memory behind a pluggable MemoryProvider ensured process runs continue executing even when BrainBox is temporarily unreachable.

4. Process Composition over Open-Ended Autonomy​

Testing demonstrated that enterprise operators were hesitant to grant unconstrained agency to autonomous conversational bots. Transitioning to explicit Process Graphs with conditional branches, parallel joins, and deterministic human approval nodes achieved far higher operational trust and zero runaway execution loops.