aurak/docs/superpowers/specs/2026-03-16-talent-assessment-management-design.md

Talent Assessment Management System - Design Document

Date: 2026-03-16 Author: Sisyphus AI Agent Status: Draft for Review

Executive Summary

Enhance the existing talent assessment system with comprehensive management functionality allowing administrators to configure assessment generation through keywords, question counts, and style requirements. The system will extract relevant content from knowledge bases and generate customized assessments based on these configurations.

---

1. Architecture Overview

Current System Flow

User selects knowledge base → System generates 3-5 questions → User answers → AI grades → Report generated

Enhanced System Flow

User selects template/inline config → System filters content by keywords → Generates N questions with specified style → User answers → AI grades → Report generated

Key Components

Backend (NestJS)

• AssessmentConfig Entity: Store reusable assessment templates
• Enhanced Question Generator: Accept config parameters for customization
• Content Filter Service: Filter knowledge base content by keywords
• Config Management API: CRUD operations for assessment templates

Frontend (React)

• Template Management Page: Create/edit assessment templates
• Enhanced Assessment Start: Support template selection + inline config
• Configuration UI: Keywords input, question count slider, style presets

---

2. Database Schema Design

New Entity: AssessmentConfig

@Entity('assessment_configs')
export class AssessmentConfig {
    @PrimaryGeneratedColumn('uuid')
    id: string;

    @Column()
    name: string;

    @Column({ nullable: true })
    description: string;

    @Column({ name: 'tenant_id', nullable: true })
    tenantId: string;

    @Column({ type: 'simple-json' })
    configuration: AssessmentConfigSettings;

    @Column({ default: true })
    isActive: boolean;

    @CreateDateColumn({ name: 'created_at' })
    createdAt: Date;

    @UpdateDateColumn({ name: 'updated_at' })
    updatedAt: Date;
}

Configuration Settings Interface

interface AssessmentConfigSettings {
    // Keywords for content filtering and AI focus
    keywords: string[];

    // Question generation parameters
    questionCount: number;           // 1-20
    difficultyDistribution: {
        standard: number;            // 0-100%
        advanced: number;            // 0-100%
        specialist: number;          // 0-100%
    };

    // Style requirements
    style: {
        tone: 'formal' | 'conversational' | 'technical' | 'business';
        questionTypes: ('multiple-choice' | 'open-ended' | 'scenario-based' | 'technical')[];
        assessmentGoals: ('knowledge-check' | 'problem-solving' | 'critical-thinking' | 'application')[];
    };

    // Content filtering
    contentFilter: {
        minContentLength: number;    // Minimum characters for content to be considered
        maxContextChunks: number;    // Max chunks to include in context
    };
}

Modified Session Entity

Add reference to assessment config:

@Entity('assessment_sessions')
export class AssessmentSession {
    // ... existing fields ...

    @Column({ name: 'config_id', nullable: true })
    configId: string | null;

    @ManyToOne(() => AssessmentConfig, { nullable: true })
    @JoinColumn({ name: 'config_id' })
    config: AssessmentConfig;

    // Inline configuration (when no template used)
    @Column({ type: 'simple-json', nullable: true, name: 'inline_config' })
    inlineConfig: AssessmentConfigSettings | null;
}

---

3. Backend Implementation

3.1 New Service: AssessmentConfigService

@Injectable()
export class AssessmentConfigService {
    async createConfig(tenantId: string, settings: AssessmentConfigSettings): Promise<AssessmentConfig>
    async getConfigs(tenantId: string): Promise<AssessmentConfig[]>
    async getConfig(id: string, tenantId: string): Promise<AssessmentConfig>
    async updateConfig(id: string, tenantId: string, settings: Partial<AssessmentConfigSettings>): Promise<AssessmentConfig>
    async deleteConfig(id: string, tenantId: string): Promise<void>
    async getActiveConfigs(tenantId: string): Promise<AssessmentConfig[]>
}

3.2 Enhanced Content Filter Service

@Injectable()
export class ContentFilterService {
    async filterByKeywords(
        content: string,
        keywords: string[],
        method: 'exact' | 'semantic' | 'hybrid' = 'hybrid'
    ): Promise<string> {
        // Extract relevant sections based on keywords
        // Use semantic similarity for broader matching
        // Return filtered content for question generation
    }
}

3.3 Enhanced Question Generator Node

Modify generator.node.ts to accept configuration:

interface GeneratorConfig {
    keywords?: string[];
    questionCount?: number;
    difficultyDistribution?: AssessmentConfigSettings['difficultyDistribution'];
    style?: AssessmentConfigSettings['style'];
}

export const questionGeneratorNode = async (
    state: EvaluationState,
    config?: RunnableConfig & { generatorConfig?: GeneratorConfig }
): Promise<Partial<EvaluationState>> {
    // 1. Filter knowledge base content by keywords if provided
    // 2. Generate questions based on configuration
    // 3. Enforce question count and difficulty distribution
    // 4. Apply style requirements to question generation
}

3.4 API Endpoints

Config Management

POST   /api/v1/assessment/configs          - Create assessment config
GET    /api/v1/assessment/configs          - List configs
GET    /api/v1/assessment/configs/:id      - Get config details
PUT    /api/v1/assessment/configs/:id      - Update config
DELETE /api/v1/assessment/configs/:id      - Delete config

Enhanced Session Start

// Existing: POST /assessment/start
// New parameters supported:
interface StartSessionRequest {
    knowledgeBaseId: string;
    language?: string;
    configId?: string;                    // Use template
    inlineConfig?: AssessmentConfigSettings; // Or use inline config
}

---

4. Frontend Implementation

4.1 New Page: AssessmentTemplatePage

Location: web/src/pages/workspace/AssessmentTemplatePage.tsx

Features:

• List existing templates with quick actions (edit, clone, delete)
• Create new template wizard
• Template preview with configuration summary
• Status toggle (active/inactive)

4.2 Enhanced Assessment Start Component

Location: web/components/views/AssessmentView.tsx

New UI Elements:

• Template selection dropdown
• "Use custom settings" toggle for inline config
• Configuration panel with:
  • Keywords input (tags)
  • Question count slider (1-20)
  • Difficulty distribution pie chart/editor
  • Style preset buttons
  • Advanced settings collapsible section

4.3 Configuration UI Components

// KeywordsInput.tsx
- Tag-based input for keywords
- Suggestions from knowledge base content
- Visual feedback for keyword matching

// QuestionCountSlider.tsx
- Slider from 1-20 questions
- Smart defaults based on content size

// DifficultyDistribution.tsx
- Interactive pie chart
- Drag to adjust percentages
- Real-time validation (must sum to 100%)

// StylePresets.tsx
- Pre-defined style combinations
- Custom style builder

---

5. Data Flow & Processing

5.1 Content Filtering Pipeline

1. Load knowledge base content
2. Apply keyword filtering (semantic similarity)
3. Chunk content into manageable segments
4. Rank segments by relevance to keywords
5. Select top N segments for context
6. Pass filtered content to question generator

5.2 Question Generation Pipeline

1. Receive configuration + filtered content
2. Analyze content for key concepts
3. Generate questions matching:
   - Specified count
   - Difficulty distribution
   - Question types
   - Style requirements
4. Validate question quality
5. Return structured question set

---

6. Security & Validation

6.1 Input Validation

// Keywords validation
@ValidateNested()
@ArrayMaxSize(50)
keywords: string[];

// Question count validation
@IsInt()
@Min(1)
@Max(20)
questionCount: number;

// Difficulty distribution validation
@ValidateNested()
@IsNotEmpty()
difficultyDistribution: {
    @IsInt()
    @Min(0)
    @Max(100)
    standard: number;
    
    // ... similar for advanced, specialist
};

// Validate sum equals 100
@ValidatorConstraint({ name: 'difficultySum', async: false })
export class DifficultySumConstraint implements ValidatorConstraintInterface {
    validate(value: any) {
        return (value.standard + value.advanced + value.specialist) === 100;
    }
}

6.2 Access Control

• Tenant Isolation: Configs scoped to tenant
• Role-Based Access: Only admins can manage templates
• Ownership Tracking: Track who created/modified each config

---

7. Error Handling

7.1 Common Error Scenarios

Error	Cause	Resolution
INSUFFICIENT_CONTENT	Filtered content too short	Relax keyword constraints or select broader keywords
INVALID_CONFIG	Config validation failed	Return validation errors with specific field issues
CONFIG_NOT_FOUND	Template ID doesn't exist	Return 404 with config ID
KEYWORD_TOO_SPECIFIC	No content matches keywords	Suggest broader keywords or use all content

7.2 Error Response Format

{
  "error": {
    "code": "INSUFFICIENT_CONTENT",
    "message": "Filtered content is too short for question generation",
    "details": {
      "minLength": 1000,
      "actualLength": 250,
      "suggestedKeywords": ["technology", "software"]
    }
  }
}

---

8. Testing Strategy

8.1 Unit Tests

Backend:

• AssessmentConfigService - CRUD operations
• ContentFilterService - Keyword filtering logic
• questionGeneratorNode - Question generation with config

Frontend:

• Configuration UI components
• Template management page
• Integration with existing assessment flow

8.2 Integration Tests

• End-to-end assessment creation with templates
• Content filtering accuracy validation
• Question generation quality assessment

8.3 Performance Tests

• Large knowledge base filtering
• Multiple concurrent template usage
• API response times under load

---

9. Migration Plan

Phase 1: Database Schema (1 day)

1. Create assessment_configs table
2. Add config_id and inline_config to assessment_sessions
3. Create indexes for performance

Phase 2: Backend API (2 days)

1. Implement AssessmentConfigService
2. Create API endpoints
3. Enhance question generator node
4. Add content filtering service

Phase 3: Frontend UI (2 days)

1. Create template management page
2. Enhance assessment start component
3. Add configuration UI components
4. Integrate with existing flows

Phase 4: Testing & Polish (1 day)

1. Comprehensive testing
2. Bug fixes
3. Performance optimization
4. Documentation

---

10. Success Metrics

10.1 Functional Metrics

• ✅ Templates can be created/edited/deleted
• ✅ Question count configurable (1-20)
• ✅ Keywords filter content effectively
• ✅ Style requirements applied to questions
• ✅ Backward compatibility maintained

10.2 Quality Metrics

• Question relevance score > 80% (user feedback)
• Content filtering accuracy > 90%
• API response time < 500ms
• Frontend load time < 2s

10.3 User Experience

• Template creation time < 2 minutes
• Assessment start with config < 10 seconds
• Configuration UI intuitive (measured by user testing)

---

11. Risks & Mitigations

Risk	Impact	Mitigation
Keyword filtering too restrictive	Low quality questions	Implement semantic matching + fallback
Config complexity overwhelms users	Poor adoption	Progressive disclosure + presets
Performance degradation with large KB	Slow generation	Content chunking + caching
Breaking existing assessments	System disruption	Feature flag + gradual rollout

---

12. Future Enhancements

1. AI-powered keyword suggestions based on content analysis
2. Template sharing across tenants (with permissions)
3. Assessment analytics - track which configs produce best results
4. Multi-language templates - store configs in multiple languages
5. Integration with HR systems - export assessment results

---

Appendix A: Configuration Examples

Example 1: Technical Interview Template

{
  "name": "Senior Developer Interview",
  "keywords": ["system design", "algorithms", "database", "API", "security"],
  "questionCount": 8,
  "difficultyDistribution": {
    "standard": 25,
    "advanced": 50,
    "specialist": 25
  },
  "style": {
    "tone": "technical",
    "questionTypes": ["technical", "scenario-based"],
    "assessmentGoals": ["problem-solving", "critical-thinking"]
  }
}

Example 2: Knowledge Check Template

{
  "name": "Weekly Training Quiz",
  "keywords": [],
  "questionCount": 5,
  "difficultyDistribution": {
    "standard": 70,
    "advanced": 30,
    "specialist": 0
  },
  "style": {
    "tone": "formal",
    "questionTypes": ["multiple-choice", "open-ended"],
    "assessmentGoals": ["knowledge-check"]
  }
}

---

Document Status: Ready for review Next Steps:

1. Review this design document
2. Approve or request changes
3. Create implementation plan using writing-plans skill