Self-Refine and Self-Correction Methods

Graham Neubig

Self-correction Process

def self_correction(x, max_iterations=3):
    y = generate(x)  # Initial generation
    
    for i in range(max_iterations):
        critique = evaluate(y, x)  # Evaluate current output
        if is_satisfactory(critique):
            break
        y = refine(y, critique, x)  # Improve based on critique
    
    return y

Design Decisions

• Critique: explicit or implicit
• Training: without or with
• Conditioning: \(x\), \(y_n\), \(c_n\)
• Tools: none, optional, or required

Deliberation Networks

Paper: Deliberation Networks: Sequence Generation Beyond One-Pass Decoding (Xia et al., 2017)

Key Innovation: Two-pass decoding with deliberation and refinement

• First-pass decoder generates initial draft sequence
• Second-pass decoder refines with global context

Deliberation Networks: Algorithm

def deliberation_networks(x):
    # First-pass: generate initial draft
    y = first_pass_decoder(x)
    
    # Second-pass: refine with full context (no iteration)
    y = second_pass_decoder(x, y)
    
    return y

Key Insight: Two-pass refinement with global context, no iterative feedback loop

Deliberation Networks: Results

Performance on Neural Machine Translation:

Learning to Model Editing Processes

Paper: Learning to Model Editing Processes (Reid & Neubig, 2021)

Key Innovation: Model the entire multi-step editing process, not just single edits

Edit Processes: Results

Key Findings:

• 22.9% relative perplexity reduction over state-of-the-art editing baseline
• Multi-step context crucial for predicting non-KEEP operations

Edit Representations: Algorithm

def edit_representations(source, target):
    # Encode the edit operation between source and target
    edit_vector = edit_encoder(source, target)
    
    # Apply learned edit to new source text
    def apply_edit(new_source):
        return neural_editor(new_source, edit_vector)
    
    return apply_edit

Key Insight: Learn semantic edit representations that generalize across different texts

Edit Representations: Details

Architecture Components:

• Edit Encoder: BiLSTM processes aligned token sequences with diff symbols (+, -, ↔, =)
• Neural Editor: Seq2seq with attention and copying mechanism
• Bottleneck constraint: Edit representation limited to 512 dimensions

Training Objective:

Self-Refine

Paper: Self-Refine: Iterative Refinement with Self-Feedback (Madaan et al., 2023)

Key Innovation: Training-free iterative improvement using self-generated feedback

Self-Refine: Algorithm

def self_refine(x, max_iterations=3):
    y = generate(x)  # Initial generation
    
    for i in range(max_iterations):
        critique = self_feedback(y, x)  # Generate self-critique
        
        if is_satisfactory(critique):
            break
            
        y = refine(y, critique, x)  # Improve based on critique
    
    return y

Key Insight: Single LLM handles generation, self-critique, and refinement through prompting

Self-Refine: Generation and Feedback

Prompt-Based Components:

Self Debug: Results

Key Findings:

• Significant improvements on code translation tasks
• Execution feedback more effective than simple prompting
• 1 sample with Self-Debug matches 16 samples without

CRITIC

Paper: CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (Gou et al., 2023)

Key Innovation: Combine self-correction with external tool verification

CRITIC: Algorithm

def critic(x, max_iterations=3):
    y = generate(x)  # Initial generation
    
    for i in range(max_iterations):
        # Use external tools to verify/critique
        critique = tool_critique(y, x)
        
        if critique.is_correct:
            break
            
        # Refine based on tool feedback
        y = refine(y, critique, x)
    
    return y

Key Insight: External tools provide objective verification and specific error feedback

CRITIC: Results