Troubleshooting

Service diagnostics

All diagnostic procedures require the app_admin role.

Check service readiness and container state:

CALL veil.core.service_status();

Wait for services after a restart (blocks up to 10 minutes):

CALL veil.core.wait_for_services();

Common issues

Services not starting

If service_status() shows containers in a pending or failed state:

1. Verify compute pools exist and have capacity
2. Check that the account has the required privileges (CREATE COMPUTE POOL, BIND SERVICE ENDPOINT)
3. Restart services:

CALL veil.core.stop_services();
CALL veil.core.start_services();
CALL veil.core.wait_for_services();

Encoding returns NULL

• Verify the encoder exists: SELECT * FROM veil.encoders_v;
• Verify the encoding service is running: CALL veil.core.service_status();
• Check that the input array dimension matches the encoder’s expected dimension

Training fails

Check the job status for the error category:

SELECT encoder_name, status, error_category
FROM veil.training_jobs_v
WHERE encoder_name = 'my_encoder'
ORDER BY created_at DESC
LIMIT 1;

Common failure categories:

• data_error: Source table not accessible, columns not found, or all-null columns
• resource_error: Compute pool capacity exceeded or GPU family unavailable
• timeout: Training exceeded the maximum runtime

GPU mode rejected

GPU mode selection checks available instance families in the consumer’s region. If no tested family is available, GPU mode is rejected.

Verify available families:

SHOW COMPUTE POOL INSTANCE FAMILIES IN ACCOUNT;

Fall back to CPU if no GPU family is listed:

CALL veil.core.set_training_compute('cpu');
CALL veil.core.start_services();

Training slot locked

Only one training job runs at a time. If a previous job is stuck:

SELECT * FROM veil.training_jobs_v WHERE status = 'running';

A stuck job can be cancelled, which releases the training slot.