Job status and time estimates

The alallo Jobs tab

The Jobs tab of the alallo dashboard lists every job in your account, newest first, with its name, training method, status, progress, estimated time left and price. Click a job to see its details: status, progress, "Time left (estimate)", the price, and the training log. The Jobs tab refreshes automatically every few seconds.

alallo job statuses

Status	Meaning
queued	The job has been accepted and is waiting to start training.
running	The job is training.
re-scheduling	The job is being moved so that it has enough capacity to run. Progress continues afterwards.
cancelling	You cancelled the job and alallo is stopping it.
cancelled	The job was cancelled.
succeeded	Training finished and the fine-tuned model is ready.
failed	The job stopped without finishing. The message explains why. A failed job costs nothing.

alallo job messages

Some alallo job statuses come with a message in the job detail:

Message	Meaning
Starting	The job is queued and alallo is preparing to start it.
Waiting for capacity	The job is queued and will start as soon as capacity is free. No action is needed.
Re-scheduling	alallo found that the job needs more capacity than first planned and is moving it.
Not available right now.	The chosen model and method cannot run at the moment. You pay nothing.
Not available right now. Choose a lower precision.	The model cannot be trained at full precision (16-bit) at the moment. Try Supervised fine-tuning with 4-bit precision. You pay nothing.
Stopped: the budget was reached	alallo stopped the job because it took far longer than planned. You pay nothing.
Stopped: the time limit was reached	The job reached the maximum running time and was stopped. You pay nothing. [TBD: the time limit per job; currently 12 hours.]
Training failed: ...	The training stopped with an error; the last relevant line of the training log follows, for example a missing column in the data. Check your data and settings. You pay nothing.
Failed. We've been notified and are looking into it.	Something went wrong on alallo's side. You pay nothing.

Re-scheduling in alallo

"Re-scheduling" means alallo moved your job because the first training steps showed that it needs more capacity than first planned. alallo continues the job from its most recent saved checkpoint. alallo may also move a job at other times to keep it running. You never pay for repeated work.

The estimated time before an alallo job starts

When you click "Estimate time and price", alallo shows an estimated time as a range, for example "40 min to 1.3 h". The range is wider on the upper side because the time a job takes depends on how soon capacity is available. If no capacity is free, the estimate says "No capacity is free right now; the job would wait in the queue." The upper end of the estimated time range also matters for cancelling: see Cancelling jobs and refunds.

Time left while an alallo job runs

The "Time left" of a running alallo job is an estimated range. At the start it is based on the estimate. Once about 5% of training is done, alallo calculates the time left from the job's actual pace, and the range becomes narrower. Times under 1.5 hours are shown in minutes; longer times in hours.

Progress of an alallo job

The progress bar of an alallo job shows the percentage of training steps completed, out of the job's Max steps. Hover over the bar to see the total number of steps. A succeeded job shows 100%.

Lowest cost and Fastest finish affect time

An alallo job with the goal "Fastest finish" aims to finish as soon as possible; a job with "Lowest cost" aims for the lowest price and is usually also quick for small jobs. The estimate shows the expected time for the goal you chose.

The alallo training log

The job detail shows the training log of the job's current or last run, with the newest lines at the bottom. Before training starts the log says "No log yet. It appears once training starts." The log shows training progress, such as the loss value every 10 steps, saved checkpoints and the end of training. Errors from the training also appear in the log.