[VoxBo] Many vbbatch sequences in the queue

[Daniel Kimberg]

Eddie wrote:
> What's interesting is that sometimes this behavior occurs and
> sometimes it doesn't. As I mentioned previously, I ran the same number
> and type of jobs the day after I got the error initially and was not
> able to generate the error again.

It's probably all about timing, which is going to be sensitive to what else is going on on the cluster, which machines are available, things like that.  If the cluster is a little busy, your jobs will tend to get staggered more, and it will be less of an issue.

> We were running anywhere between 11-65 simultaneous jobs, that each
> dispatched 1-4 (normally 3-4) vbbatch sequences. Each sequence had
> roughly 37 or 47 jobs.

That could definitely run afoul of the scheduler's rate limitations.  I'll try to tweak the code soon to open it up a bit.

dan