[VoxBo] permutation testing question

[Daniel Y Kimberg]

Ranjani wrote:
> I was wondering what the difference is between the "vbpermgen" and
> "vbpermmat" tools for permutation testing.

vbpermgen is the graphical interface for asking VoxBo to do a
permutation test.  vbpermmat is a program that's mostly used
internally, it generates a matrix in which each column represents a
separate permutation.

> Also, when using the vbpermgen tool, there is an option in the GUI
> to select either a Signed or Regular permutation.  I think the
> default is for a signed permutation, but I was wondering what the
> difference is between the two types.

These should probably be called "order" and "sign flipping," but
here's the short version on what they mean.  What we're calling
"regular" is a test where each permutation re-arranges the order of
the observations.  A sign permutation means randomly flipping the
signs of the observations.  These tests are only valid when under the
null hypothesis there's no expected difference between the test
statistics for permuted and unpermuted data (but under H1 there is).

Neither test is good for autocorrelated data (e.g., single-subject
BOLD data).  If you're doing a group analysis in which you're
comparing some test statistic to zero (i.e., a model in which the only
independent variable is an intercept), then you need to use the sign
permutation, since the test statistic doesn't care about order.  If
you're doing a comparison between two groups, or an analysis of a
single subject's ASL data, then an order permutation would be a good
choice.

It would probably be better if the vbpermgen interface didn't have a
default.  I worry that it's too easy to make a nonsensical choice by
assuming that the defaults are always reasonable.

dan