Big dataset: how to handle?

[andrealistanti]

Hi, I am programming an experiment about word recognition. The experiment is very simple: we have a long list of words and pseudowords, a hundred of them (67 words, 33 pseudowords) are shown on the screen one at a time and the participants have to respond whether they know them or not.

PC Ibex picks a random sample of words and pseudowords from the list via the function “pick”:

function Pick(set,n) {
    assert(set instanceof Object, "First argument of pick cannot be a plain string" );
    n = Number(n);
    if (isNaN(n) || n<0) n = 0;
    this.args = [set];
    set.remainingSet = null;
    this.run = function(arrays){
        if (set.remainingSet===null) set.remainingSet = arrays[0];
        const newArray = [];
        for (let i = 0; i < n && set.remainingSet.length; i++)
            newArray.push( set.remainingSet.shift() );
        return newArray;
    }
}
function pick(set, n) { return new Pick(set,n); }

In the sequence, I specify the number of words and pseudowords I want to get picked:
rshuffle(pick(randomize("words"),67),pick(randomize("pseudowords"),33))

The problem is that the full dataset has 32400 words and 10921 pseudowords. The current code works only up to a total of ca. 5000. If I try to use the full dataset I get this message: “Range error: maximum call stack size ecceded”.

Is there a way to solve the problem?

Many thanks!

Andrea

[Jeremy]

Hi Andrea,

Would you mind sharing a link to your study? I cannot seem to reproduce the problem myself

Jeremy

[andrealistanti]

Sure, here it goes.

https://farm.pcibex.net/r/rpAXut/

Thanks a lot

Andrea

[Jeremy]

Hi Andrea,

That link is invalid. If you create a new project to share, make sure to do so while logged in, so it can be saved in your account; projects created while logged out are automatically deleted 30 minutes after their last edit

Jeremy

[andrealistanti]

Dear Jeremy,
thanks a lot for your reply and for your help, and sorry about the link. In the meantime I managed to solve the problem, so for the moment I do not need to bother you. Thanks a lot!

Andrea

[Author]

Posts