Lists randomization

PennController for IBEX Forums Support Lists randomization

Tagged: 

Viewing 10 posts - 1 through 10 (of 10 total)
  • Author
    Posts
  • #9568
    rsw123
    Participant

    Hi Jeremy,

    My question is about how to randomize the lists in my script.

    The experiment has four lists: List1, List2, List3 and List4. And to say, I want to assign each list of 10 participants (40 in total). I was wondering how could I achieve this?

    Should I include one column in the csv file called List? and how could I write the Sequence line?

    Many thanks in advance!

    #9570
    Jeremy
    Keymaster

    Hi,

    Yes, this is described in the tutorials here and here

    You can name the column “Group” or “List”, it doesn’t matter, PennController will use it to subset the table to the rows whose corresponding column contains only one of the possible values

    Circulate the link to your experiment ending with server.py?withsquare=0 to some participants, with server.py?withsquare=1 to some other participants, and so on, until you get 10 submissions for each list

    Jeremy

    #9585
    rsw123
    Participant

    Many thanks for the link! I got the idea.

    But before continue to the randomization step, I tried to run the following script and it returned the error message “Alert from preview: ERROR: There must be some items in the running order!” Things went well before adding the “practice” part. Could you please give me some hints on what happening here? (the demo link also attached here: https://farm.pcibex.net/r/QzMAyX/)

    // Remove command prefix
    PennController.ResetPrefix(null)
    
    // Control trial sequence
    Sequence("information", "consent", "consentform", "demographic", "instructions", "practice", randomize("experimental-trial"), "send", "completion_screen")
    
    // Information sheet
    newTrial("information",
        newHtml("informationsheet", "informationsheet.html")
            .cssContainer({"width":"720px"})
            .print()
        ,
        newButton("continue", "Click here to continue")
            .center()
            .print()
            .wait(getHtml("informationsheet").test.complete()
            )
    )
    
    // Consent info
    newTrial("consent",
        newHtml("consent", "consent.html")
            .cssContainer({"width":"720px"})
            .checkboxWarning("You must consent before continuing.")
            .print()
        ,
        newButton("continue", "Click here to continue")
            .center()
            .print()
            .wait(getHtml("consent").test.complete()
                      .failure(getHtml("consent").warn())
            )
    )
    
    // Consent form
    newTrial("consentform",
        newHtml("consentform", "consentform.html")
            .cssContainer({"width":"720px"})
            .checkboxWarning("You must consent before continuing.")
            .print()
        ,
        newButton("I consent to items above")
            .center()
            .print()
            .wait(getHtml("consentform").test.complete()
                      .failure(getHtml("consentform").warn())
            )
    )
    
    // demographic
    newTrial("demographic",
        newHtml("demographic", "demographic.html")
            .cssContainer({"width":"720px"})
            .checkboxWarning("You must consent before continuing.")
            .print()
        ,
        newButton("continue", "Click here to continue")
            .center()
            .print()
            .wait(getHtml("demographic").test.complete()
                      .failure(getHtml("demographic").warn())
            )
    )
    
    // instructions
    newTrial("instructions",
        newHtml("instructions", "instructions.html")
            .cssContainer({"width":"720px"})
            .checkboxWarning("You must consent before continuing.")
            .print()
        ,
        newButton("continue", "Click here to continue")
            .center()
            .print()
            .wait(getHtml("instructions").test.complete()
                      .failure(getHtml("instructions").warn())
            )
    )
    
    // practice
    newTrial("practice",
        // will print all Text elements, horizontally centered
        defaultText.center().print()
        ,
        newText("Practice Session")
        ,
        newText("You will now be given eight practice trials to familiarize yourself with the experiment. During the practice trials, you will receive feedbacks on whether you recreate the pattern correctly and on whether your decision about the object is correct.")
        ,
        newText("Please press 1 if the object has been mentioned in the preceding sentence, or press 2 if has not been mentioned.")
        ,
        newText("You should do this as quickly and accurately as possible.")
        ,
        newText("When you are ready, press C to start a practice run.")
        ,
        newKey("keypress", "C")
        .wait()  // Finish trial upon press on c
    )
    #9593
    Jeremy
    Keymaster

    Hi,

    The script that’s actually running has PennController.ResetPrefix(null) too far down (it should be the first line of the script) and it misses a comma between "demographic" and "instructions" in the Sequence command

    This is most likely due to the Save function bug on the farm: try copying the content of your script file, deleting it from your project and recreating it and pasting the content back in, and it should work

    Jeremy

    #9694
    ediachek
    Participant

    Hi Jeremy,

    Just launched my experiment yesterday for some pilot data and it works great! Thank you for working on the bugs and for your continued support.

    I noticed just one thing — I have 2 phases of the experiment: study and test. And I also have 16 experimental lists, and participants are assigned to a list randomly through the SetCounter(“counter”, “inc”, 1); command at the beginning of the experiment. I assumed that participants will be assigned to the same study and test lists but I’m seeing that some participants completed list 3 for study and list 11 for test.

    Is there a way to fix that so that each person completes the same list for study and test?

    Here is the link to the expt: https://farm.pcibex.net/r/dmrkYm/

    Best,
    Yev

    • This reply was modified 1 year, 4 months ago by ediachek.
    #9699
    Jeremy
    Keymaster

    Hi Yev,

    Your two tables to have 16 groups each, but only 7 groups are shared by the two tables (10,11,12,13,14,15,16). The other 9 groups are different:  1 (study) vs 1 (test),  2 (study) vs 2 (test), etc.

    Because   (the space character) comes before any digit in the alphabetical order, the second possible value of group for example in the study table is  2, but 10 in the test table (because 1 comes before 2 and 0 comes before 1). So any participant running with a counter value of 2 (or modulo 2) will be assigned group  2 from the study table, and group 10 from the test table

    Jeremy

    #9702
    ediachek
    Participant

    Hi Jeremy,

    Thank you! I would never be able to figure this out.

    So I tried deleting the (invisible?) space character using Excel functions but I’m not confident it worked, so I’ve decided to assign numbers 51-59 to lists 1-9. Would that work do you think?

    Best,
    Yev

    #9703
    Jeremy
    Keymaster

    All you need is to make sure you have the exact same possible values across the two tables. If you replace all the 1-digit group values from both tables with numbers starting with 5 then you could achieve that, as long as you don’t introduce another space character in front of 5 in one of the two tables

    The other solution would be to open the study table in a text editor (for example, the editor on the farm) and delete the space character in front of all the 1-digit group values

    Jeremy

    #9798
    ajessen
    Participant

    Hi Jeremy,
    I have a question regarding the distribution of the lists as well.
    I ran an experiment which had 12 lists, and lasted about 40 minutes. It was published via prolific. Since there participants often fill up quickly, I set the counter earlier than at the end of the experiments, so that not everybody signing up within the first participant doing the experiment will get list1.
    The counter was set after the training session and after they got ask to click a button for the experiment to start. I figure that there is a big chance that participants who got that far will also finish it – but i am aware that not everybody might do so.
    When checking the results, it showed that the distribution of the lists was kind of erratic. It started with 3, then 5, 7, 9, then again 3, 5, maybe 6, 10, then again 3, without going at least until 11 or 12. 1 and 2 left out.
    We changed the uploaded excel file to delete all lists that where presented already more than 2 (we wanted just 2 participant per list), but I hope there is another way to make sure, that the distribution of the lists is at least a bit more reliable.

    Here is the code:

    Sequence( "welcome" , "consent", "instruction", "practice1", "practice2","send", "trialTriangle", "start experiment", "counter", "experimental-trial", "send", "final");
    
    [all elements in between]
    
    newTrial("practice2",
      
        newText("top", "zgomot")
        ,
        newText("left", "copil")
        ,
        newText("right", "speria")
        , 
     //   newImage("triangle", "Triangle.png")
      //  .size(140,140)
        //,
    
        newCanvas("trialTriangle", 200,200)
            .settings.add( 50,0, getText("top").color("blue"))
            .settings.add(-30,140, getText("left"))
            .settings.add(120,140, getText("right"))
            .center()
            .print()
            ,
        newTextInput("response", "")
            .center()
            .log()
            .print()
            .wait(getTextInput("response").testNot.text(""))
            );
    newTrial("start experiment",
        defaultText.center().print(),
        newText("Când sunteți gata să începeți experimentul, apăsați SPAȚIU.")
        ,
        newKey(" ").wait()  // Finish trial upon press on spacebar
    );
    
    SetCounter("counter", "inc", 1);
    
    Template("draft_list_2.csv", row =>
        newTrial("experimental-trial", 
        newText("top", row.TOP).color(row.Coloring_Code=="TOO" ? "blue":"black"),
        newText("left", row.LEFT).color(row.Coloring_Code=="OLO" ? "blue":"black"),
        newText("right", row.RIGHT).color(row.Coloring_Code=="OOR" ? "blue":"black"),
        newCanvas("trialCanvas", 200,200)
            .settings.add( 50,0, getText("top"))
            .settings.add(-30,140, getText("left"))
            .settings.add(120,140, getText("right"))
            .center()
            .print()
            .log(),
        newTextInput("response", "")
            .center()
            .log()
            .print()
            .wait(getTextInput("response").testNot.text(""))
            )

    and here the link to the full experiment

    https://farm.pcibex.net/r/DYuHqf/
    it’s the demonstration link

    Thank you very much!!

    Best, Anna

    #9800
    Florian Schwarz
    Moderator

    Hi Anna,
    (Jeremy is traveling right now and may not be able to answer before a while, so I’m jumping in.)

    There unfortunately is no fool-proof way of ensuring equal number of submissions across lists in an automated way. The only method giving you full control is to post separate experiments on Prolific (with restrictions set to allow people not to take more than one) with urls to your experiment that hard-code the group/list-assignment, in the way Jeremy described above:

    “Circulate the link to your experiment ending with server.py?withsquare=0 to some participants, with server.py?withsquare=1 to some other participants, and so on, until you get 10 submissions for each list”

    Then you can make sure that each list gets to the desired number of participants.

    Hope that helps,
    Florian

Viewing 10 posts - 1 through 10 (of 10 total)
  • You must be logged in to reply to this topic.