Welcome and thank you for participating in our data collection! On the next page, you will write 15 unique commands to a robot in 5 virtual household scenes, i.e., 3 commands per scene.
A personal assistant robot can navigate the virtual environment, pick up and place objects. It perceives the environment with its camera. Please see the video below of how the robot operates in the environment.
Video:
Rules:
When giving commands, assume the robot is as intelligent as a human.
When giving commands, assume the robot starts in the same position in each scene.
Each command is separate and should not refer to any previous command. Assume the robot resets to the starting position for each command.
Each command must include picking up and placing objects and moving between rooms. Do NOT both pick AND place in the starting room.
Only refer to objects, furniture, and rooms in the scene. Refer to them in a way that best describes them, like using their labels (objects only), generic names, attributes, what is inside a room, and/or spatial relation to others, etc.
Only command the robot to pick up a labeled object a human can pick up with one hand. For example, the TV is labeled, but a human cannot pick it up with one hand.
Some object labels are incorrect; please do not use incorrect labels to describe objects e.g., “floor” for a lamp.
Do NOT command to rotate an object or lay it on its side.
Do NOT pick up or open/turn on any of the following objects in your commands, even if they exist in the scene. You can refer to them in relation to other objects. For example, “Pick up the bat next to the laptop" but not "Pick up the laptop."
Laptop
Drawers (Do NOT place anything inside)
Lamp (sometimes mistakenly labeled “floor” or “desk”)
TV
Desk
Examples:
“Go to the living room and grab the purple pillow on the couch, then go to the office and place it on the chair.”
“Go to the living room and pick up the basketball next to the table, then go to the can in the same room and drop it inside.”
“Run to the bedroom and grab my phone off the bed, then bring it back to me in the kitchen.”
“Take the baseball bat next to the nightstand in the bedroom and go to the office then put it by the white chair.”
“Find the bowl that’s on the right of the apple and take it to the office that has the laptop. Put the bowl on the blue table, specifically next to the book.”
FAQ:
Q: Can I refer to specifics in the scene, like the Vincent van Gogh painting on the wall?
A: Yes.
Q: Can I refer to objects/furniture that do not show labels when I hover the cursor on them, such as a desk?
A: Yes, as long as you do not tell the robot to pick them up. For example, you can tell the robot to place an object on a desk but not to pick up the desk.
Q: Can I pick up an object and not place it, e.g., "Pick up the remote and wait near the sofa"?
A: No.
Q: Can I throw or smash objects?
No. Please only pick and place.
Q: Do I have to explicitly specify rooms in my commands?
No, they can be implied. Your command must instruct the robot to move across rooms. You can do that either implicitly, e.g., "Put the pillow by the couch.", where the pillow and the couch are in separate rooms, or explicitly, e.g., "Go to the office and take the pillow to the couch in the living room."
Q: Can I use the room the robot starts at in my commands?
A: Yes, as long as the robot also goes to other rooms and picks/places in the other rooms. E.g. "Take the apple and put it on the living room table", or "Go to the living room, take the ball, and put it on top of the office desk", the office being the starting room.
Q: Do I have to perform the tasks in addition to giving commands?
A: No. You can if it helps you brainstorm.
Q: Is any personal information collected?
A: No.
You can use the Instructions button at the bottom of the next page to review the instructions and video. You will NOT lose your progress.