MetaVQA

We first modify a question-type-dependent context-free grammar (CFG) according to the scene graph. Then, we randomly sample a CFG tree and instantiate the referral in the question template by traversing leaf nodes according to the tree.Meanwhile, we compile a set of functional programs for all <o> token bottom-up. Each program enforces constraints from child nodes and filters the objects in the scene into an object set that satisfies these constraints. Once a program finishes, its return value informs the next program's constraints. Once all programs are terminated, the final set contains all objects grounded by the referral, and this set is post-processed to retrieve the final answer.

We conduct benchmark experiments on VQA tasks. The model receives questions and corresponding visual data, which may consist of either a single frame (Static) or five frames (Dynamic and Safety) captured from six different views. The model is trained to predict the correct answer using token strings, and it is evaluated with questions from all three supertypes. For detailed explainations, please refer to the paper. Note that we are continue to expand this baseline to more models and more tasks.