Meta AI (Facebook):

Mephisto Dataset Collection Tool

Aim: to leverage the Mephisto tool. Areas of interest:

• Entirely novel and complex tasks that may require rounds of iteration to design and experiment with.
• Tasks that involve live interaction between multiple workers in some form, either collaborative or competitive.
• Tasks that benefit from having a model-in-the-loop to assist workers.
• Tasks that require intricate worker filtering specifications, for pairing, ensuring specified groups, or otherwise.
• Tasks that can be built on existing frameworks, but may benefit from incorporating support for that framework into Mephisto.

Funding: up to $37,500

https://ai.facebook.com/research/request-for-proposals/mephisto/

Dynabench Data Collection and Benchmarking Platform

submissions along the theme of “rethinking benchmarking” that leverage the Dynabench platform. Areas of interest:

• New tasks or benchmarks that would benefit from having a dynamic (adversarial) component.
• Experiments that examine concrete questions about the human-and-model-in-the-loop paradigm for data collection or evaluation.
• Proposals for moving Dynabench beyond natural language processing and/or beyond the English language.

Funding: up to $55,000

https://ai.facebook.com/research/request-for-proposals/dynabench/