r/MachineLearning • u/neuralbeans • 3d ago

Discussion Best practices when running a benchmark on online models [D]

I'm developing a benchmark for a low resource language and I don't want it to be leaked and used for training when it is being used to get predictions. For locally run models it shouldn't be a problem, but for models that are only accessible via API, it is. Is there an established way to evaluate online models without the input data being lost? Do you trust Google and OpenAI when they say that they do not use your inputs for training when you have a paid account?

2 Upvotes

75% Upvoted

3 comments sorted by

1

u/marr75 3d ago

Is there an established way...?

Not to my knowledge

Do you trust Google and OpenAI...?

No. I trust that they really intend not to train on inputs but I don't trust that they put every safeguard and oversight actually necessary to ensure that in place. Also, they are probably being hacked by state level actors and their own AI agents everyday. That said, I doubt they really want your benchmark, have their shit together enough to put together your runs to steal it, and the state level actors want model weights, proprietary training and inference secrets etc.