That's a good question! There's some pretty easy math involved that you can try yourself.
Let's use this week's challenge as an example!
Win 77.9528x, twice in a row
Hit 20.16 as a roll number
What are the basic probabilities for these? Well, 77.9528x has a 1.27% (0.0127 probability) win chance, and rolling any given number is 1/10000. So we get:
(0.0127)^2 = 0.00016129, so inverting it (1/0.00016129), we get 6200 rolls
(1/10000) = 0.0001, so inverting it (1/0.0001) we get 10000 rolls
So the expected roll count is 6200 + 10000 = 16200 rolls.
That gives you an estimate of the average number of rolls people will take to complete the challenge...but the cost can become unpredictable when higher multipliers come into play. In those cases, variance can incur a much greater cost on you than the house edge - so be careful with which challenges you try! I generally stick to challenges that leave a significant profit margin between the cost and the expected prize.
It really does pay to do the math in advance; last week's challenge had an expected cost of about 40k and carried high risk of variance, making it so difficult that only 29 players finished it. That means they split the base prize of 0.01 - that's only 34k a person, which means on average, they actually lost money. And that's not even counting the people that failed to complete the challenge after sinking a lot of money into it!