A reminder that you get out what you put in.
While getting ready for work and reading the morning news, I found a story about how Madden 2015, a football video game for those not familiar, was able to predict that exact score of the Super Bowl and not only that, but how it would happen with the Patriots coming back at the end of the game for a win. It even predicted scary close stats for many players. You can read the article here.
How is it possible that Madden 2015 was able to do so well with it's predictions?
It simply has an accurate model. EA Sports spends tens of millions of dollars each year on the development of the Madden series of games. Over time they have developed an ever more sophisticated model of what a player is and how players interact. They know what stats are important to track and how those statistics interact with each other.
In their early days they may have had a few stats like strength, speed, and accuracy. Now I'm sure they are collecting things like vision, awareness, handling pressure, how many times a running back breaks left vs. right, and many more. They have reached a whole new level as far as what the software does with those numbers. EA Sports has been working on their model for over 25 years now, and at this point, it should be accurate.
Relating this to Simulation, it is always important to remember that the results you get out are as accurate to the data that you put in. It's easy to overlook and to feel safe. Some companies, particularly those in the aerospace industry, spend millions of dollars yearly to gather and collect data for calibrations of their simulations. You don't have to be that extreme, but some gut checks along the way can help you do a tolerance analysis.
- How close are my loads represented to what we would see in real life? They are distributed a little more evenly than we expect in operation. There could be up to a 10% difference.
- How closely do my fixtures represent reality? The fixtures I used represent true rigidity. In reality there would be some flexibility, but I don't expect more than a 5% difference in results from that.
- How much do my material properties vary? I used a general average value, but variances can range in the +/- 10%.
If we ran an analysis and had answered these questions in our head, we could add these up and say that our model is within 25% of the true expected value. Now if we can attribute some of these to having opposing effects, say the representation of the load increases stress while the fixtures decrease it, then its even closer than that.
Most people use factors of safety of at least 3 in their designs, so 25% is still covered by that, however it will never be a bad thing to know how good your model is, especially when you are making design decisions from it. The next time your run a simulation, ask yourself, "how accurate is my model?" It may not be at EA Sports level, but that doesn't mean you can't find ways to improve it.
By: Brandon Donnelly, Simulation Applications Engineer
P.S. For another example of calibrating models, see the article here about simulating combustion engines here.