How Many GPUs Does It Take to Run ChatGPT
- Author: Tom Goldstein
- Full Title: How Many GPUs Does It Take to Run ChatGPT?
- Category: articles
- Document Tags: #tech
- URL: https://twitter.com/tomgoldsteincs/status/1600196981955100694
Highlights
- On Azure cloud, each A100 card costs about $3 an hour. That's $0.0003 per word generated.
But it generates a lot of words! The model usually responds to my queries with ~30 words, which adds up to about 1 cent per query. (View Highlight)
- I estimate the cost of running ChatGPT is $100K per day, or $3M per month. (View Highlight)