How Many GPUs Does It Take to Run ChatGPT?

Metadata

• Author: Tom Goldstein
• Full Title: How Many GPUs Does It Take to Run ChatGPT?
• Type: #snippet✂️
• Document Tags: #tech
• URL: https://twitter.com/tomgoldsteincs/status/1600196981955100694

Highlights

• On Azure cloud, each A100 card costs about $3 an hour. That's $0.0003 per word generated.
  But it generates a lot of words! The model usually responds to my queries with ~30 words, which adds up to about 1 cent per query. (View Highlight)
• I estimate the cost of running ChatGPT is $100K per day, or $3M per month. (View Highlight)