Calculating Automation TAM for LLMs
The Importance of TAM
One of the earliest and most crucial steps in the investment process is understanding the market and revenue opportunity. This is often referred to as TAM—Total Addressable Market.
The absolute value and the rate of change of TAM are key indicators for investment, and they can even outweigh other factors such as the specific characteristics of a company, its competition, and management in most cases. (Small market and awesome investment? Small upside!)
The AI Opportunity
Assessing the AI opportunity requires a TAM analysis. While AI is very general purpose with reach beyond a single market or TAM, one of the more obvious and largest sub-markets is in using LLMs and adjacent tech for workforce automation, and we have the data and tools to put a number on that.
Estimating LLM Automation TAM
The TAM for labor automation through LLMs is calculated by identifying tasks that can be automated. This is then multiplied by the monetary wage employers pay for these tasks and the total employment in the sector, then discounting that total TAM by some ROI.
For instance, if legal and accounting jobs could be fully automated, the TAM would equal the total wages paid in these sectors, adjusted by a discount rate for ROI. The discount rate accounts for the fact that employers will not pay a software subscription equal to the total employment cost.
Gathering Data
To perform this analysis, we need comprehensive data. Here are the key data sources:
O*NET Task Statements list (link): Provides a detailed list of task descriptions for every profession in the US economy - a total of 19,265 tasks!
BLS’s Occupational Employment and Wage Statistics (link): Gives the wage data for every profession (>900 professions).
O*NET-SOC 2021 National Employment Matrix (link): Serves as a bridge between the first two data sets, effectively mapping tasks to wages. This “crosswalk” is a helpful bridge between the two, effectively grouping granular profession classifications like “Biofuels Production Managers” and “Biomass Power Plant Managers” into a single profession called “Industrial production managers” of which the BLS uses to bucket the wage data. This narrows total professions and therefore wages down to about 780.
Pure Automation with GPT-3.5-Turbo
How can we possibly sift through 19,000 task statements, and reason about which can and cannot be fully automated by LLMs and related technology? The answer is simple: Automate it with an LLM.
We’ll use the OpenAI API and the gpt-3.5-turbo model to
- Look at every position and task in the US economy
- Assume that LLMs can leverage audio and visual data to finish tasks effectively
- Tell us, one by one, with a simple ‘yes’ or ‘no’ answer, whether or not the task can be fully automated. Below is our code. The prompt is in the ‘primer’ and ‘question_to_ask’ variables.
import pandas as pd
import openai
import time
openai.api_key = "sk-xxx"
primer = "You are a helpful analyst. Assume LLMs advance to the point where they can retrieve factual information with a high degree of confidence and an extraordinarily low error rate. Assume software can leverage LLMs combined with audio and visual data to finish work tasks effectively, at par or more effectively than humans."
# initialize an empty list to store the answers
gpt_answers = []
# loop through each row in the DataFrame
for index, row in tasks_df.iterrows():
# delay of n seconds to stay below OpenAI's max requests per minute
time.sleep(0.02)
# retrieve the content from the 'job and task' column for the current row
job_and_task_content = row['job and task']
# construct the question
question_to_ask = f"I will show you a job and a task associated with that job. Please tell me if you believe that task can be 100% automated given our assumptions. Only respond 'yes' or 'no'. {job_and_task_content}?"
# use the ask function to query the GPT-3.5-turbo model and store the answer
for model in ["gpt-3.5-turbo"]:
res = openai.ChatCompletion.create(
model=model,
messages=[
{"role": "system", "content": primer},
{"role": "user", "content": question_to_ask}
]
)
answer = res['choices'][0]['message']['content']
# add the answer to the list
gpt_answers.append(answer)
#print(row['job and task'], answer)
# add the list of answers as a new column in the DataFrame
tasks_df['GPT_answer'] = gpt_answers
A couple of hours and about $7 in API fees later, we are armed with data to proceed to our core analysis.
The Calculation
Our datasets categorize tasks as either Core or Supplemental. We'll assume that 80% of a profession’s wage is attributed to Core tasks and 20% to Supplemental tasks.
Now, we’ll take every job and task in the economy, assign a dollar value to that task, and sum to an LLM automation TAM based on the total number of laborers performing that task:
import numpy as np
# calculate the part of 'automatable wages' coming from 'core_tasks'
core_wages = np.where(
summary_analysis['core_tasks'] != 0,
(summary_analysis['A_MEDIAN'] * summary_analysis['automated_core_tasks'] / summary_analysis['core_tasks']),
0
) * 0.8
# calculate the part of 'automatable wages' coming from 'supplemental_tasks'
supplemental_wages = np.where(
summary_analysis['supplemental_tasks'] != 0,
(summary_analysis['A_MEDIAN'] * summary_analysis['automated_supplemental_tasks'] / summary_analysis['supplemental_tasks']),
0
) * 0.2
# calculate 'automatable wages' by summing the above two
summary_analysis['automatable_wages'] = core_wages + supplemental_wages
# calculate 'total_automatable_wages' by multiplying 'automatable_wages' by 'TOT_EMP'
summary_analysis['total_automatable_wages'] = summary_analysis['automatable_wages'] * summary_analysis['TOT_EMP']
Some conclusions
Total TAM is very large. Fully automate-able wages in the economy from advanced LLMs sums to $2.7 trillion. We can infer that employers will spend some amount up to but not exceeding that number, and will seek an ROI. Assuming 15% IRR on LLM-related investments over a five year period, LLM automation TAM is about $1.3 trillion today.
What’s most interesting are the implications for specific industry verticals ripe for LLM automation. Management, customer service reps, and software developers are most likely to have their task sets automated, and potentially represent the largest TAMs for vertical LLM software products. Startups targeting automation should consider these verticals are ripe for disruption.
Feel from to run my analysis yourself. I’ve linked to the most recent datasets above (mine are a little stale by a few months). The datasets I used and code I wrote are here. Remember, OpenAI’s models are not deterministic, so when you run the code again, results won’t be apples to apples.