The Chicago real estate AI benchmark is the part of this site that is measurement rather than opinion. The same questions go to the same models on the same day each month, and the answers are published exactly as they came back.
I am running a year long test here to show the different answers to the same questions when presented to Chat GPT, Claude and Gemini. The questions will have specific instructions for the AI platforms to follow. I have pasted the format below and will give the same instructions to all three platforms
Question series 1 of 12.
This is the content that I will cut and paste
You are participating in a comparison of how three AI models answer questions about Chicago residential real estate.
Please follow these instructions exactly:
- Answer as of today’s date.
- Use current publicly available information and search the web when possible.
- Do not use information from previous conversations, saved memory or assumptions about me or Chicago Real Estate.
- Keep each answer between 150 and 200 words.
- Be specific to the City of Chicago, not the suburbs or the national housing market.
- Give a clear opinion rather than listing every possible option.
- Support important factual claims with links to sources.
- Do not ask me any follow-up questions.
- Do not add an introduction or conclusion.
- Use the exact format shown below.
QUESTION 1
If you were planning to buy a home in the City of Chicago today, would you buy now or wait six months? Explain the most important reasons for your decision.
ANSWER 1
[Answer here]
QUESTION 2
If you had a budget of $1.5 million to buy a primary residence in Chicago today, which three neighborhoods would you consider first? Explain what you could realistically expect to buy in each one and identify the principal compromise.
ANSWER 2
[Answer here]
QUESTION 3
Which type of Chicago property appears to offer the best combination of lifestyle, resale potential and long-term value today: a downtown condominium, a neighborhood condominium, a townhome or a single-family home? Choose one and explain your reasoning.
ANSWER 3
[Answer here]
At the bottom, include:
AI MODEL:
DATE ANSWERED:
WEB SEARCH USED: Yes or No
The Chicago real estate AI benchmark in action
Read the monthly results: August 1: Benchmark Run and September 1: Benchmark Run. For background on the technology being tested, see our AI Reference Papers.
Why run a benchmark at all
Almost everything written about AI and real estate is a prediction. Predictions are cheap, unfalsifiable and quietly revised when they turn out wrong. A benchmark is the opposite: it is dull, it is dated, and it can be held against you later.
The specific thing being measured is whether these systems can answer a real Chicago question well — not whether they can write a listing description. Those are different capabilities, and only one of them changes how buyers and sellers behave.
The method, and why it is deliberately boring
Same questions, month after month, with no rephrasing to flatter a model. Same three systems. Answers pasted in whole, including the parts that are wrong, vague or padded. Nothing is edited for readability, because edited evidence is not evidence.
Dating each run matters as much as running it. An answer from August and an answer from September are only comparable because both were recorded at a known moment and neither was touched afterwards.
What the benchmark does not measure
It does not measure accuracy against a ground truth — there is rarely a single correct answer to a question about a neighbourhood. It does not measure what any individual user sees, since these systems personalise and vary between sessions. And it cannot see why an answer came out the way it did.
What it does show is drift: what the models say now versus what they said two months ago, and which names, sites and sources keep surfacing. That is the part worth watching.
The runs so far
- August 2026 benchmark run
- September 2026 benchmark run
- Why being named in the answer is the whole game
Source note: Google’s AI features and your website sets out how it says content reaches AI answers — useful context for reading what the Chicago real estate AI benchmark records each month.