I Tested 6 AIs with a Mexican Law Query: From Brilliant to Dangerous

View on LinkedIn

"Which artificial intelligence is best?"

It is a question I hear constantly in legal circles, where the adoption of these technologies grows day by day. To find an answer — at least within the Mexican legal context — I set out to conduct a simple yet revealing experiment.

This was not an exhaustive scientific study involving hundreds of questions, but rather a more nuanced exercise: presenting the leading AI models currently available on the market with a single legal query. A question that appears simple on its face but requires complex expertise to be answered adequately: what to do when the Mexican Institute of Industrial Property (IMPI) issues an official communication finding a likelihood of confusion with another trademark.

The choice of this question was deliberate. As an intellectual property specialist, I know it requires not only technical knowledge of trademark law, but also strategic capacity and a practical understanding of proceedings before the IMPI. It is the kind of query any trademark attorney faces on a regular basis — and one whose answer can mean the difference between losing or saving a trademark registration.

EXPERIMENT SCORES

Gemini 2.0 Experimental : 9.2/10

Claude 3.5: 8.5/10

ChatGPT O1 Pro: 8.3/10

Perplexity: 7.8/10

Grok: 6.5/10

Llama 3.2: 5.7/10

The results of this experiment revealed striking differences in the quality, accuracy, and usefulness of each model's responses, exposing specific strengths and weaknesses in each AI when it comes to legal advice.

GEMINI 2.0 Experimental: THE STRATEGIC LEADER

Gemini 2.0 stood out for its comprehensive and strategic approach. It excelled in three key respects:

1. Impeccable legal structure: It presented the available options in hierarchical order, from the least to the most drastic, including this brilliant gradation: "Argue the distinction between the marks > Limit the scope of protection > Submit a consent letter > Partially withdraw the application > Modify the mark."

2. Practical analysis: It was the only model that delved into the possibility of negotiating with the owner of the cited mark, specifically suggesting "contacting the owner of the cited mark and negotiating the obtainment of a consent letter" — an advanced strategy that the other models omitted.

3. Tactical advice: It included crucial details such as analyzing "whether the objection relates to a specific class that is not essential to your business," demonstrating a deep understanding of the commercial considerations at stake.

CLAUDE 3.5 Sonnet: THE PRECISE MINIMALIST

Claude 3.5 shone through its effective conciseness and technical precision. Its key strengths were:

1. Immediate clarity: It opened with a reassuring yet precise statement: "This does not necessarily mean you have lost the proceeding."

2. Practical focus: It provided the specific statutory deadline: "normally two months from the date of notification of the official communication" — a crucial detail that other models left vague.

It did, however, fall short of delving into more sophisticated trademark defense strategies.

CHATGPT O1 PRO: THE CAUTIOUS METHODOLOGIST

It distinguished itself through a systematic approach, albeit with some notable limitations:

Strengths:

1. Appropriate professional disclaimer: "I am not your legal advisor, but on an informational basis I can share some general guidelines" — clearly establishing the limits of its advice.

2. Clear categorization: It organized the information into well-defined sections such as "Read the official communication carefully," "Identify the distinguishing elements of your mark," etc.

Limitations:

1. Excess of generalities: For example, suggesting "include all possible documentation" without specifying which types of documents would be relevant.

PERPLEXITY: THE METICULOUS ACADEMIC

It stood out for its rigor in documentation, but with some focus-related issues:

Strengths:

1. Specific references: It cited concrete sources for each assertion, such as "according to source [1]..."

2. Specific deadlines: It correctly mentioned the "two-month period to respond" and the existence of an "additional extraordinary two-month period."

Weaknesses:

1. Excess of citations: The abundance of references disrupted the flow of practical information.

GROK: THE IMPRECISE GENERALIST

Although it provided basically correct information, its response was problematic in several respects:

1. Technical inaccuracies: For example, it failed to specify the exact deadlines for responding.

2. Confused structure: It mixed basic concepts with advanced strategies without a clear hierarchy.

LLAMA 3.2: THE BEGINNER

It showed the most significant limitations:

1. Incomplete information: It stated an incorrect deadline of "30 days" when the correct period is two months.

2. Superficiality: Its recommendations were too general to be of practical use.

COMPARATIVE RISK ANALYSIS

1. Deadline Risks:

• Llama 3.2: HIGH RISK for stating an incorrect deadline

• Gemini 2.0, Claude 3.5, ChatGPT O1 Pro: LOW RISK for correctly specifying the deadlines

• Perplexity: LOW RISK with source documentation

• Grok: MEDIUM RISK due to ambiguity regarding deadlines

2. Strategy Risks:

• Gemini 2.0: LOW RISK for offering multiple viable strategic options

• The others: MEDIUM RISK for omitting important strategies

3. Omission Risks:

• Gemini 2.0, ChatGPT O1 Pro: LOW RISK for comprehensive coverage

• The others: MEDIUM RISK for omitting important aspects of the process

CONCLUSION

This detailed analysis reveals a clear hierarchy in the ability of AI models to handle specialized legal queries:

1. Undisputed leader: Gemini 2.0, for its combination of technical precision and strategic vision.

2. Second tier: Claude 3.5 and ChatGPT O1 Pro, for their balance of precision and practicality.

3. Third tier: Perplexity, for its academic yet useful approach.

4. Basic tier: Grok and Llama 3.2, with significant limitations.

Nevertheless, even the best AI showed limitations that underscore the importance of human professional judgment. Current technology is a valuable support tool, but it is not a replacement for the specialized attorney.

DETAILED SCORE BREAKDOWN

The scores assigned are based on five fundamental criteria, each evaluated on a 10-point scale:

Gemini 2.0 (9.2/10)

• Technical accuracy: 9.5/10 (precise information on deadlines and procedures)

• Legal strategy: 9.5/10 (multiple viable and well-ordered options)

• Practicality: 9.0/10 (directly applicable advice)

• Structure: 9.0/10 (clear and logical organization)

• Risk management: 9.0/10 (appropriate warnings on critical points)

Claude 3.5 (8.5/10)

• Technical accuracy: 9.0/10 (correct information, though less exhaustive)

• Legal strategy: 8.0/10 (basic options well explained)

• Practicality: 9.0/10 (direct and applicable advice)

• Structure: 8.5/10 (clear organization, though simpler)

• Risk management: 8.0/10 (basic but sufficient warnings)

ChatGPT O1 Pro (8.3/10)

• Technical accuracy: 8.5/10 (correct information with some generalities)

• Legal strategy: 8.0/10 (more conservative approach)

• Practicality: 8.5/10 (good practical advice)

• Structure: 9.0/10 (excellent point-by-point organization)

• Risk management: 7.5/10 (appropriate disclaimer but limited warnings)

Perplexity (7.8/10)

• Technical accuracy: 8.5/10 (well-documented information)

• Legal strategy: 7.0/10 (basic options without strategic depth)

• Practicality: 7.5/10 (useful information but overly academic)

• Structure: 8.0/10 (clear organization, though interrupted by citations)

• Risk management: 8.0/10 (good source documentation)

Grok (6.5/10)

• Technical accuracy: 7.0/10 (basically correct but imprecise information)

• Legal strategy: 6.0/10 (limited options)

• Practicality: 6.5/10 (overly general advice)

• Structure: 6.5/10 (confused organization)

• Risk management: 6.5/10 (insufficient warnings)

Llama 3.2 (5.7/10)

• Technical accuracy: 5.0/10 (error on statutory deadlines)

• Legal strategy: 5.5/10 (very basic options)

• Practicality: 6.0/10 (overly general advice)

• Structure: 6.0/10 (basic organization)

• Risk management: 6.0/10 (insufficient warnings)

Evaluation Methodology:

1. Technical accuracy: Assesses precision regarding deadlines, procedures, and legal terminology.

2. Legal strategy: Measures the quality and range of strategic options proposed.

3. Practicality: Evaluates the direct applicability of the advice provided.

4. Structure: Analyzes the organization and clarity with which information is presented.

5. Risk management: Examines the quality of warnings and risk-related considerations.

These scores reflect the current performance of AI models on specific legal tasks, though it is important to note that the technology evolves rapidly and these evaluations could change within a matter of months.