Methodology for the Academic Career Interest Badge in Cortex

Standardizing how academic intent is detected in personal statements through Thalamus AI Insights to support fair, efficient, and equitable application review. 

‍

Original publication: September 24, 2025 

Last updated: October 13, 2025 

At Thalamus, our guiding philosophy has always been to ensure that technology enhances, rather than replaces, the human elements of graduate medical education (GME) application review and selection. Through Artificial Intelligence: The Thalamus Way, we’ve articulated our commitment to developing artificial intelligence that is transparent, reliable, and designed to empower holistic review while honoring the judgment of faculty and program leaders and highlighting core components of candidate applications.  

The new Academic Career Interest feature described in the methodology below builds upon this philosophy by offering programs a consistent signal to surface applicants who express or imply aspirations in teaching, research, and scholarly leadership (and vice versa).  Just as our transcript normalization methodology has advanced efficiency and comparison in evaluating core clerkship performance, this new initiative brings a structured approach to another dimension of application review. 

This work is the first joint feature developed following Thalamus’s acquisition of Medicratic. The “Academic Career Interest” badge leverages methods first piloted in Medicratic’s Halsted, then refined through thousands of real-world application reviews, and now embedded into Cortex as part of Thalamus AI Insights. By integrating these innovations into our platform, we are creating tools that not only support faculty in holistic review, but just as importantly, surface academic career interest.  This provides applicants with confidence that the values and goals they choose to express in their personal statements are being consistently highlighted and fairly recognized in the review process. 

As with all of our AI initiatives, the Academic Career Interest feature is being introduced in beta with transparency, continuous validation, and community feedback at its core. We see this as an iterative step forward in responsibly harnessing AI to reduce interpretive variability, mitigate bias, and elevate the dimensions of applicant narratives that matter most to academic medicine. In doing so, Thalamus reaffirms its commitment to connecting the docs—bringing together programs and applicants through technology that is fair, inclusive, and grounded in the values of the GME community. 

‍

I. Background 

In residency and fellowship recruitment, programs assess not only applicants’ academic readiness, but also their long-term professional goals. University-based, research-intensive, and teaching-focused programs may place particular emphasis on candidates committed to scholarship, education, and leadership. For such programs, discerning academic career interest may represent an important element for holistic review of applicants through their selection process. 

Historically, the personal statement has acted as a primary narrative vehicle through which applicants may describe their career goals. Unlike exam scores or recommendation letters, it can offer a direct articulation of motivation in the candidate’s own words, often revealing whether they envision themselves as a future educator, researcher, or academic leader.  

However, interpretations of these narratives may vary widely. Some applicants state their aspirations explicitly (e.g. “I have a career interest in becoming an academic anesthesiologist.”), while others embed them indirectly within experiences or values. This variability can complicate effort to identify academic career interest reliably, particularly when faculty must review large volumes of applications under time constraints. 

To address this concern, Thalamus developed an AI Insights tool for Cortex, its technology-assisted application screening platform designed to facilitate holistic review.  This feature utilizes a large language model (LLM) analysis to identify key characteristics within personal statements to support more objective and data-driven decisions.  The tool is part of a broader initiative to highlight and aggregate key data for programs reviewing thousands of applications through processes that highlight applicant achievements. 

The first measured characteristic to be included in Thalamus AI Insights is a badge to denote “Academic Career Interest,” which detects expressed and/or implied language of academic career ambition. By surfacing these signals in a consistent manner, the tool enables programs to incorporate academic career interest into the holistic review process with greater clarity and efficiency, thereby highlighting applicants as well. 

‍

II. Introduction 

This document describes the conceptual foundation and operational design of the “Academic Career Interest” badge in Thalamus AI Insights. The feature provides a systematic assessment of whether a personal statement conveys expressed and/or implied language of academic career ambition and/or desired commitment to a career in academic medicine.  In other words, this may signal a consistent, auditable signal of certain common indicia of academic career ambition, including common indicia that may be more nuanced or less explicit in ways that are easily missed in the manual review process.  Its role is supportive: to generate a consistent, auditable signal that can inform review processes, not to supplant faculty expertise or judgment. 

‍

III. Objectives 

The primary objective of the tool is to standardize detection of academic career interest in personal statements, reducing interpretive variability across reviewers. Historically, sentiment shared in personal statements have been an informative source of information about applicants’ long-term career goals, but interpretations may be subject to reviewer workload, time pressure, and individual bias. The tool seeks to mitigate these sources of inconsistency by offering programs a structured, reproducible assessment of academic career orientation, and highlighting applicants who have chosen to emphasize that element of themselves in an application. 

‍

IV. Methods 

1. Data Source and Preparation 

The initial model was prepared using a random sample of 500 personal statements for the ERAS 2024 (Sept 2023- March 2024) residency application season, accounting for a diversity of applicant types (US-MD, US-DO, IMG), medical schools and applicant characteristics (geography, self-identity, etc.)  The full body of each personal statement was processed without truncation or summarization. Minor preprocessing included character normalization and removal of formatting artifacts 

2. LLM Processing 

The machine-readable text representations were processed using an LLM, specifically the GPT-4o-mini model hosted on Microsoft Azure’s OpenAI service.  This solution was selected given Thalamus utilizes Microsoft Azure for cloud hosting, which promotes overall data and model security. Through Thalamus’s contractual enterprise relationship with Microsoft, neither the data input, nor the trained model is publicly utilized or used to train any other GPT solution outside of Thalamus.  This solution was fully vetted by Thalamus’s data security and compliance teams.   

A validated, fixed prompt was used to instruct the model to classify career interest in academic medicine based on defined conceptual criteria (examples in table below). This approach leveraged prompt engineering rather than parameter modification, ensuring consistent and reproducible outputs while maintaining alignment with defined task requirements. 

3. Conceptual Definition of Academic Career Interest 

Within the context of Thalamus AI Insights, “Academic Career Interest” is defined as whether an applicant’s personal statement conveys, either explicitly or implicitly, that they envision a professional trajectory in which teaching, research, or scholarly leadership are central components. Because the tool analyzes the personal statement only, its determinations are limited to what applicants choose to describe in this component of their application. 

“Academic Career Interest” may be conveyed directly, through a clear statement of plans to pursue an academic career, or indirectly, when teaching or research is positioned as central to the applicant’s future goals. The detection of “Academic Career Interest” in a personal statement does not guarantee that that applicant will ultimately follow such a path; it reflects only the aspirations they chose to emphasize at the time of application. 

The presence of this badge indicates that our system was able to clearly identify language in the personal statement, suggesting an interest in an academic or research-oriented career path. The absence of the badge simply means that such intent was not identifiable from the statement. Some applicants may express these goals in other parts of their application or in less overt ways. 

Note: Thalamus AI Insights is not designed to capture every passing mention of teaching or research or any other phrasing. Instead, it focuses on statements where these activities are presented as integral to the applicant’s envisioned career. In this way, the output is intended to approximate the interpretive standard an experienced faculty reviewer might apply when reading a personal statement. Representative phrases that would typically meet or not meet this threshold are shown in Table 1, which provides a set of illustrative cases modeled on typical language found in personal statements. 

Each statement level output includes a binary classification (“Yes” or “No”) and a brief rationale summarizing the evidence for the classification.  While the “Academic Career Interest Badge” evaluates entire personal statements rather than isolated passages, these excerpts illustrate the kinds of language that would generally meet (=Yes) or not meet (=No) the interpretive threshold for signaling academic career interest. 

These outputs are stored with metadata for versioning, traceability, and quality control. 

‍

4. Manual Review of Model Outputs 

To increase accuracy and reliability, the output of the initial sample “Academic Career Interest” was manually verified by human reviewers.   

Discrepancies between AI and human review were analyzed to identify patterns of misclassification and opportunities for refinement. Attention was paid to borderline cases, such as statements that described positive experiences with teaching or research but did not clearly articulate long-term commitment. Insights from this process informed iterative improvements to the model prompt and overall methodology. 

Since the initial model, a stratified random sample of personal statements (from tens of thousands of applications reviewed in either Halsted and Cortex) have been selected for review by Thalamus subject matter experts (SMEs) in medical education. Reviewers independently assessed each statement for “Academic Career Interest”, and their determinations were compared to the model’s outputs. 

5. Display of “Academic Career Interest” badge in Cortex 

As of September 24^th 2025, Cortex displays a section on each applicant summary labeled “AI Insights.” The tool has been marked as “beta” during its pilot phase.  Also included is a direct link to this methodology document so that program reviewers may easily refer to this documentation as needed and to promote transparency.  

For each applicant, Thalamus AI Insights reviews the personal statements using the previously described methodology to assign a binary of “Yes” or “No” in regard to “Academic Career Interest.”  For applicants, whose personal statements are assigned a value of “Yes,” an “Academic Career Interest” badge will be visible. 

For an assigned value of “No,” no badge will display. 

Program reviewers may also scroll over the “Academic Career Interest” badge, which will display a tool tip explaining the rationale for the badge display: “Personal statement includes expressed and/or implied language indicating career interest in academic medicine.” 

‍

‍

6. Quality Assurance and Ongoing Validation 

Quality checks included error categorization, cross-verification, and reviewer agreement benchmarking. Ongoing validation occurs in real time during each application cycle, supported by continuous program feedback and internal monitoring to improve accuracy, reliability, and relevance.  The model was originally deployed in Medicratic Halsted’s product in the 2024 ERAS application season, and prior to deployment in Cortex had reviewed 25,000+ applications.  The model was retrained and revalidated, alongside an updated data pipeline for inclusion in Cortex.  Since its initial release, the tool has been regularly spot-checked with a random sampling of personal statements and human review to ensure consistency and reliability. 

 V. Limitations 

While Thalamus AI Insights and the “Academic Career Interest” badge provide a systematic framework for evaluating residency and fellowship personal statements, there are apparent limitations.  Most notably, the writing present in personal statements and the subsequent reading/analysis of such by human or AI review are inherently subjective. Thereby, even the most sophisticated models cannot fully capture the nuance of individual applicant aspirations. Some applicants may understate or omit academic goals despite genuine interest, while others may reference teaching or research in ways that are difficult to categorize. The binary nature of the model output provides interpretive clarity, but also simplifies the complex narratives present in personal statements. As with all elements of the application, the “Academic Career Interest” badge should therefore be understood as one factor among many in a holistic review process, and not as a definitive measure of applicant potential. 

Finally, cultural and linguistic variation in how applicants describe career goals may affect interpretation. Bias mitigation was incorporated into model design by using a representative and generalizable subset of applicant personal statements and characteristics, including self-identification and medical school types (US MD, US DO, US and non-US citizen international medical school graduates (IMGs, etc.).  Additional work is anticipated to compare the badge and language patterns to outcome (i.e. pursuing a career as an academic physician), while also noting that this interest may change over time. 

These limitations highlight the inherent challenges in analyzing personal statements for both humans and artificial intelligence. As the Thalamus AI Insights continues to be developed, we will work through perceived, relative and absolute limitations in this methodology to innovate based on user feedback, as well as to continue advances in the development of the tool and broader technology at-large. 

Hallucinations and Model Misinterpretation 

As with all large language models, limits also include a risk of hallucinations — where the system may misinterpret, over-generalize, or fabricate structure from ambiguous transcript data. While safeguards are in place, including prompt engineering, human validation, and error reporting workflows, no model is immune from misclassification. Programs should therefore treat academic career interest outputs as decision-support only and always confirm critical details against the personal statement when uncertainty arises. 

VI.  Frequently Asked Questions (FAQs) 

General FAQs

Q1. Why did Thalamus AI Insights start with a badge on “Academic Career Interest?” versus other applicant criteria? What does it mean that the tool was released in beta? 
A1. Thalamus’s AI development philosophy and strategy centers around product inclusivity and care in assessing the impact of the tools that are incorporated into our software.  Given AI in GME recruitment is still mostly novel to the academic medicine community, we wanted to start with this metric given it has not been historically used as a major determinant in whether an applicant is/is not selected for an interview.  Additionally, there are many other elements in an application that could also serve to support this analysis, including research, experience, presence of other degrees (PhD, MPH, etc.) and more. Thereby, it allows programs to start to see how AI can be used to enhance their holistic review of applications without substantially affecting the core review process.  The “Academic Career Interest” badge was released in beta to ensure programs reviewing applications are aware that the tool is new and to use its output alongside the review of official application materials and documents. 

Q2. Will other badges be included in Thalamus AI Insights during the ERAS 2026 residency recruitment season?  
A2. Currently, the “Academic Career Interest” badge is the only planned feature for this season.  Our goal is to assess feedback from the community and define other key criteria that are effective, impactful, and equitable across the breadth of the applicant pool.  This includes conversations with programs, applicants, and medical schools alongside working groups and advisory boards.  For those interested in providing feedback or testing additional features, please contact our Product Experience team by submitting a Thalamus Support Request Form.  

FAQs for Programs 

Q1. Does the “Academic Career Interest” badge replace my judgment as a faculty reviewer? 
A1. No. The badge is designed to provide a consistent and auditable signal, not to replace faculty expertise. It highlights when an applicant’s personal statement conveys academic career interest, and highlights certain indicia of academic interest, but the ultimate interpretation of that context and weight in decision-making remains with the program. 

Q2. How accurate is the tool, and how do I know it isn’t introducing bias? 
A2. The model was developed using a representative sample of applicant personal statements across diverse applicant types (US-MD, US-DO, IMG, etc.). Outputs are regularly verified from a bias perspective and spot-checked by subject matter experts, and quality assurance processes are built into each cycle. Transparency and ongoing validation are central to Thalamus’ approach. 

Q3. How is this different from just reading the personal statement myself? 
A3. Faculty review is always encouraged, but time and volume pressures can make it difficult to interpret nuanced statements consistently across thousands of applications. The badge standardizes this element and provides context. This should not displace the need to review personal statements but rather supplement their review. 

Q4. Can I see the rationale behind a badge? 
A4. Yes. Programs can hover over the badge to see a short explanation of why it was applied. This ensures transparency in how the tool interprets the applicant’s personal statement and allows reviewers to contextualize its use within their overall evaluation. 

Q5. How does this align with Thalamus’ broader AI strategy? 
A5. The “Academic Career Interest” badge continues Thalamus’ mission to apply AI responsibly, supporting fairness, efficiency, and transparency in graduate medical education. It builds on our methodology for transcript normalization and our integration of Medicratic’s innovations, extending AI insights into new dimensions of applicant review. 

‍

FAQs for Applicants 

Q1. If I want an Academic Career Interest badge, do I need to phrase my goals in a specific way? 
A1. No. The tool is designed to recognize both explicit and implicit expressions of academic interest. If your personal statement describes teaching, research, or scholarly leadership as central to your envisioned career, the system will likely recognize that intent and display the badge across programs. 

Q2. Will not receiving an Academic Career Interest badge affect my chances of being selected for interview and/or not matching? 
A2. The badge is one data point in a much larger holistic review process. Many applicants pursue non-academic career paths and are highly valued by programs.  There are many programs that also are looking for applicants who have expertise in clinical knowledge/practice or will end up working in practice environments outside of academia. The badge simply helps programs that wish to identify candidates with academic career aspirations.  The tool has been released in beta, and programs have been provided guidance as to what the badge means to supplement their review.  Subsequent work is being planned to provide an applicant portal for the 2027 ERAS season (starting July 2026). 

Q3. Can programs filter my application in or out based on the Academic Career Interest badge? 
A3. No. The badge is a static indicator that highlights whether a personal statement conveys academic career interest. Programs cannot filter, exclude, or auto-screen applicants based on this badge. It was created to provide additional context for reviewers, not to determine selection decisions. 

Q4. What if my career goals change after I submit my personal statement? 
A4. The badge reflects only what you chose to emphasize at the time of your application. It is not a prediction of your ultimate career path, but rather a structured way for programs to interpret your statement consistently. 

Thalamus is working on additional tools for applicants with regard to this feature to provide better understanding and guidance.  Overall, we would not recommend any modification in personal statement composition regarding the language you choose or writing it in a way different than you would for human review. 

‍

VII. Conclusion 

The novel “Academic Career Interest” badge determined through Thalamus AI insights is a prototype released in beta, designed with similar Thalamus features with a focus on design thinking, product inclusivity, accessibility, transparency and equity.  The tool contributes to graduate medical education recruitment by introducing a standardized, transparent approach to interpreting personal statements for evidence of academic career orientation.  

When integrated responsibly, it enables programs to surface an important dimension of applicant motivation with greater efficiency and consistency. The tool should be understood as one element in a broader evaluative framework. Faculty expertise, contextual judgment, and holistic review remain central to fair and effective recruitment.  

As with all innovation, this feature will continue to evolve with the input of the GME community. We invite program directors, applicants, and other stakeholders to provide feedback, challenge assumptions, and partner with us in ensuring that technology strengthens fairness, inclusivity, and transparency in the transition to residency and fellowship. Our goal remains to support the academic medicine community in making well-informed, equitable decisions about application review and selection.