Assessment data shows that highly selective universities are doing a worse job developing talent ... [+]
Two studies from firms deeply enmeshed in the evaluation of technical talent suggest that elite universities are failing short when it comes to developing the technical capabilities of their students.
The first study comes from CodeSignal, a leading technical interview and assessment platform. Their 2024 University Ranking Report examines the degree to which graduates of universities exhibit the technical skills that employers seek. According to the report, when schools are ranked by objectively measuring their students’ technical skills, it is evident talent comes from everywhere, not just the schools traditionally recognized as top engineering schools.
CodeSignal is well-positioned to analyze the capabilities of recent graduates, having worked for years to develop standards for assessing skills as part of the hiring process for leading tech companies. In their study, they look at how job applicants from different colleges performed on technical skills assessments using the General Coding Framework. As it is safe to assume that job applicants are taking the skill assessments because they want the jobs in question, their performance on those assessments will likely be an accurate measure of their current level of achievement.
The results are clear: schools that are the most selective for input (admissions) are not the leaders when it comes to output (talent development). For example, colleges perennially topping the list of most selective institutions were either further down in the rankings (Stanford at #18) or missing entirely (Harvard). Within the University of California system, UCLA and UC San Diego are placed above UC Berkeley. On the world stage, the vaunted Indian Institute of Technology (Kanpur) would barely squeak into the top 50, coming slightly below the University of Santa Clara. The competitiveness of entry for these institutions does not tightly correlate with the performance of their graduates.
What makes these findings all the more remarkable is that CodeSignal’s assessment is not a measure of ability or long-term potential but only of current knowledge. This is in contrast to traditional college admissions, which are supposedly focused on selecting students for ability, with the implicit value proposition of the more selective colleges being that they will do a better job of developing student talent than their less selective rivals. CodeSignal’s results call this into question. If the college admissions committees are doing their job properly, the students entering more selective colleges should have higher ability or potential. If one thinks of the subsequent talent development process as a race, these students could be thought of as starting with a 10-yard lead. If colleges were doing a comparable job of developing talent and realizing potential, one would expect the students starting with a lead to finish ahead of their peers rather than coming in behind. This suggests one of two conclusions: 1) The admissions process is doing a poor job of identifying talent, or 2) The colleges are doing a poor job of developing talent. Either conclusion raises serious concerns.
Cutting Edge Companies Are The Best Talent Source
Similar observations were made by Ilya Kirnos of SignalFire, a tech venture company named one of the most innovative companies in data science. The article opens with a provocative observation: "A Harvard diploma, a PhD, or a stint at Google are no longer the best signifiers of the top minds in artificial intelligence.” Indeed, among AI start-up hires in 2023, only 20% had a PhD or were coming from top schools. Instead, the hiring focus has increasingly shifted to those who have real-world experience at cutting-edge AI companies (what Kirnos amusingly dubs “the AI-vy league"), specifically “OpenAI, Anthropic, MosaicML (now part of DataBricks), Cohere, AI21 Labs, Hugging Face, Stability AI, Midjourney, and Inflection." These companies have become the go-to places for hiring AI talent. The article further notes that “while graduating from [Ivy League]
schools can certainly help people get jobs at top startups, engineers still have to pass rigorous technical interviews and prove themselves on the job to stay employed there.”When asked why he thought that this shift was happening, Kirnos identified two elements. The first was the amount of computational power (“compute”) available to engineers at the companies compared to what is available at the universities. The second was the differences in the amount of data available. Large companies have access to compute and data that universities do not. So, while theoretical research of the most abstract kind can occur naturally at universities, anything fundamentally data-driven will be outpaced by industry.
What Does This All Mean?
Even if college admissions are not truly meritocratic, hiring for the most competitive jobs is. That students graduating from traditional elite institutions are not achieving the best outcomes on objective measures of performance should prompt parents and students to question whether the investment of time and money in those institutions will be worthwhile. While the degree from such an institution may secure the interview, in fields where actual knowledge and skills are crucial, the degree will ultimately hold little value unless the skills are there to back it up. Institutions that develop talent will outpace those that just develop resumes. Furthermore, no matter how good a university is, if the goal is developing cutting-edge technical expertise, the university can only take you so far. You will be better off securing a position in an innovative company and getting to work rather than spending years in university preparation for an ever-changing landscape. After all, there will be ample time later to transition to a university position when you grow weary of working.
