Algorithmic progress in computer vision

We investigate algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision. We estimate a model, informed by work on neural scaling laws, and infer a decomposition of progress into the scaling of compute, data, and algorithms. Using Shapley values to attribute performance improvements, we find that algorithmic improvements have been roughly as important as the scaling of compute for progress computer vision. Our estimates indicate that algorithmic innovations mostly take the form of compute-augmenting algorithmic advances (which enable researchers to get better performance from less compute), not data-augmenting algorithmic advances. We find that compute-augmenting algorithmic advances are made at a pace more than twice as fast as the rate usually associated with Moore's law. In particular, we estimate that compute-augmenting innovations halve compute requirements every nine months (95\% confidence interval: 4 to 25 months)

Who is leading in AI? An analysis of industry AI research

AI research is increasingly industry-driven, making it crucial to understand company contributions to this field. We compare leading AI companies by research publications, citations, size of training runs, and contributions to algorithmic innovations. Our analysis reveals the substantial role played by Google, OpenAI and Meta. We find that these three companies have been responsible for some of the largest training runs, developed a large fraction of the algorithmic innovations that underpin large language models, and led in various metrics of citation impact. In contrast, leading Chinese companies such as Tencent and Baidu had a lower impact on many of these metrics compared to US counterparts. We observe many industry labs are pursuing large training runs, and that training runs from relative newcomers -- such as OpenAI and Anthropic -- have matched or surpassed those of long-standing incumbents such as Google. The data reveals a diverse ecosystem of companies steering AI progress, though US labs such as Google, OpenAI and Meta lead across critical metrics

Chinchilla Scaling: A replication attempt

Hoffmann et al. (2022) propose three methods for estimating a compute-optimal scaling law. We attempt to replicate their third estimation procedure, which involves fitting a parametric loss function to a reconstruction of data from their plots. We find that the reported estimates are inconsistent with their first two estimation methods, fail at fitting the extracted data, and report implausibly narrow confidence intervals--intervals this narrow would require over 600,000 experiments, while they likely only ran fewer than 500. In contrast, our rederivation of the scaling law using the third approach yields results that are compatible with the findings from the first two estimation procedures described by Hoffmann et al

Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning

We analyze the growth of dataset sizes used in machine learning for natural language processing and computer vision, and extrapolate these using two methods; using the historical growth rate and estimating the compute-optimal dataset size for future predicted compute budgets. We investigate the growth in data usage by estimating the total stock of unlabeled data available on the internet over the coming decades. Our analysis indicates that the stock of high-quality language data will be exhausted soon; likely before 2026. By contrast, the stock of low-quality language data and image data will be exhausted only much later; between 2030 and 2050 (for low-quality language) and between 2030 and 2060 (for images). Our work suggests that the current trend of ever-growing ML models that rely on enormous datasets might slow down if data efficiency is not drastically improved or new sources of data become available

The Compute Divide in Machine Learning: A Threat to Academic Contribution and Scrutiny?

There are pronounced differences in the extent to which industrial and academic AI labs use computing resources. We provide a data-driven survey of the role of the compute divide in shaping machine learning research. We show that a compute divide has coincided with a reduced representation of academic-only research teams in compute intensive research topics, especially foundation models. We argue that, academia will likely play a smaller role in advancing the associated techniques, providing critical evaluation and scrutiny, and in the diffusion of such models. Concurrent with this change in research focus, there is a noticeable shift in academic research towards embracing open source, pre-trained models developed within the industry. To address the challenges arising from this trend, especially reduced scrutiny of influential models, we recommend approaches aimed at thoughtfully expanding academic insights. Nationally-sponsored computing infrastructure coupled with open science initiatives could judiciously boost academic compute access, prioritizing research on interpretability, safety and security. Structured access programs and third-party auditing may also allow measured external evaluation of industry systems

Algorithmic progress in language models

We investigate the rate at which algorithms for pre-training language models have improved since the advent of deep learning. Using a dataset of over 200 language model evaluations on Wikitext and Penn Treebank spanning 2012-2023, we find that the compute required to reach a set performance threshold has halved approximately every 8 months, with a 95% confidence interval of around 5 to 14 months, substantially faster than hardware gains per Moore's Law. We estimate augmented scaling laws, which enable us to quantify algorithmic progress and determine the relative contributions of scaling models versus innovations in training algorithms. Despite the rapid pace of algorithmic progress and the development of new architectures such as the transformer, our analysis reveals that the increase in compute made an even larger contribution to overall performance improvements over this time period. Though limited by noisy benchmark data, our analysis quantifies the rapid progress in language modeling, shedding light on the relative contributions from compute and algorithms

Endoscopic Cystolithotripsy for a Giant Stone in The Orthotopic Neobladder: A Case Report

Radical cystectomy and urinary diversion is an excellent treatment option for invasive bladder cancer. Ileal conduit and orthotopic neobladder have been applied as the most frequent urinary diversion methods for many years. Stone formation is a rare complication in the ortotopic neobladder. In the case presented, a 67-year-old-man who had undergone radical cystectomy and orthotopic neobladder reconstruction ten years ago with no complaints in the following five years presented with fever, dysuria, and urinary frequency. We detected a 10.8 cm stone in the neobladder, and the giant stone was fragmented by endoscopic cystolithotripsy via transurethral approach. Complete stone clearance was achieved

Fronto-striatal structures related with model-based control as an endophenotype for obsessive–compulsive disorder

Recent theories suggest a shift from model-based goal-directed to model-free habitual decision-making in obsessive-compulsive disorder (OCD). However, it is yet unclear, whether this shift in the decision process is heritable. We investigated 32 patients with OCD, 27 unaffected siblings (SIBs) and 31 healthy controls (HCs) using the two-step task. We computed behavioral and reaction time analyses and fitted a computational model to assess the balance between model-based and model-free control. 80 subjects also underwent structural imaging. We observed a significant ordered effect for the shift towards model-free control in the direction OCD>SIB>HC in our computational parameter of interest. However less directed analyses revealed no shift towards model-free control in OCDs. Nonetheless, we found evidence for reduced model-based control in OCDs compared to HCs and SIBs via 2nd stage reaction time analyses. In this measure SIBs also showed higher levels of model-based control than HCs. Across all subjects these effects were associated with the surface area of the left medial/right dorsolateral prefrontal cortex. Moreover, correlations between bilateral putamen/right caudate volumes and these effects varied as a function of group: they were negative in SIBs and OCDs, but positive in HCs. Associations between fronto-striatal regions and model-based reaction time effects point to a potential endophenotype for OCD

Aggregating human judgment probabilistic predictions of COVID-19 transmission, burden, and preventative measures

Aggregated human judgment forecasts for COVID-19 targets of public health importance are accurate, often outperforming computational models. Our work shows aggregated human judgment forecasts for infectious agents are timely, accurate, and adaptable, and can be used as tool to aid public health decision making during outbreaks

Cutaneous Metastasis from Squamous Cell Carcinoma of The Bladder: A Case

Objective: To report a case with cutaneous metastasis of bladder squamous cell carcinoma. Very few cases of skin metastases from the urinary bladder are reported in the literature.Case report: Cutaneous metastasis of bladder squamous cell carcinoma is an extremely rare clinic entity associated with poor prognosis. Cutaneous metastasis is accepted as a late manifestation of systemic spread. A 58-year-old man had undergone radical cystoprostatectomy with pelvic lymph node dissection in November 2013. The pathology report showed a moderately differentiated squamous cell bladder carcinoma with a staging of T3N0M0. Then, a large regional cutaneous lesion measuring 4x3.5 cm with an ulcerated and necrotic appearance located in the suprapubic area was detected in February 2014, and the lesion was removed with a 2 cm safety margin. The pathologic examination revealed squamous cell carcinoma. This report describes an interesting and rare case of cutaneous metastasis of bladder squamous cell carcinoma as the primary presentation of metastatic disease with a generally dismal prognosis.Conclusion: The skin metastasis of bladder squamous cell carcinoma is a rare clinical entity with a poor prognosis. Clinicians should be aware of this rare entity