Disruptive Innovations and Disruptive Assurance: Assuring Machine Learning and Autonomy

Autonomous and machine learning-based systems are disruptive innovations and thus require a corresponding disruptive assurance strategy. We offer an overview of a framework based on claims, arguments, and evidence aimed at addressing these systems and use it to identify specific gaps, challenges, and potential solutions

Development of a testing protocol for oil solidifier effectiveness evaluation

Chemical countermeasures for oil spill remediation have to be evaluated and approved by the U.S. Environmental Protection Agency before they may be used to remove or control oil discharges. Solidifiers are chemical agents that change oil from a liquid to a solid by immobilizing the oil and bonding the liquid into a solid carpet-like mass with minimal volume increase. Currently, they are listed as Miscellaneous Oil Spill Control Agent in the National Contingency Plan and there is no protocol for evaluating their effectiveness. An investigation was conducted to test the oil removal efficiency of solidifiers using three newly developed testing protocols. The protocols were qualitatively and quantitatively evaluated to determine if they can satisfactorily differentiate effective and mediocre products while still accounting for experimental error. The repeatability of the three protocols was 15.9, 5.1, and 2.7 %. The protocol with the best performance involved measuring the amount of free oil remaining in the water after the solidified product was removed using an ultraviolet–visible spectrophotometer and it was adopted to study the effect of solidifier-to-oil mass ratio, mixing energy, salinity, and beaker size (i.e., area affected by the spill) on solidifier efficiency. Analysis of Variances were performed on the data collected and results indicated that the beaker size increased spreading, which reduced removal efficiency. Mixing speed appears to impart a ceiling effect with no additional benefit provided by the highest level over the middle level. Salinity was found to be mostly an insignificant factor on performance

WE ARE STILL PLAYING: A META-ANALYSIS OF GAME-BASED LEARNING IN MATHEMATICS EDUCATION

The purpose of this meta-analysis was to investigate the effectiveness of the use of games as part of mathematics instruction on academic achievement in grades Kindergarten to 12 in the United States. There were 17 studies selected for investigation published from 2010 to 2023 that focused on game-based learning and mathematics. This meta-analysis fills the gap in the knowledge by examining classes that are using game based learning across three platforms of instruction: nondigital games, digital on computers, and mobile devices. The findings from this meta-analysis suggest that the usage of game-based learning in a classroom has a positive effect on students’ mathematics achievement in addition to the suggestion that the findings can apply to more academic domains beyond mathematics. The moderator variables that were examined in this inquiry were category of games (if a game used was classified as a serious game, an educational game, or a simulation), game platform (if the digital game was played on a computer or a mobile device), the students’ grade level, gender, and frequency of the game-based learning activities. The overall effect size of using game-based learning in the classroom was 0.30 and there were statistically significant findings regarding gender comparisons and grade level comparisons. Out of the 14 moderator analyses conducted, 12 were found to be of statistical significance. One of the difficulties identified was classifying studies that used mobile devices given that most studies used the term “apps,” which is not sufficient for classification as iii one does not know if apps referred to the type of digital device or an app on an iPad, phone, computer, or other technological device. The designing of lessons plan with game-based learning activities requires several factors. Beyond just the technology selected, the type of game and if the game required any modifications that would need additional investigation. A teacher’s familiarity with the game used in the classroom also would be a benefit. Additionally, educators seeking to use game-based learning should include considerations for how the instructional time is used for game-based learning. The findings of this study provide suggestions for the length of time students should spend on a game and how often games should be used for learning. The findings from this meta-analysis provide the implication that game-based learning could be used beyond just mathematic education, as statistical significance was found regarding problem-solving activities. In addition, future research should consider the definitions of in the field of game-based learning and the changing technology platforms. This dissertation, written under the direction of the candidate’s dissertation committee and approved by the members of the committee, has been presented to and accepted by the Faculty of the School of Education in partial fulfillment of the requirements for the degree of Doctor of Education. The content and research methodologies presented in this work represent the work of the candidate alone

SOM neural network design – a new Simulink library based approach targeting FPGA implementation

The paper presents a method for FPGA implementation of Self-Organizing Map (SOM) artificial neural networks with on-chip learning algorithm. The method aims to build up a specific neural network using generic blocks designed in the MathWorks Simulink environment. The main characteristics of this original solution are: on-chip learning algorithm implementation, high reconfiguration capability and operation under real time constraints. An extended analysis has been carried out on the hardware resources used to implement the whole SOM network, as well as each individual component block

Attribution Patching Outperforms Automated Circuit Discovery

Automated interpretability research has recently attracted attention as a potential research direction that could scale explanations of neural network behavior to large models. Existing automated circuit discovery work applies activation patching to identify subnetworks responsible for solving specific tasks (circuits). In this work, we show that a simple method based on attribution patching outperforms all existing methods while requiring just two forward passes and a backward pass. We apply a linear approximation to activation patching to estimate the importance of each edge in the computational subgraph. Using this approximation, we prune the least important edges of the network. We survey the performance and limitations of this method, finding that averaged over all tasks our method has greater AUC from circuit recovery than other methods.Comment: 6 main paper pages, 6 additional pages. NeurIPS 2023 ATTRIB Worksho

Interpreting Attention Layer Outputs with Sparse Autoencoders

Decomposing model activations into interpretable components is a key open problem in mechanistic interpretability. Sparse autoencoders (SAEs) are a popular method for decomposing the internal activations of trained transformers into sparse, interpretable features, and have been applied to MLP layers and the residual stream. In this work we train SAEs on attention layer outputs and show that also here SAEs find a sparse, interpretable decomposition. We demonstrate this on transformers from several model families and up to 2B parameters. We perform a qualitative study of the features computed by attention layers, and find multiple families: long-range context, short-range context and induction features. We qualitatively study the role of every head in GPT-2 Small, and estimate that at least 90% of the heads are polysemantic, i.e. have multiple unrelated roles. Further, we show that Sparse Autoencoders are a useful tool that enable researchers to explain model behavior in greater detail than prior work. For example, we explore the mystery of why models have so many seemingly redundant induction heads, use SAEs to motivate the hypothesis that some are long-prefix whereas others are short-prefix, and confirm this with more rigorous analysis. We use our SAEs to analyze the computation performed by the Indirect Object Identification circuit (Wang et al.), validating that the SAEs find causally meaningful intermediate variables, and deepening our understanding of the semantics of the circuit. We open-source the trained SAEs and a tool for exploring arbitrary prompts through the lens of Attention Output SAEs