Evaluating testing methods by delivered reliability

There are two main goals in testing software: (1) to achieve adequate quality (debug testing), where the objective is to probe the software for defects so that these can be removed, and (2) to assess existing quality (operational testing), where the objective is to gain confidence that the software is reliable. Debug methods tend to ignore random selection of test data from an operational profile, while for operational methods this selection is all-important. Debug methods are thought to be good at uncovering defects so that these can be repaired, but having done so they do not provide a technically defensible assessment of the reliability that results. On the other hand, operational methods provide accurate assessment, but may not be as useful for achieving reliability. This paper examines the relationship between the two testing goals, using a probabilistic analysis. We define simple models of programs and their testing, and try to answer the question of how to attain program reliability: is it better to test by probing for defects as in debug testing, or to assess reliability directly as in operational testing? Testing methods are compared in a model where program failures are detected and the software changed to eliminate them. The “better” method delivers higher reliability after all test failures have been eliminated. Special cases are exhibited in which each kind of testing is superior. An analysis of the distribution of the delivered reliability indicates that even simple models have unusual statistical properties, suggesting caution in interpreting theoretical comparisons

Probing the helium-graphite interaction

Two separate lines of investigation have recently converged to produce a highly detailed picture of the behavior of helium atoms physisorbed on graphite basal plane surfaces. Atomic beam scattering experiments on single crystals have yielded accurate values for the binding energies of several· states for both (^4)He and (^3)He, as well as matrix elements of the largest Fourier component of the periodic part of the interaction potential. From these data, a complete three-dimensional description of the potential has been constructed, and the energy band structure of a helium atom moving in this potential calculated. At the same time, accurate thermodynamic measurements were made on submonolayer helium films adsorbed on Grafoil. The binding energy and low-coverage specific heat deduced from these measurements are in excellent agreement with those calculated from the band structures

Quantum Weakly Nondeterministic Communication Complexity

We study the weakest model of quantum nondeterminism in which a classical proof has to be checked with probability one by a quantum protocol. We show the first separation between classical nondeterministic communication complexity and this model of quantum nondeterministic communication complexity for a total function. This separation is quadratic.Comment: 12 pages. v3: minor correction

Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach

Intelligent assistants are handling increasingly critical tasks, but until now, end users have had no way to systematically assess where their assistants make mistakes. For some intelligent assistants, this is a serious problem: if the assistant is doing work that is important, such as assisting with qualitative research or monitoring an elderly parent’s safety, the user may pay a high cost for unnoticed mistakes. This paper addresses the problem with WYSIWYT/ML (What You See Is What You Test for Machine Learning), a human/computer partnership that enables end users to systematically test intelligent assistants. Our empirical evaluation shows that WYSIWYT/ML helped end users find assistants’ mistakes significantly more effectively than ad hoc testing. Not only did it allow users to assess an assistant’s work on an average of 117 predictions in only 10 minutes, it also scaled to a much larger data set, assessing an assistant’s work on 623 out of 1,448 predictions using only the users’ original 10 minutes’ testing effort

Testing Linear-Invariant Non-Linear Properties

We consider the task of testing properties of Boolean functions that are invariant under linear transformations of the Boolean cube. Previous work in property testing, including the linearity test and the test for Reed-Muller codes, has mostly focused on such tasks for linear properties. The one exception is a test due to Green for "triangle freeness": a function f:\cube^{n}\to\cube satisfies this property if $f(x),f(y),f(x+y)$ do not all equal 1, for any pair x,y\in\cube^{n}. Here we extend this test to a more systematic study of testing for linear-invariant non-linear properties. We consider properties that are described by a single forbidden pattern (and its linear transformations), i.e., a property is given by $k$ points v_{1},...,v_{k}\in\cube^{k} and f:\cube^{n}\to\cube satisfies the property that if for all linear maps L:\cube^{k}\to\cube^{n} it is the case that $f(L(v_{1})),...,f(L(v_{k}))$ do not all equal 1. We show that this property is testable if the underlying matroid specified by $v_{1},...,v_{k}$ is a graphic matroid. This extends Green's result to an infinite class of new properties. Our techniques extend those of Green and in particular we establish a link between the notion of "1-complexity linear systems" of Green and Tao, and graphic matroids, to derive the results.Comment: This is the full version; conference version appeared in the proceedings of STACS 200

Chromatic number, clique subdivisions, and the conjectures of Haj\'os and Erd\H{o}s-Fajtlowicz

For a graph $G$, let $\chi(G)$ denote its chromatic number and $\sigma(G)$ denote the order of the largest clique subdivision in $G$. Let H(n) be the maximum of $\chi(G)/\sigma(G)$ over all $n$-vertex graphs $G$. A famous conjecture of Haj\'os from 1961 states that $\sigma(G) \geq \chi(G)$ for every graph $G$. That is, $H(n) \leq 1$ for all positive integers $n$. This conjecture was disproved by Catlin in 1979. Erd\H{o}s and Fajtlowicz further showed by considering a random graph that $H(n) \geq cn^{1/2}/\log n$ for some absolute constant $c>0$. In 1981 they conjectured that this bound is tight up to a constant factor in that there is some absolute constant $C$ such that $\chi(G)/\sigma(G) \leq Cn^{1/2}/\log n$ for all $n$-vertex graphs $G$. In this paper we prove the Erd\H{o}s-Fajtlowicz conjecture. The main ingredient in our proof, which might be of independent interest, is an estimate on the order of the largest clique subdivision which one can find in every graph on $n$ vertices with independence number $\alpha$.Comment: 14 page

The development of the Meaning in Life Index (MILI) and its relationship with personality and religious behaviours and beliefs among UK undergraduate students

The scales available for assessing meaning in life appear to be confounded with several related constructs, including purpose in life, satisfaction with life, and goal-directed behaviour. The Meaning in Life Index (MILI), a new instrument devised as a specific measure of meaning in life, was developed from responses to a pool of 22 items rated by a sample of 501 undergraduate students in Wales. The nine-item scale demonstrated sufficient face validity, internal consistency, and scale reliability to commend the instrument for future use. With respect to personality, the MILI scores were most strongly predicted by neuroticism (negatively), and less strongly by extraversion (positively) and psychoticism (negatively). With respect to several religious behavioural variables, those who attended church at least weekly returned significantly higher MILI scores than those who attended church less frequently. Intrinsic religiosity was the only orientation to be significantly associated with the MILI scale scores, although the magnitude of the association was smaller than anticipated. These results suggest that meaning in life is associated more strongly with individual differences in personality than with specific religious behaviours and attitudes. The implications of these results are discussed in terms of individual's personal values and attitudes that might underlie their experience of a meaning in life