Large expert-curated database for benchmarking document similarity detection in biomedical literature search

Document recommendation systems for locating relevant literature have mostly relied on methods developed a decade ago. This is largely due to the lack of a large offline gold-standard benchmark of relevant documents that cover a variety of research fields such that newly developed literature search techniques can be compared, improved and translated into practice. To overcome this bottleneck, we have established the RElevant LIterature SearcH consortium consisting of more than 1500 scientists from 84 countries, who have collectively annotated the relevance of over 180 000 PubMed-listed articles with regard to their respective seed (input) article/s. The majority of annotations were contributed by highly experienced, original authors of the seed articles. The collected data cover 76% of all unique PubMed Medical Subject Headings descriptors. No systematic biases were observed across different experience levels, research fields or time spent on annotations. More importantly, annotations of the same document pairs contributed by different scientists were highly concordant. We further show that the three representative baseline methods used to generate recommended articles for evaluation (Okapi Best Matching 25, Term Frequency-Inverse Document Frequency and PubMed Related Articles) had similar overall performances. Additionally, we found that these methods each tend to produce distinct collections of recommended articles, suggesting that a hybrid method may be required to completely capture all relevant articles. The established database server located at https://relishdb.ict.griffith.edu.au is freely available for the downloading of annotation data and the blind testing of new methods. We expect that this benchmark will be useful for stimulating the development of new powerful techniques for title and title/abstract-based search engines for relevant articles in biomedical research.Peer reviewe

Citrus Red Mite Control, 1982

Abstract Several numbered chemicals were evaluated at varying dosages for citrus red mite control of lemons in the Yuma Valley during Mar. and Apr. Single-tree plots were replicated four times for each of the treatments. The infestation on the new leaves (bright green) was moderate during Mar. and Apr. Approximately 200 gal/acre was applied on 23 Mar. with a John Bean Sprayer at 350 lb/in2. Each tree was throroughly sprayed (TO with ca. 3 gal per tree using a hand-gun. Field checks were made with the aid of a 20X magnifying lens recording all motile forms.</jats:p

Citrus Thrips Control on Lemons, 1982

Abstract Various experimental materials were evaluated in these trials including synthetic pyrethroids at different rates. All treatments were applied 27 Apr. to one-tree plots replicated four times. Approximately 3 gal per tree was applied with a hand-gun connected to a John Bean Sprayer (3 x 60 = 180 gal/acre) and 350 lb/in2. This amount of spray was a thorough coverage (TO. The thrips population was moderate but uniform throughout the test area. Thrips counts were made at 2, 6, 10, 16, and 21 days following treatments with the “black box” technique.</jats:p

Aphid and Weevil Control on Alfalfa, 1982

Abstract Several materials were tested in this experiment for the control of aphids and the Egyptian alfalfa weevil on alfalfa at the Yuma Mesa Exp. Stn. during March. The spray treatments were applied with a small-plot sprayer at 15 gal/acre and 40 lb/in2 (CO2 controlled). The boom was equipped with six flat nozzles (8001). Plots consisted of four replicates 8 ft by 80 ft (total 0.05 acre per treatment). The aphid population averaged ca. 60% blue alfalfa and 40% pea aphid. During the test period the overall population was more or less declining. Likewise, the weevil population was also declining, especially by 22 Mar., but the damage continued to be noticeable.</jats:p

Control of Carmine Spider Mite on Cotton, 1983

Abstract Varous newer chemical compounds were evaluatd for the control of the carmine spider mite on cotton at the Yuma Experiment Station in 1983. Plots consisted of 25 ft of row replicated four times. Treatments made on 23 Aug. were applied with a 2-liter CO2 pressurized sprayer unit using 20 gal/acre at 40 lb/in2 . Evaluations were made from 80 leaf samples per treatment (20 per plot) taken at random and examined in the laboratory with a SteroZoom microscope 3, 6, and 14 days following treatment. In all cases treatments appeared to be more efficient after 3 and 6 days with an increase at 14 days. this reflects limited residual action, partial ovicidal effectiveness, and the resurgence fo mites not killed. The most effective treatment was MK-936 at the higher rate (0.02 lb (AI)/acre). FMC54800 and UC81341 treatments provided satisfactory control. The standard. Comite, and FMC Spray Oil were not as effective as other treatments after 14 days but both of these were at a low rate. At this time of year Comite should be used at 2 pt/acre and the Spray Oil at 1% or higher. Coverage is another problem in controlling mites in that they inhabit the undersides of leaves. No phytotoxicity was observed.</jats:p

Aphid and Egyptian Alfalfa Weevil Larva Control, 1983

Abstract Thirteen treatments were evaluated for the control of aphids and Egyptian alfalfa weevil. Approximately 90% of the aphid population during the experiment were blue alfalfa aphid, the others pea aphid. The average height of stems on 19 Feb. was 10 in. This alfalfa was on sandy silt soil at the Yuma Mesa Citrus Stn. Plots of 8 ft by 50 ft were replicated four times. The applications on 19 Feb. were made with a small-plot sprayer at a rate of 20 gal/acre and 40 lb/in2 (pressurized with CO2). The number of aphids per stem on 19 Feb. averaged 10.53. Counts 2, 4, 7, 10, and 14 days following treatments are given. The applications coincided with the peak of abundance for the Egyptian alfalfa weevil larvae present.</jats:p

Control of Citrus Red Mite on Lemons, 1982

Abstract Two newer materials were compared with Vendex and Kelthane for control of the citrus red mite on lemons. Each treatment consisted of one-tree plots replicated four times. Applications were made 22 Mar. with a John Bean Sprayer using a hand-gun at 300 lb/in2, ca. 3 gal per tree (180 gal/acre). Mite counts were made in the grove with the aid of a 20X magnifying lens 3, 10, 14, and 21 days after spraying.</jats:p

Evaluation Op Chemicals for Control Op Citrus Thrips on Lemons, 1983

Abstract Six materials were tested for control of citrus thrips on Lisbon lemons at the Yuma Mesa Citrus Station during Apr. 1983. Each treatment consisted of 0.5-acre plots and was replicated four times. Sprays were applied with a Myers Speed Sprayer at a rate of 125 gal/acre and 400 lb/in2. Treatments were made at night on 7 Apr. Results represent the average number of thrips per terminal at 4, 7, 14, and 21 days following spraying. The black box technique was utilized for thrips counts. This method involves rapping the terminals on a 1/4 in. screen fitted over a 6 by 8-in. wood box, approximately 2 in. deep.</jats:p