This study demonstrates thatultrashort- and short-chain PFAAs are now abundant in the indoor environmentand in humans and warrants further research on potential adverse healtheffects of these exposures.Per- and polyfluoroalkyl substances (PFAS) make up alarge groupof fluorinated organic compounds extensively used in consumer productsand industrial applications. Perfluorooctanesulfonic acid (PFOS) andperfluorooctanoic acid (PFOA), the two perfluoroalkyl acids (PFAAs)with 8 carbons in their structure, have been phased out on a globalscale because of their high environmental persistence and toxicity.As a result, shorter-chain PFAAs with less than 8 carbons in theirstructure are being used as their replacements and are now widelydetected in the environment, raising concerns about their effectson human health. In this study, 47 PFAAs and their precursors weremeasured in paired samples of dust and drinking water collected fromresidential homes in Indiana, United States, and in blood and urinesamples collected from the residents of these homes. Ultrashort- (with2 or 3 carbons [C2–C3]) and short-chain (with 4–7 carbons[C4–C7]) PFAAs were the most abundant in all four matricesand constituted on average 69–100% of the total PFAA concentrations.Specifically, trifluoroacetic acid  and perfluoropropanoicacid  were the predominant PFAAs in most of the samples.Significant positive correlations ( This studydemonstrates that ultrashort- and short-chainPFAAs are now prevalent in the indoor environment and in humans. However, their manufacturing and use have been regulatedover the last two decades due to their environmental persistence,bioaccumulation, and toxicity to wildlife and humans.2 PFOS was added to the Annex B  of theStockholm Convention on Persistent Organic Pollutants in 2009 andPFOA was listed under Annex A  of the conventionin 2019.3 As a result of these restrictions,other PFAS have become more widely used, including shorter-chain PFAAs(with less than 8 carbons in their structure) that were introducedas less persistent and bioaccumulative alternatives because of theirsmaller molecular size.4 However, recentstudies have shown that short-chain PFAAs are now ubiquitous in indoorand outdoor environments. Several of these shorter-chain compoundshave been detected in indoor dust,5 air,6 drinking water,8 aquatic systems,9 soil, and sediment.11 It also has been shown that short-chain PFAAs are mobile12 and can be transported over long distances throughseawater and reach remote areas.14 Moreover,the two ultrashort-chain PFAAs, trifluoroacetic acid  andperfluoropropanoic acid , have also been found in the environment,including snow, surface, and groundwater,13 precipitation,16 sediment, soils, and sludge.17 TFA andPFPrA are industrial chemicals, used as laboratory reagents and catalysts,as well as byproducts in chemical synthesis.19 In addition, these two compounds can be formed from various indirectsources, including atmospheric oxidation of PFAA precursors or chlorofluorocarbons,21 novel refrigerants,22 the photodegradation or thermolysis of fluoropolymers in the environment,23 and biological degradation of plant-protectingagents or pharmaceuticals,20 all posing as potential sources of the ultrashort-chain PFAAs inthe environment. Furthermore, multiple studies suggest that PFAA precursors,such as side-chain fluorotelomer-based polymers, fluorotelomer alcohols(FTOHs), and polyfluoroalkyl phosphate esters (PAPs), as well as long-chainPFAAs (with more than 8 carbons in their structure), can degrade toshorter-chain PFAAs in the environment.24 Consequently, due to their high persistence and mobility,26 perfluorobutanesulfonic acid  and its salts were identifiedas substances of very high concern under the European Union Registration,Evaluation, Authorization, and Restriction of Chemicals (REACH) Programin 2019,27 and concerns were also raisedcalling for the urgent and effective regulation of other short-chainPFAAs.28Per- and polyfluoroalkyl substances (PFAS)are a large group offluorinated organic compounds that are extensively used in variousindustrial and consumer applications such as water-, grease-, andstain-repellents, surfactants, and lubricants.31 breast milk,33 and urine.34 A study from Tianjin, China,reported that TFA was commonly detected at levels comparable to thoseof several long-chain PFAAs in human blood.18 The levels of PFBS in Swedish women’s blood increased ata rate of around 11% per year between 1996 and 2010.31 The detection frequencies of several short-chain PFAAs(C4–C7) have been increasing in breast milk with a doublingtime of ∼4 years on a global scale from 1996 to 2019.32 Perfluorobutanoic acid , perfluorohexanoicacid , and perfluoroheptanoic acid  were frequentlydetected in urine samples taken from the general population in theUnited States in 2013–2014.34In recent years, the ultrashort-and short-chain PFAAs have alsobeen detected in human blood,35In vitro and in vivo studies have demonstrated that exposure to short-chainPFAAs, such as PFBA, PFBS, PFHxA, and PFHpA, can have adverse effectson the reproductive, developmental, hepatic, and renal systems aswell as lipid metabolism.43 Recent epidemiological research demonstrates that PFBS and PFHpAcan disrupt gonadotropins as well as free androgen levels in fetuses.44 Moreover, perfluoropentanoic acid can alter the thyroid function in newborns, further indicating thedevelopmental toxicity of these short-chain PFAAs in humans.45There is growing evidence of the toxicity of some ultrashort- andshort-chain PFAAs. The acute toxicity of TFA and PFPrA on freshwaterinvertebrates was found to be higher than that of the longer-chainPFAAs.46 In addition, PFBA,PFBS, PFPeA, PFHxA, and PFHpA were frequently detected in source andtreated drinking water collected from 25 water treatment plants inthe U.S. (detection frequencies [DF]: 88–100%).8 A recent study from China has also found ultrashort- andshort-chain PFAAs in indoor and outdoor dust, with TFA as the mostabundant.5 Moreover, our previous studyhas demonstrated that breastfeeding is an important exposure pathwayto short-chain PFAAs (C4–C7) in nursing infants.32 In addition, biotransformation of PFAA precursorscan contribute to human exposure to short-chain PFAAs.50 For example, biotransformation of perfluorophosphate monoesters(monoPAPs), PAP diesters (diPAPs), and polyfluoroalkyl carboxamideshas resulted in the formation of shorter-chain PFAAs in invivo studies.51 Multiple studies have indicatedthat the precursors of short-chain PFAAs are ubiquitous in indoordust,54 air,56 and consumer products ;59 however, the significance of their biotransformationto shorter-chain PFAAs as an indirect exposure pathway in humans remainslargely unknown.There are severalpathways through which humans can be exposedto ultrashort- and short-chain PFAAs. PFAAs with a shorter carbonchain were more frequently detected in U.S. bottled water comparedto the long-chain PFAAs, with PFPrA contributing 42% to the detectedPFAA concentrations.n = 324 of matchedserum, urine, dust, and drinking water samples collected from 81 participants)for a suite of 47 PFAAs and their precursors. These included 2 ultrashort-(2–3 carbons) and 4 short-chain (4–7 carbons) perfluorocarboxylicacids (PFCAs), 1 ultrashort- (3 carbons) and 2 short-chain (4 or 5carbons) perfluoroalkanesulfonic acids (PFSAs), and 14 long-chainPFAAs [PFCAs with >7 carbons and PFSAs with >5 carbons], aswell as24 PFAA precursors . Our goal wasto examine the current PFAS exposure patterns in people and theirresidences, investigate the associations between the chemical levelsin four sampled matrices, and estimate the relative contributionsof dust and drinking water uptake to the overall body burden.In this study, we analyzed paired samples ofresidential dust anddrinking water collected from households in Indiana, United States,as well as paired blood serum and urine samples collected from theresidents of these homes  were recruited from the Person to Person(P2P) Health Interview Study cohort . The study was approved by the Indiana University InstitutionalReview Board and all participants signed an informed consent formbefore participation. Dust, drinking water, blood, and urine wereall paired and collected on the same day . All samples were keptin a cooler with ice packs before being delivered to the laboratoryat the end of each sampling day. Blood serum separation was conductedon the day the samples were delivered to the laboratory. The sampleswere stored at −80 °C until analysis. Demographic, behavioral,and housing information was collected from each participant usingquestionnaires administered at the time of sample collection (see Table S1 for detailed survey questions).All samples, including dust, drinkingwater, blood, and urine, were collected between the months of Augustand December 2020 in the state of Indiana, United States. Participants(Supporting Information (Tables S2–S6).All samples were analyzed for 47 PFAS,including 23 PFAAs (14 PFCAs and 9 PFSAs) and 24 PFAA precursors  using liquidchromatography- and gas chromatography–mass spectrometry. Thecomplete list of analytes and the details of the analytical methodsand quality control measures are provided in the The distribution of demographic and housingcharacteristics in the study population was examined using means (withtheir standard deviations), frequencies, and counts. Separately foreach matrix, detection frequencies for all PFAAs and PFAA precursorswere assessed, and selected percentiles were used to examine distributions of analyte concentrations. Thetotal concentrations were defined as the sum of all of the analyteconcentrations measured for that specific analyte group. The contributionof each analyte to the total concentration was determined based onthe ratio of the median concentration of that analyte to the mediantotal concentration.For the statistical analyses, concentrationsbelow method detection limits (MDLs) were imputed with MDL/2. Thereported concentrations were blank corrected by subtracting the averageblank levels from the sample levels. All PFAA and PFAA precursor concentrationswere logarithmically transformed  for downstream analyses.Correlations between the concentrations of PFAAs and PFAA precursorsdetected in more than 50% of the samples across matrices were examinedby using Spearman correlation coefficients.60 was appliedto estimate the resulting serum concentrations (Cdust to serum and Cwater to serum) and relative source contributions(RSCs) of dust  and drinkingwater intake (RSCdust to serum and RSCwater to serum) for PFAAs with >50% detection. The details of this analysisarepresented in the Supporting Information. Lastly, we assessed whether concentrations of total PFAA and PFAAprecursor varied across housing characteristics using a Mann–Whitneytest for the comparison of the logarithmically transformed concentrations.A one-compartmenttoxicokinetic (TK) modelAll statistical analyses were conducted using IBM SPSS Statistics24 and Sigma Plot 13.n = 81)is given in 2), while 67% were overweight (25–29.9 kg/m2) or obese (≥30 kg/m2). Participants spent,on average, about 17 ± 5.0 h per day in their homes. None ofthe participants worked in an environment posing occupational PFASexposure .A summary of the participants’demographic and housing characteristics , and 77% reported having vinylwallpaper in their homes. Seventy percent of homes were less than20 years old. More than half of the homes were vacuumed less frequently(some days), while 44% were vacuumed most days or every day , contributing 75% to the dust total PFAAconcentration .5These findings were similar to those from a recentstudy from Chinathat reported TFA and PFPrA as the predominant PFAAs in indoor dust(sampling year 2017)61 butwere up to 7 times lower than those found in dust collected in earlieryears, such as from Massachusetts in 2009 62 and Wisconsin in 2008 .63The two C8 PFAAs, PFOA and PFOS, were alsofrequently detectedin dust , but their concentrations  were lower than those of the ultrashort-chainPFAAs. These concentrations were similar to those measured in dustcollected from North Carolina in 2014–2016 ,Among other long-chain PFAAs, perfluorohexanesulfonicacid (PFHxS)was found at a median concentration of 2.7 ng/g (DF 73%), which waslower than that for PFOS and PFOA. The rest of the long-chain PFAAs,with a few exceptions, were frequently detected (up to 70%) but contributedonly a minor portion to the total PFAA concentrations (<1%).61 In addition, 8:2 FTOH wasdetected in 64% of the samples at a median concentration of 52 ng/g.Other precursors were detected less frequently and at lower concentrations,except for 2-(N-ethylperfluorooctanesulfonamido)-ethanol(EtFOSE) that was found at concentrations reaching up to 100,000 ng/g.Exceptionally high concentrations of EtFOSE reaching up to 75,500ng/g reported previously64 have been associated with the extensive use of carpets or fabrictreatments in homes.65 The median total precursor concentrationin dust (the sum of the 16 detected precursors) was 850 ng/g, twiceas high as the median dust total PFAA concentration (360 ng/g), eventhough the total precursor concentrations were likely underestimatedbecause only several precursors were measured.54PFAA precursors were primarily detected in dust, with mono- anddiPAPs found in 51–100% of the samples 2. The coTFA was the predominant PFAA in drinkingwater (median of 79 ng/L) with a detection frequency of 95% and an84% contribution to the total PFAA concentration in drinking water.PFPrA was the second most abundant PFAA in drinking water (median6.9 ng/L) detected in 95% of the samples, contributing 7.4% to thetotal PFAA concentration. The short-chain PFAAs were frequently detectedas well but at lower concentrations than TFA and PFPrA and contributedup to 2.6% to the total PFAA concentration. Generally, the detectionfrequency for the ultrashort- and short-chain PFAAs was higher indrinking water compared to dust. For example, PFPeA was detected in70% of drinking water samples but only in 10% of dust samples. Similarly,perfluoropropanesulfonic acid , PFBS (C4), and perfluoropentanesulfonicacid  were frequently found in drinking water  but less so in dust .66 but lower than those reported from China in 2012 (median155 ng/L).67 The levels of the short-chainPFAAs detected in this study (0.035–2.5 ng/L) were comparableto those collected from the 24 states across the United States (0.79–3.6ng/L).8 Inefficient water filtration couldbe the cause of the frequent detection of short-chain PFAAs in tapwater because of the challenges in the removal of these compoundsby wastewater treatment processes12 or by drinkingwater filters.68The levels of TFA in drinking water found in our study were consistentwith those detected in drinking water from the United States collectedin 1994–1995 (range 41–150 ng/L),69 Abiotic transformation studies of 6:2 diPAP or of long-chainPFAAs  have also reported thattheir oxidation can lead to the formation of shorter-chain PFAAs.71 In addition, fluorinated gases such as hydrochlorofluorocarbonsand hydrofluorocarbons, fluorinated pesticides and pharmaceuticals,some plastics , andaqueous film-forming foams have been identified as precursors of TFAin the environment72 and may at least partiallyexplain the abundance of TFA in dust and drinking water found in thecurrent study.A total oxidizableprecursor (TOP) assay showed that unknown precursorsof TFA and PFPrA constituted up to 94% of the total measured PFAAconcentrations in biochemically treated leachate.60 Thedecreasing levels of PFOA and PFOS in dust and drinking water couldbe related to their phase-out in the past two decades.2 Among other long-chain PFAAs, PFHxS and PFNA were foundat median concentrations of 0.17 ng/L (DF 88%) and 0.11 ng/L (DF 65%),respectively, which were lower than those of the C8 PFAS. The restof the long-chain PFAAs, were generally detected less frequently thanin dust (2.5–49%) and contributed only a minor portion to thetotal PFAA concentrations (<1%). Perfluoro-4-ethylcyclohexanesulfonicacid (PFECHS), a cyclic C8 PFAA, was detected in 44% of the drinkingwater samples. Previous studies indicate the PFECHS contaminationin drinking water could be related to airports located nearby.73 Here, PFECHS was detected more frequently intap water collected from homes located in a county with a local airportcompared to locations without nearby airports .PFOA and PFOS were also frequently detected indrinking water , but their concentrations were lower than thoseof the shorter-chain PFAAs. The levels of PFOA and PFOS in drinkingwater measured here were 2–10 times lower than those in the nationwide studiesfrom the United States in 1989  andin 2007 .Among precursors, only a few were detectedin 11–31% ofdrinking water samples, including perfluorooctanesulfonamide (FOSA),perfluorobutanesulfonamide (FBSA), and 6:2 fluorotelomer sulfonate(6:2 FTSA) at concentrations reaching 1 ng/L.18 Most of the other short-chainPFAAs were detected in ∼80% of the samples but at lower concentrationsand contributed less than 2% to the total PFAA levels.Similar to the dust and drinking water samplesanalyzed here, ultrashort-chain PFAAs were the most abundant PFAAsin serum as well. Specifically, TFA was the predominant PFAA in serumsamples  and constituted 57% of the serumtotal PFAA concentration. PFPrA was found in 99% of the samples ata median concentration of 1.0 ng/mL and contributed ∼10% tothe serum total PFAA levels. These findings were consistent with thosefrom a recent study from China (sampling year 2017) that reportedTFA and PFPrA in serum at median concentrations of 8.5 and 0.48 ng/mL,respectively.As expected,PFOA and PFOS were also frequently detected (DF 99% each) in serum.Median concentrations of PFOA and PFOS in serum were 0.63 and 1.5ng/mL, constituting 5.9 and 14% of the total PFAA concentration, respectively.The concentrations of the C8 PFAS were lower than those of TFA andcomparable to those of PFPrA in these samples. PFHxS was found inalmost all samples (DF 99%) at levels comparable to those of PFOAand PFOS (median 0.78 ng/mL) and constituted ∼8% of the totalPFAA concentration, while the rest of the long-chain PFAAs were onlyminor contributors (<2%).e.g., PFBA and PFBS)are able to biomagnify in a terrestrial food chain,74 providing a possible explanation for the higher levelsof the short-chain PFAAs in blood. Although high solubilities and Kow values of the short-chain PFAAs indicatethat they would not significantly partition to lipids, multiple studieshave demonstrated that C4–C7 PFAAs exhibit strong binding affinitiesto blood proteins .78 Furthermore, TFA was reportedto bind to proteinaceous fractions and lipids in biota.79 These findings suggest that the protein binding affinity could bea driving force behind the bioaccumulation mechanism of the ultrashort-and short-chain PFAAs in human blood.Overall, the total ultrashort-and short-chain PFAAs contributeda greater portion to the total PFAA concentration in serum (69%),demonstrating that the levels of the shorter-chain alternatives accumulatingin blood exceed those of the legacy long-chain PFAAs. It was recentlyreported that short-chain PFAAs  were significantlylower than those of the long-chain PFAAs . The latter findings as well as thehigher water solubility of these shorter-chain PFAAs ((0.35–9.7)× 105 mg/L) compared to their longer-chain counterparts(4.7 × 10–8–0.21 mg/L) may explain themore frequent detection of the ultrashort- and short-chain PFAAs inurine.81All ultrashort- and short-chain PFAAs were detectedin urine with the DF ranging from 1.2 to 88%. PFPeA was the most abundantPFAA in urine  and constituted 89% ofthe urine total PFAA concentration. PFBA was the second most abundantPFAA detected in 60% of the urine samples at a median concentrationof 0.33 ng/mL and contributed 9.2% to the urine total PFAA concentration.PFPrA was also frequently detected (DF 56%) but at much lower concentrations(median of 0.051 ng/mL) and contributed 1.4% of the urine total PFAAconcentration. TFA was detected only in 31% of the samples but wasfound at high concentrations in some of the samples with its maximumconcentration reaching 290 ng/mL. Among the long-chain compounds,PFOA, PFOS, and PFNA were the only compounds detected in urine (DF7.4–30%). The estimated average renal clearance rates of TFA,PFPrA, and PFBA , werefound to be 1–3 orders of magnitude higher than those of PFOAand PFOS ,r = 0.40, p < 0.001 and r = 0.28, p = 0.01,respectively). The serum levels of PFBA and PFHpA were significantlycorrelated with those in water . Among the long-chain PFAAs, PFHxS andPFOS levels in dust and serum were significantly and positively associated. PFPrAwas the only PFAA for which there was a significant association betweenthe levels in water and urine .The correlations between the logarithmicallytransformed concentrations of PFAAs detected in more than 50% of thesamples across matrices were examined using Spearman correlation analysis,and the results are presented in e.g., dietary)82 forthese compounds in this population. Previous research has shown thatdaily exposure to PFBS, PFPeS, PFHxA, and PFHpA via drinking waterintake results in increased levels of these compounds in serum.30 Although short-chain PFAAs have shorter half-livesin humans compared to the legacy PFAAs,30 these compounds are not efficiently removed in water treatment processes,10 which may result in continuous exposure throughconsumption of tap water. Our findings show that the high abundanceof ultrashort- and short-chain PFAAs in drinking water from municipalsources is a potential environmental health problem that should betaken into consideration in assessing the risk of exposure to PFASin the general population. In addition, a significant positive relationshipbetween the TFA concentrations in dust and those in serum indicatesthat dust intake could also be an important exposure pathway for thiscompound.To the best of our knowledge, this is the first reportof significantcorrelations between the concentrations of the ultrashort- and short-chainPFAAs in drinking water and serum samples collected from the generalpopulation of the United States. These associations suggest that consumptionof drinking water may be a significant exposure pathway for theseshorter-chain PFAAs, even in the general population with no knownPFAS-contaminated sites nearby. On the contrary, the lack of significantassociations between the levels of the long-chain PFAAs, includingPFOA and PFOS, in drinking water and serum suggests other sources. In addition, the dust concentrations of 8:2diPAP and 8:2 FTOH were significantly correlated with the serum concentrationsof PFPrA  andPFHpA in serum ,respectively. No significant correlation was found between the dustconcentrations of PFAA precursors and urinary PFAA levels.Spearman correlation coefficients ofthe logarithmically transformed concentrations of PFAA precursorsin dust with the concentrations of the ultrashort- and short-chainPFAAs in serum and urine are shown in e.g., food packaging materials, cosmetics, personalcare products, and floor finishing).83 PAPs have been found as the most abundant PFAA precursorsin indoor dust and on human skin.84 Previous in vivo research shows that 4:2 and 6:2 mono- and diPAPscan transform to the corresponding short-chain PFCAs (4:2 PAP/diPAPto PFBA and PFPeA and 6:2 PAP/diPAP to PFHxA and PFHpA).51 Moreover, 6:2 diPAP can break down to producePFPeA in rat blood.85 However, the PFAAswith the shortest chain length, TFA and PFPrA, were not measured inthe latter studies. Considering the widespread exposure to PAPs andthe evidence of their transformation to short-chain PFAAs in animaland environmental studies, it is possible that these compounds areprecursors of short-chain PFAAs in humans. In addition, consumptionof fluorinated drugs can also contribute to the burden of TFA in humans.72 Further research is warranted to investigatethe biotransformation mechanisms of PAPs and other precursors in humans.The strong relationships between the dust concentrationsof mono-and diPAPs and the serum concentrations of the ultrashort- and short-chainPFAAs suggest similar sources for these PFAS groups. PAPs are a groupof fluorotelomer PFAS that have been widely used in various applications. In addition, thelevels of PFAAs and PFAA precursors in dust collected from homes withless frequent vacuuming were up to 3 times higher than in homes that were vacuumed more often. High total PFAA and PFAA precursorconcentrations in dust collected from homes with carpet suggest thatcarpet can be an important source of PFAAs and PFAA precursors inthe indoor environment, consistent with previous research on indoorPFAA sources.86The total PFAA andPFAA precursor dust levels in homes with carpet  were 2–4 times higher compared to homeswithout carpet . The estimatedratios of RSCwater to RSCdust for the ultrashort-and short-chain PFAAs were higher (7.1–263) than those forthe long-chain PFAAs (0.4–14), suggesting that for the shorter-chainPFAAs consumption of drinking water is a more significant exposurepathway than dust intake. These findings are consistent with previousstudies demonstrating a lower significance of PFAA intake from dustingestion.88 The RSCs determined here showthat dust and water intake only contribute 2.0–20% to the totalPFAA body burden, indicating the existence of other exposure pathwaysfor these compounds. For example, the median RSC of drinking waterintake was 11% for PFOA, consistent with previously reported findings,90 while for PFOS it was 2.8%. Another potential exposure route couldbe dietary intake (which was estimated at 66–99% for PFOS exposurein previous studies93); however, it was not evaluated in this study. Inaddition, the biotransformation of PFAA precursors frequently detectedin food packaging materials,94 cosmetics,59 and indoor dust could also be an indirect pathway for PFAA exposure.54The results of the one-compartmenttoxicokinetic (TK) model are provided in 97 Thus, the reported levels of these PFAS should beinterpreted with caution. In addition, labeled TFA standards werenot available to us when this work started; thus, the extraction efficiencyof TFA was not evaluated based on labeled standards. Finally, urinaryelimination was assumed to be the primary excretion pathway for theultrashort- and short-chain PFAAs measured in these samples, and otherexcretion pathways were not evaluated.This study had several limitations, includinga small sample size and a cohort limited in diversity and geographiccoverage. We were not able to determine the contribution of diet orbiotransformation of PFAA precursors or fluorinated pharmaceuticalsto the total body burden of ultrashort- and short-chain PFAAs. TheC2–C5 PFAAs have only one MRM transition and certain chemicalinterferences may coelute with these compounds because of a sharedquantitative ion channel when using low-resolution tandem mass spectrometryfor their analysis.in vivo and in vitro biotransformation studies of PFAAprecursors that have examined the formation of the ultrashort-chainPFAAs are lacking. Our findings warrant urgent research focused onthe ultrashort-chain PFAAs to elucidate their sources, potential humanexposure pathways, and the effects of these exposures on human health.This is the first study that reports asubstantial prevalence of the ultrashort PFAAs, C2 TFA and C3 PFPrA,in the U.S. indoor environment and the general population. In mostof the samples analyzed in this study, the levels of these two ultrashort-chainPFAAs were higher than or comparable to the levels of legacy PFOSand PFOA and constituted on average 66–92% of the total PFAAconcentrations. However, the sources of these ultrashort-chain PFAAsin this study remain unknown. The toxicokinetic model applied hereshows that consumption of drinking water and dust intake contributedonly ∼20% to the total PFAA levels in blood, suggesting otherexposure pathways for these compounds. On the other hand, high levelsof several PFAA precursors in dust and a strong relationship foundbetween the dust levels of some precursors, such as PAPs, with thoseof TFA and PFPrA in blood indicate common sources and suggest thatbiotransformation of PAPs could be a potential indirect source ofthe ultrashort-chain PFAAs in humans. However,


Learning effective molecular feature representation to facilitate molecular property prediction is of great significance for drug discovery. Recently, there has been a surge of interest in pre-training graph neural networks (GNNs) via self-supervised learning techniques to overcome the challenge of data scarcity in molecular property prediction. However, current self-supervised learning-based methods suffer from two main obstacles: the lack of a well-defined self-supervised learning strategy and the limited capacity of GNNs. Here, we propose Knowledge-guided Pre-training of Graph Transformer (KPGT), a self-supervised learning framework to alleviate the aforementioned issues and provide generalizable and robust molecular representations. The KPGT framework integrates a graph transformer specifically designed for molecular graphs and a knowledge-guided pre-training strategy, to fully capture both structural and semantic knowledge of molecules. Through extensive computational tests on 63 datasets, KPGT exhibits superior performance in predicting molecular properties across various domains. Moreover, the practical applicability of KPGT in drug discovery has been validated by identifying potential inhibitors of two antitumor targets: hematopoietic progenitor kinase 1 (HPK1) and fibroblast growth factor receptor 1 (FGFR1). Overall, KPGT can provide a powerful and useful tool for advancing the artificial intelligence (AI)-aided drug discovery process. Accurate property prediction relies on effective molecular representation. Here, the authors introduce KPGT, a knowledge-guided self-supervised framework that improves molecular representation, leading to superior predictions of molecular properties and advancing AI-driven drug discovery. In recent years, artificial intelligence (AI)-based approaches have played an increasingly pivotal role in predicting molecular properties, offering remarkable efficiency and cost-effectiveness5. One of the primary challenges of AI-based approaches for molecular property prediction is the representation of molecules7. The early machine learning-based approaches involved preliminary attempts toward representing molecules using basic handcrafted features10. Among these, the most prominent ones include molecular descriptors13, which quantitatively characterize the physical and chemical profiles of small molecules, and fingerprints16, which utilize binary strings to signify the presence of specific substructures within a molecular structure. The prediction methods based on these representations are highly dependent on complicated feature engineering strategies, consequently compromising their generalizability and flexibility.The identification of molecules with desired properties presents one of the most significant challenges in the drug discovery field, given the considerable time and resources required for experimentally determining molecular properties19, convolutional neural networks (CNNs)22, and graph neural networks (GNNs)26, excel in modeling molecular data across various formats, from the simplified molecular-input line-entry system (SMILES)27 to molecular images and molecular graphs. Nevertheless, the limited availability of labeled molecules and the vastness of the chemical space have constrained their prediction performance, particularly when handling out-of-distribution data samples29. Along with the remarkable achievements of self-supervised learning methods in the fields of natural language processing31 and computer vision33, these techniques have been employed to pre-train GNNs and improve the representation learning of molecules, leading to substantial improvements in the downstream molecular property prediction tasks42.Recent years have witnessed the emergence of deep learning-based methods as potentially useful tools for predicting molecular properties, primarily due to their remarkable capability of automatically extracting effective features from simple input data. Notably, a diverse range of neural network architectures, including recurrent neural networks (RNNs)36, GROVER38, and GEM42, typically involve the modification of molecular graphs by means of node or subgraph masking, followed by the prediction of the masked components38, or the utilization of contrastive learning objectives to align the modified graphs with their corresponding originals in latent space37. Molecules inherently possess characteristics tightly linked to their structures, which implies that even minor modifications of molecular graphs can lead to the loss of their semantic information. This natural characteristic of molecules potentially limits current self-supervised learning-based methods for molecular graphs to capture structural similarity among molecules and fails to capture the rich semantic information related to molecular properties encoded within their chemical structures 43. Moreover, in the absence of semantic information on molecular graphs, the only dependency between the masked nodes and their adjacent nodes is the valency rules, which often fail to guide the model to make accurate predictions for the masked nodes  as backbone models. However, GNNs can only provide limited model capacity, as they suffer from over-smoothing when increasing their numbers of layers47. Additionally, GNNs might struggle to capture long-range interactions between atoms, arising from their standard practice of exchanging information only among one-hop neighbors during the message passing process49. Recent advancements in backbone networks, particularly transformer-based models50, have emerged as game-changers52. These models, characterized by an increasing number of parameters and the capability to capture long-range interactions, present promising avenues to comprehensively model the structural characteristics of molecules57.Existing self-supervised learning methods generally rely on GNNs , particularly designed to accurately model molecular graph structures, with a knowledge-guided pre-training strategy aiming to capture both structural and semantic knowledge of molecules. After pre-training on a large-scale dataset consisting of approximately two million molecules, KPGT demonstrated a significant performance enhancement on 63 molecular property datasets. Moreover, we showcased the practical applicability of KPGT by successfully utilizing it to identify potential inhibitors for two antitumor targets, hematopoietic progenitor kinase 1 (HPK1) and fibroblast growth factor receptor (FGFR1). In summary, KPGT offers a powerful self-supervised learning framework for effective molecular representation learning, thereby advancing the field of AI-aided drug discovery.In this study, we introduce KPGT50, consisting of multiple layers of a multi-head attention module and a feed-forward network. LiGhT takes the molecular line graphs as input, which represent the adjacencies between edges of the original molecular graphs 28, Supervised+Attribute Masking (MaskingSup)28, Supervised+Edgepred (EdgepredSup)28, and Supervised+Infomax (InfomaxSup)28 also integrate additional knowledge by introducing supervised graph-level pre-training tasks in their self-supervised learning frameworks, such as predicting the presence of molecular motifs or bio-activities of molecules. Although the self-supervised learning strategies of these methods are not well-defined, they still outperformed other baseline methods that do not incorporate additional knowledge. This observation demonstrated the significance of integrating additional knowledge to enhance the efficacy of pre-training on molecules.We initially compared KPGT to 19 state-of-the-art self-supervised learning-based methods , which are pivotal in the field of drug discovery and development. We conducted a comparison between KPGT and 28 baseline methods from the TDC leaderboards. These baselines included 16 deep learning-based methods and 12 machine learning-based methods 66. In this evaluation, we assessed KPGT against 24 baseline methods from the MoleculeACE benchmarking platform, comprising seven deep learning-based methods and 17 machine learning-based methods  exhibited highly competitive prediction performance, even surpassing the previously designed deep learning-based methods in multiple cases. For the datasets from TDC, machine learning-based methods served as the most competitive baseline methods on 13 out of 22 datasets  for molecules from the CYP3A4 dataset using the pre-trained KPGT. Next, we randomly sampled 200 molecules from the CYP3A4 dataset to form a test set, while the remaining molecules constituted the training set. We then employed k-nearest neighbor classification (kNN) based on the neural fingerprints to make predictions for the molecules in the test set. We compared the performance of KPGT with two classic fingerprints widely used in the field, namely extended-connectivity fingerprints (ECFP)14 and RDKit fingerprint (RDKFP)44, along with two neural fingerprints derived from two state-of-the-art self-supervised learning-based methods, GROVER38 and GraphCL35. Figure After demonstrating the superiority of KPGT in predicting molecular properties, we conducted further investigations to unveil the reasons underlying its remarkable performance. We initiated this exploration by analyzing the latent space constructed by KPGT after pre-training. For this analysis, we employed a dataset comprising measurements of bio-activity  for 12,328 molecules targeting the CYP3A4 enzyme, which plays a crucial role in drug metabolismWe carried out additional tests to further verify our findings. More specifically, we first retrieved the closest data point in the latent space for each molecule from the sampled test set, yielding 200 pairs of molecules. Next, we calculated five descriptors that play important roles in drug discovery, including molecular LogP (MolLogP), molecular weight (MolWt), topological polar surface area (TPSA), number of rotatable bonds (NumRotatableBonds), quantitative estimate of drug-likeness (QED), and synthetic accessibility (SA), for the sampled molecules and queried ones. We then measured the correlation between the pairs of sampled and queried molecules for each descriptor in terms of Spearman’s rank correlation coefficient (Spearman’s r) and Pearson correlation coefficient (Pearson’s r). Figure 70 to visualize the molecular representations within the test set. From the results, KPGT provided a clear separation for the representations of inhibitors and non-inhibitors, indicating that KPGT can learn the distinguishable features of molecules with different properties in the finetuning process 75. The availability of high-quality experimental data for HPK1 and FGFR1 significantly facilitates the development and validation of the AI-based computation model, providing adequate data for evaluating the practicality and prediction performance of KPGT. In this section, we carried out evaluation tests, drug repurposing, and docking analyses for both targets, serving as a proof-of-concept validation of the effectiveness of KPGT in real-world drug discovery scenarios.Hematopoietic progenitor kinase 1 (HPK1) and fibroblast growth factor receptor (FGFR1), which are implicated in a variety of cancer types, have been extensively studied for antitumor therapyTo facilitate the identification of potent inhibitors against HPK1, we collected 4442 molecules with experimentally determined potency against HPK1 from previous patents and research, measured in terms of the negative logarithm of the half maximal inhibitory concentration (pIC50). We comprehensively evaluated the prediction performance of KPGT on this dataset using three distinct splitting approaches, including scaffold splitting, time splitting, and domain transfer (“Methods”). Comparison results between KPGT and 19 self-supervised learning-based baseline methods are detailed in Fig. 76. Then we finetuned KPGT on the pIC50 dataset of HPK1 inhibitors and made predictions for the molecules from the FDA dataset. Supplementary Table 77.We next sought to use KPGT to identify potential HPK1 inhibitors through drug repositioning. More specifically, we first obtained 2718 US Food and Drug Administration-approved (FDA) drugs (denoted as the FDA dataset) collected from DrugBank79, a widely used docking software, was employed for these tests. The reference protein-ligand structure (PDB ID: 7SIU80) guided the identification of the binding pocket. As depicted in Fig. 84, signifying the substantial potential for these molecules as HPK1 inhibitors. Additionally, we conducted an in-depth analysis of the protein-ligand interactions for all the molecules that had not been reported in the literature using a widely applied protein-ligand interaction profiler named PLIP85. Figure 80). These observations showcased that the molecules can tightly bind to HPK1, validating the reliability of the docking results. Supplementary Fig. To strengthen our findings, we further conducted docking analyses for the top 20 predictions from KPGT. Autodock Vina75. We first collected 12,461 existing molecules for FGFR1 from patents and previous studies with experimental pIC50 values. We evaluated the prediction performance of KPGT on the FGFR1 dataset under the scaffold splitting and time splitting settings. Figure 86) was employed as a reference for the binding pocket identification. As shown in Fig. 85, the ligand brigatinib was tightly bound to the protein FGR1. Specifically, it formed four hydrophobic interactions, one hydrogen bond, and one salt bridge with FGFR1 . In Supplementary Fig. Next, as a second prototypical example, we conducted tests on FGFR1, another promising druggable target associated with tumor progression and invasion57), and KPGT-LiGhT+GIN (replacing backbone model LiGhT with GIN45). Additionally, for a fair comparison, we ensured that all models had approximately the same number of parameters (around 3.5 million). Supplementary Table To validate the effectiveness of the specific design choices of KPGT, we conducted comprehensive ablation studies. Specifically, we introduced several modified frameworks based on KPGT with specific restrictions: KPGT-Pretrain (without pre-training), KPGT-KN (without knowledge nodes), KPGT-PE (without path encoding module), KPGT-DE (without distance encoding module), KPGT-LG , KPGT-LiGhT+Graphormer . This discrepancy indicated that the additional knowledge incorporated in KPGT effectively guided the model to predict the masked nodes, thereby enabling it to capture the rich semantics of molecules. Moreover, this observation aligned with the finding in the field of CV that applying larger masking rates in self-supervised learning can yield improved performance on downstream tasks32.We also investigated the effect of different masking rates on the prediction performance of KPGT. We individually pre-trained KPGT with masking rates of 15, 30, 50, and 60%, and subsequently evaluated their prediction performance. From the results, setting the masking rate to 50% achieved the best prediction performance In this study, we develop and establish KPGT, a self-supervised learning framework that offers improved, generalizable, and robust molecular property prediction through significantly enhanced molecular representation learning. By leveraging a high-capacity backbone model called LiGhT, KPGT comprehensively captures the inherent structural information within molecular graphs. More importantly, KPGT introduces a knowledge-guided pre-training strategy that can robustly address the limitations of previously ill-defined pre-training approaches, empowering our model to provide semantic-enriched molecular representations. In addition, KPGT incorporates several finetuning strategies that effectively integrate the acquired knowledge from the pre-trained model, leading to improved performance on downstream molecular property prediction tasks. With these advancements, KPGT achieved great improvement on 63 datasets compared with several baseline methods. Remarkably, KPGT showed practical applications in the identification of multiple potential inhibitors for two antitumor targets.13 can calculate over 1800 molecular descriptors, presenting an opportunity to incorporate a broader range of knowledge to further enrich the representation learning of molecules. Moreover, further studies could encompass the integration of three-dimensional (3D) molecular conformations into the pre-training process, thus enabling the model to capture vital 3D information regarding molecules, and potentially enhancing the representation learning capabilities. Additionally, while KPGT currently employs a backbone model with approximately one hundred million parameters, along with pre-training on two million molecules, exploring larger-scale pre-training could offer even more substantial benefits for molecular representation learning. Overall, we anticipate our proposed method will offer a general self-supervised learning framework for accelerating AI-aided drug discovery.Despite the advantages of KPGT for effective molecular property prediction, there remain a few limitations. First, the integration of additional knowledge is the most distinguishable feature of our proposed method. Except for the 200 molecular descriptors and 512 RDKFP employed in KPGT, there is potential to incorporate various other types of additional informative knowledge. For instance, MordredNv stands for the number of nodes. We initialize the features of nodes and edges in the molecular graph via RDKit44. The atom and bond features used in this work are summarized in Supplementary Tables vi and edge ei,j as Dv and De stand for the dimensions of features of nodes and edges, respectively.Given a SMILES representation of a molecule, we first abstract it as a molecular graph mentclass2pt{minim57, we transform the molecular graph ei,j in For each edge For every two edges in To fully exploit the structural information from molecules, especially the chemical bonds that have been neglected in the previously defined transformer architecturesWe calculate the initial feature embedding 50, which consists of multiple transformer layers. More specifically, given the node feature matrix l first feeds it into a multi-head self-attention module:Hl−1 stands for the node feature matrix at the (l − 1)-th layer, k-th head at layer l, Nh stands for the number of self-attention heads, and Hl is then passed to an FFN:87, GELU(⋅) stands for the GELU activation function88, and l.We then propose Line Graph Transformer (LiGhT) to encode features of the molecular line graph. LiGhT is built upon a classic transformer encoderDirectly applying the above classic transformer architecture can lead to a significant loss of structural information of molecules since it ignores the connectivity of graphs. Therefore, we further employ path encoding and distance encoding modules to introduce the structural information into the multi-head self-attention layer.89, Np stands for the length of the path, n-th node in the shortest path, n-th node in the path, Dp stands for the dimension of the path embedding.For each pair of nodes 57, we also leverage the distances between pairs of nodes to further encode the spatial features of the molecular line graphs. More specifically, given nodes di,j stands for the derived distance between Dd stands for the dimension of the distance embedding.Following refs. Ap and Ad are the path encoding matrix and the distance encoding matrix, respectively.Then, to introduce the encoded structural information into the model, we rewrite the formula of the attention matrix entclass1pt{minimaHere, we discuss the main advantages of our proposed model compared with the previously defined graph transformers:57. In our case, LiGhT fills this gap and fully exploits the intrinsic features of chemical bonds.First, by representing molecular graphs as line graphs, LiGhT emphasizes the importance of chemical bonds in molecules. Chemical bonds are the lasting attractions between atoms, which can be categorized into various types according to the ways they hold atoms together, resulting in different properties of the formed molecules. However, the previously defined transformer architectures either omit the edge features or only introduce chemical bonds as the bias in the self-attention module, ignoring the rich information from chemical bonds57 when encoding the paths, they only consider the edge features and ignore the node features in the paths. On the other hand, our path encoding strategy incorporates the features of the complete paths between pairs of nodes, thus encoding the structural information more precisely compared to the previous methods.Second, although strategies like path encoding have already been proposed in previous graph transformer architecturesIn summary, LiGhT provides a reliable backbone network for accurately modeling the structural and semantic information of molecular line graphs.44. Additionally, knowledge can encompass the experimentally measured characteristics of molecules, such as the comprehensive information on the bio-activities of 456,000 molecules across 1310 bioassays in the preprocessed ChEMBL dataset90. We employed 200 molecular descriptors and 512 RDKit fingerprints in our proposed method, which can be readily generated using RDKit44, a widely used cheminformatics tool. The complete list of these molecular descriptors and examples of RDKit fingerprints can be found in Supplementary Table In this study, we define knowledge as any quantifiable information that characterizes the features of molecules. This includes various types of information, such as molecular descriptors and fingerprints that are easily accessible through current cheminformatics toolsOur pre-training strategy is based on a generative self-supervised learning scheme, which first randomly selects a proportion of nodes in graphs. Then for each selected node, it is replaced with a mask token, a random node or the unchanged node with a ratio of 8:1:1. Finally, the model learns to predict the type of the original node with a cross-entropy loss. In the pre-training, we also randomly mask a proportion of the initial features of K nodes and learn to predict the masked molecular descriptors and fingerprints. The prediction of the masked molecular descriptor is formulated as a regression task equipped with an RMSE loss, while the prediction of fingerprint is formulated as a binary classification task equipped with a cross-entropy loss.61, re-initialization (ReInit)61, FLAG62 and, L2-SP63. LLRD and ReInit are proposed mainly based on the fact that different layers of a model capture different kinds of information, where the bottom layers tend to encode the information more general to the downstream tasks while the top layers tend to encode information related to the pre-training tasks. More specifically, LLRD implements the discriminative learning rates for different layers of a model. This is achieved by setting an initial learning rate of the top layer and using a multiplicative decay rate to decrease the initial learning rate layer-by-layer from top to bottom. ReInit re-initializes the parameters of the top-n layers of a model before finetuning. FLAG is a data augmentation method that iteratively augments the node features by injecting the gradient-based adversarial perturbations during finetuning. L2-SP proposes a regularization scheme to explicitly promote the similarity of the finetuned model with the initial one in the finetuning process.To fully take advantage of the abundant knowledge captured in the pre-training stage, KPGT introduces four finetuning strategies, including layer-wise learning rate decay (LLRD)91 version 1.10.0 and DGL92 version 0.7.2 with CUDA version 11.3 and Python 3.7. We implemented a 12-layer LiGhT as the backbone network with 768 hidden units and 12 self-attention heads. A mean pooling operation that averaged all the nodes in individual graphs was applied on top of the model to extract the molecular feature representations. An Adam optimizer with weight decay 1e−6 and learning rate 2e−4 was used to optimize the model. The model was trained with a batch size of 1024 for a total of 100,000 steps. The KPGT had around 100 million parameters. We set the masking rate of both nodes and additional knowledge to 0.5. The pre-training of KPGT took about two days on four Nvidia A100 GPUs. More detailed configurations of KPGT in the pre-training and finetuning processes are summarized in Supplementary Table All models were implemented in PyTorch38, a scaffold splitting scheme was utilized to partition each dataset into training, validation, and test sets with a ratio of 8:1:1. This splitting scheme ensured the molecules in the test sets differed structurally from those in the training set, offering an ideal scenario to evaluate the robustness and generalizability of prediction models. We utilized the area under the receiver operating characteristic curve (AUROC) to evaluate classification tasks, while for regression tasks, we employed the root-mean-square error (RMSE) as the evaluation metric. The results were reported based on three independent runs with different random seeds.In our first benchmarking test in the “Results” section, we employed 11 molecular property datasets, including eight classification datasets and three regression datasets. Detailed information of these datasets is available in Supplementary Note The TDC benchmarking platform provides 22 molecular property prediction datasets. These datasets span a broad spectrum of molecular properties, including absorption, distribution, metabolism, excretion, and toxicity (ADMET). We strictly adhered to the evaluation protocols provided by the original TDC benchmarking platform. Each dataset was split into training, validation, and test sets with a ratio of 7:1:2 using a scaffold splitting scheme. The mean absolute error (MAE) or Spearman’s rank correlation coefficient (Spearman’s r) was employed to evaluate the prediction performance of KPGT on regression tasks, while AUROC or the area under the precision-recall curve (AUPRC) was used to evaluate the prediction performance of KPGT on classification tasks For the tests on the datasets for HPK1, we evaluated the prediction performance of KPGT on this dataset in three distinct scenarios: (1) scaffold splitting, i.e., splitting the dataset as training, validation, and test sets with a ratio of 8:1:1 using scaffold splitting; (2) time splitting, i.e., splitting the dataset according to the time that the corresponding patents were published (the molecules from the patents published before 2021 were used as training while those molecules published after 2021 were used as testing); and (3) domain transfer, i.e., collecting 1615 molecules with the negative logarithm of inhibitory constants (pKi) against HPK1 as an independent test set. We excluded the molecules that overlapped with the pre-training data employed by KPGT in this test. Spearman’s r and Pearson correlation coefficient (Pearson’s r) were employed as the evaluation metrics. The results were reported based on three independent runs with different random seeds.For the tests on the datasets for FGFR1, we evaluated the prediction performance of KPGT in the scaffold splitting and time splitting scenarios. We excluded the molecules that overlapped with the pre-training data employed by KPGT in this test. Spearman’s r and Pearson’s r were employed as the evaluation metrics. The results were reported based on three independent runs with different random seeds.Further information on research design is available in the Supplementary InformationPeer Review FileReporting SummarySource Data


We show that existing literature documenting general performance gains of NSEs does not always conform to the results obtained by the same NSEs in topic segmentation. If Transformers-based encoders do improve over previous approaches, fine-tuning in sentence similarity tasks or even on the same topic segmentation task we aim to solve does not always equate to better performance, as results vary across method being used and domains of application. We aim to explain this phenomenon and the relative poor performance of NSEs in news shows by considering how well different NSEs encode the underlying lexical cohesion of same-topic segments; to do so, we introduce a new metric, ARP. The results from this study suggest that good topic segmentation results do not always rely on good cohesion modelling on behalf of the segmenter and that is dependent upon what kind of text we are trying to segment. Also, it appears evident that traditional sentence encoders fail to create topically cohesive clusters of segments when used on conversational data. Overall, this work advances our understanding of the use of NSEs in topic segmentation and of the general factors determining the success (or failure) of a topic segmentation system. The new proposed metric can quantify the lexical cohesion of a multi-topic document under different sentence encoders and, as such, might have many different uses in future research, some of which we suggest in our conclusions.Neural sentence encoders (NSE) are effective in many NLP tasks, including topic segmentation. However, no systematic comparison of their performance in topic segmentation has been performed. Here, we present such a comparison, using supervised and unsupervised segmentation models based on NSEs. We first compare results with baselines, showing that the use of NSEs does often provide improvements, except for specific domains such as news shows. We then compare over three different datasets a range of existing NSEs and a new NSE based on  Topic segmentation—automatically segmenting a long text into topically coherent units—is a well known problem in natural language processing, and the first step for a number of downstream applications. A long transcript from a news show, for example, could be divided into single news stories so as to help an end user in retrieving more relevant and specific information  or a lonIn the mean time, however, methods for determining and expressing similarities between texts have seen a major shift in NLP from word count based methods to geometric distance between dense text representations from pre-trained neural representations , which . DrawingRQ1: Do NSEs lead to better performance than previously used features in topic segmentation?RQ2: Which NSEs work best for this task? Is their ranking consistent with what existing literature has noted about their general performance and how it varies across domains?RQ3: Is the success (or failure) of NSEs always a consequence of better lexical cohesion modelling by the encoders?While answering the first two questions requires a survey of different encoders, the third one involves the creation of new techniques to quantify lexical cohesion under different encoders and datasets. We do that by introducing a new metric, Average Relative Proximity (ARP). We show that NSEs often do improve over previously used features, but in certain domains improvements are not always significant; and NSEs that are successful in a variety of text similarity tasks do not necessarily improve performance in topic segmentation. We investigate and explain this with the fact that the factors behind topic segmentation success are not necessarily correlated with a better capacity of identifying cohesive segments and, in certain domains, topic segmentation can rely much more on local cues, making it more similar to a text classification task.Traditionally, topic segmentation involves the segmentation of text such as books or articles , and traOne of the earliest techniques for topic segmentation, TextTiling, used two adjacent sliding windows over sentences and compared the two by means of cosine similarity between the relative bag-of-words vector representations . The samAnother early approach used the distance between sentence representations in a dynamic programming framework. Similarly to TextTiling, initial work used bag-of-words sentence representations , with lacorpus, showing good improvements over non-neural and unsupervised methods. Since then, most literature has focused on using hierarchical recurrent neural networks , which has been often used in the context of pre-trained models for tasks such as semantic retrieval, question answering and, generally, tasks for which a single output is produced for one or multiple sentences . This doFor many bigger than word-level tasks, in fact, neural models require suitable text representations. Early approaches used simple averaging of word vectors in a sentence or paragraph to give a representation of their semantics, and this can perform quite well in many tasks . More recorpus . At the same time, using sentences or utterances is advantageous when using text representations such as those of interest here, as such units of text are closer to the original pre-training settings of such models, which often are not capable of encoding extremely long chunks of text. In this work then we adopt this setting and we predict whether a topic boundary occur at the level of each individual sentence (in the case of monologue datasets) or speaker turn —see details in the Dataset section.Previous work has variously used words, sentences, utterances or paragraphs as the input for topic segmentation systems. Recent work has mostly settled on the use of sentences as the basic units for monologue text , and uttThis approach comes with the obvious limitation of potentially missing cases in which there might be more than one topic shift inside a single sentence or speaker turn. In all domains we consider, however, and in the majority of work on topic segmentation the granularity of the segmentation is much coarser, as we aim to individuate thematic units that typically span multiple sentences. The annotations for all the datasets we used, therefore, never included multiple topic boundaries in a single sentence, justifying the use of sentences as basic units.In general, In such a pipeline, the input (a monologue text or a multi-party conversation transcript) is first tokenized into sentences or speaker turns Specifically, we have used the popular punkt sentence tokenizer  to extraAs mentioned, previous works experimented with some NSEs in the context of topic segmentation. These early attempts did not focus on explicitly comparing different encoders; to do that, in this work we compare 13 different NSEs of which four pre-date large transformer-based language models, six are derived from the popular transformer encoder BERT  and its 1https://huggingface.co/ighina/roberta_topseg_wiki (pre-trained on Wikisection), https://huggingface.co/ighina/RoBERTa_TopSeg_BBC (pre-trained on BBC3000) and https://huggingface.co/ighina/RoBERTa_TopSeg_QMSUM (pre-trained on QMSUM).The models, each pre-trained on the respective target dataset, are publicly available at: , while for all the other models we used the original implementations and pre-trained weights2https://www.sbert.net/index.html), for BERT and RoBERTa, the torch implementation from huggingface was used (https://huggingface.co/docs/transformers/model_doc/bert), for DAN and USE the implementation from tensorflow_hub , while for infersent and SimCSE we used their official implementation respectively .We used the sentence_transformers library for extracting para-xlm and GloVe-based sentence embeddings : This architecture from the universal sentence encoders’ family consists of a deep averaging network (DAN) that simply averages word embeddings as an input for sentence-level pre-training. This network has been pre-trained on the tasks and with the methodologies described by Universal Sentence Encoder V5 (USE): This architecture from the universal sentence encoders’ family consists of a transformer encoder pre-trained in the same way as the more compact DAN architecture (itecture . We abbrInfersent (Inf): The infersent architecture was proposed by Average GloVe embeddings (GloVe): This is a very common baseline for sentence encoders that simply averages word embeddings inside a sentence, in this case using GloVe word embeddings .RoBERTa CLS token (RoBERTa): RoBERTa has been proposed as a robustly optimised version of BERT, capable of outperforming the original BERT in a variety of tasks : This model derives from RoBERTa, but here the sentence representation is obtained by averaging all output tokens from the last layer of the RoBERTa model. The model is further fine-tuned on a dataset of paraphrases to make vectors of paraphrases closer to each other in the embedding space, as described by SimCSE RoBERTa (SimCSE): This encoder is a version of RoBERTa architecture fine-tuned with a contrastive unsupervised objective described by via the contrastive objective described by https://github.com/princeton-nlp/SimCSE. By using SimCSE and Para-xlm we aim to test whether those pre-training strategies that improve performance of sentence encoders for a variety of other tasks : We further include a NSE specifically pre-trained for topic segmentation in the target domain. To do so, we employed the training set of each dataset we used to fine-tune RoBERTa with the objective of making sentences (or speaker turns in the case of QMSUM) from the same topic segment closer in the embedding space. This encoder draws from the evidence collected by Below we present the topic segmentation models using NSEs, alongside with a description of how they work and (if applicable) how they were trained.DeepTiling : this system implements the classic TextTiling algorithm, but using neural sentence embeddings instead of the bag-of-words features used in the original implementation.Having a NSE , where Cue Word: This baseline is a simple rule-based method. We automatically detect the ten most frequent words from the final sentence (excluding function and stop words) of each segment in the training set and we then derive a rule to create a new segment where at least one of these words is detected. This method, as we will see, can be surprisingly effective in scenarios in which segments tend to end with a formulaic expression, such as those appearing in certain news shows.We used three datasets covering a range of domains: face-to-face meetings (QMSUM), written articles (Wikisection) and TV news broadcasts (BBC3000). The datasets have been chosen to cover the type of domains in which topic segmentation has been traditionally investigated: the conversational domain, where usually datasets extracted from business meetings have been used ; the booQMSUM .For every NSE-BiLSTM and TextSeg experiment we used 256 hidden units and two recurrent layers, to match the original configuration of lgorithm  was usedvia grid-search over the following sets of parameters: For the unsupervised systems, we have found the optimal values for threshold and window size on the training set of each dataset and encoder For TopicTiling, we first fitted a topic model with 50 hidden topics on each respective training set and then used that to obtain posterior probabilities for the words in each dataset. Before fitting TextTiling and TopicTiling, stop words were removed and lemmatization was applied.via bootstrapping the difference in average test results as explained in Finally, we also tested the hypothesis of the results of the best performing of our models for every dataset being significantly better than the best performing baseline from the relative dataset, by using a two-sample hypothesis testing for mean difference A general look at the table confirms that the answer to our first question is positive: NSEs seem to outperform all of the baselines, as at least one of the neural sentence encoders proposed always perform better than the baselines based on word-level or non-neural features.If we compare DeepTiling with the other two unsupervised baselines, TextTiling and TopicTiling, we see that in fact all NSEs perform better than the baselines, except for the very good performance of TextTiling in BBC3000 which places it in 6th position.In the case of TextSeg, the baseline model is competitive with NSE-BiLSTM in Wikisection and BBC3000, where it beats most non-transformer NSEs. In Wikisection the supervised models using transformer-based encoders lead to improvements over this baseline and the best configuration using TopSeg encoder significantly outperforms TextSeg.In the case of BBC3000 dataset, instead, using TextSeg leads to mostly better results than NSE-based models and, even though, the best NSE configuration is marginally better than the TextSeg baseline, such an improvement is not significant. This evidence is discussed in more details later, where the specific case of the news shows domain is analysed in more details.For what concerns the conversational domain, TextSeg performs worse than the majority of the NSE-BiLSTM models but for the ones based on GloVe and, surprisingly, A surprising result is that of fine-tuned transformers for BBC3000. In this case they all underperform with respect to the relative baselines, but for TopSeg being the best NSE in DeepTiling. This evidence will be further explored in the next section.As a quick note about the random and rule-based baselines, all models seem to generally perform better than these settings but for the striking results of Cue Word baseline on QMSUM and, especially, BBC3000 where it outperforms most unsupervised systems. This evidence is indeed very interesting and the next sections will expand on this.When we compare the various sentence encoders and our two models presented in In terms of encoders used, Transformer-based encoders generally outperform the alternatives, in line with existing literature on sentence encoders. GloVe consistently performs the worst when used with NSE-BiLSTM, with the only notable exception of Wikisection, where Infersent is marginally worse even than GloVe.If we look at the ranking of the encoders for DeepTiling or even at the ranking of Transformer-based encoders for NSE-BiLSTM, however, we realise that results contradict existing literature in many respects.DeepTiling seems to benefit from GloVe embeddings more than from RoBERTa (Wikisection) or BERT based ones (QMSUM): this seems at odds with the evident superiority of those methods when using a supervised approach, while also contradicting the ranking of these encoding methods in existing benchmarks for sentence encoder evaluation. Different pooling strategies for BERT and RoBERTa do not necessarily improve results when looking at DeepTiling, while NSE-BiLSTM largely confirm that averaging different layers result in better results than simply using the CLS token; a notable exception to this rule, however, can also be seen in the supervised results of BERT CLS, which is the best among the non fine-tuned encoders.Looking at the fine-tuned RoBERTa models, it can immediately be seen that in general fine-tuning the basic model yields bigger improvements for DeepTiling rather than for NSE-BiLSTM. A striking result is the fact that TopSeg does not perform the best when coupled with NSE-BiLSTM for QMSUM and BBC datasets. As the encoder is specifically fine-tuned on the topic segmentation task we would have expected that setting to always score the best.This is indeed the case when looking at DeepTiling, that is hardly surprising considering that in fact the cosine distance between adjacent embeddings is at the base both of the pre-training strategy and of the unsupervised algorithm. Using TopSeg with DeepTiling, then, is equivalent to injecting information into the model in a supervised fashion, which obviously gives a great advantage to this setting when compared with the other totally unsupervised ones.The discrepancies between literature on neural sentence encoders and the use of the same in topic segmentation are evident in the fact that methods such as SimCSE and Para-XLM do not seem to yield any consistent improvement over the non fine-tuned baselines and this is true both when looking at NSE-BiLSTM and  DeepTiling models.The fact that transformer-based models tend to be better instead conforms to what we could have expected and suggest that maybe a better interpretation of the results can be achieved by thinking at each individual dataset.The three domains we are looking at, in fact, present notable differences with each other, and the internal segment coherence might have a very different role in a conversational setting such as QMSUM and a text-based benchmark like Wikisection. This is evident when we look more closely at the varying degree of success that a simple rule-based baseline like Cue Word can have in different datasets. The simple baseline performs worse than any other system for a dataset like Wikisection, it is competitive in QMSUM and it actually outperforms every DeepTiling configuration but the TopSeg one for BBC3000.We hypothesise that part of the reason for this phenomenon lies in the fact that certain domains present more conventional, formulaic ways of terminating a topic than others. In news shows, especially, repetition of the name of the channel or a greeting to the current reporter are both common ways in which a segment might end, while in a meeting various disfluencies such as “um” or fillers such as “right” tend to appear more often towards the end of discussing a topic, as there are fewer things left to say and interlocutors start to consider whether they forgot something .The same logic highlighting the importance of specific cue words for achieving good segmentation results in certain domains is certainly to be linked to the good results of TextSeg for BBC3000 shown in the previous section. TextSeg is a hierarchical supervised system modelling word-level embeddings. At the same time, it uses a max pooling approach to create sentence representations from words  and lateThe factors leading to a specific model or neural sentence encoder outperforming another one in topic segmentation, therefore, are not limited to how well that encoder can encode similar texts closer in space, as supervised systems such as NSE-BiLSTM and, especially, TextSeg might actually focus more on individuating specific transition marks, while even a model like DeepTiling might perform well enough thanks to a closing sentence being on average further in the embedding space from other sentences.In the next section, then, we introduce a novel method to quantify the cohesion of ground truth segments under each dataset and each embedding. By correlating this quantity with our results we aim to isolate the influence of embeddings’ coherence on topic segmentation results, so to better answer the point raised above.Recent work in topic modelling has showed how simple clustering performed on top of sentence embeddings is an effective way of obtaining coherent topics that can in many cases perform better than more complex neural and traditional topic models . At the Therefore, we quantify this phenomenon directly in the embedding space as the total variance of a group of embeddings that are supposed to be in the same segment according to ground truth topic segmentation labels . When compared to the variance of embeddings from different, adjacent segments , this can give us an idea of how well each encoder can embed individual sentences from the same segment to be closer in the embedding space as opposed to sentences from different segments. Secondly, by comparing such a quantity of relative proximity to downstream topic segmentation results, we can draw conclusions about the importance that encoding same segment closer in space has in different domains.More formally, we define Average Relative Proximity (ARP) as a metric to evaluate the overall coherence of segments in the embedding space. The method first computes the intra-segmental variance:In order to overcome the standard deviation bias of getting bigger for smaller samples, we force the intra-variance to be computed on a collection of embeddings having the same length as the ones from which the inter-variance is computed. This is simply done by adding a cutting point The two scores are aggregated to obtain a single score representing the relative proximity (RP) of embeddings belonging to the same segment. We compute it asi.e., the sum of the two variances) forces the score in the range of While the constant 100 serves solely the purpose of better visualisation, the normalization factor  overlapping topics underlying the entire corpus.The ARP method thus defined can highlight the relative cohesion of a sequence of topically coherent segments. The method differs from previous coherence measures used for topic modelling evaluationWhile the ranking can tell us which encoders seem to be more successful in encoding coherent segments, showing the ARP score for random segments is useful to check that indeed the ARP scores for any given setting are better than random.In this way we can immediately see that this condition holds true for all the encoders in Wikisection and BBC, but not for the worst performing encoders in QMSUM. This evidence is likely to reflect what we observed in the previous section about this dataset and its conversational nature. In conversation, given the numerous disfluencies and speech acts, we do expect embeddings to be distributed more randomly within the same segment and the low ARP scores are indeed a strong confirmation that this is happening and, as a consequence, the ability of encoders to embed intra-segmental sentences closer in space might be less relevant in this context.The same can be observed for BBC3000; even though all sentence encoders are able to encode intra-segmental sentences closer in space, resulting in positive and significantly greater than 0 ARP scores, the magnitude of the scores is quite low compared to what we can observe in the case of Wikisection, once more reflecting a lower intra-segmental coherence in that dataset/domain.We can further confirm the above intuition by looking at QMSUM, instead, consistently shows relatively weak correlation between ARP scores and topic segmentation results, further confirming what we observed before.Finally, we can look at the ranking of the encoders by their relative ARP scores and draw some conclusions about these as well, while keeping in mind the bigger picture just described.In almost every case, TopSeg scores the highest as it has been directly trained for representing coherent segments in the target domain. We can see that the other two fine-tuned encoders, Para-xlm and SimCSE, have varying success in the same task, as their ARP scores are quite low for BBC3000 and QMSUM. A very surprising result is that of Infersent placing itself first for QMSUM; even though this would be worth further exploration, this can also be partly explained by the fact that scores are anyway always very poor in this dataset. The relatively good performance of Transformer-based encoders, however, seems to be confirmed also under this metric, as in every case the top three encoders mostly belong to that category.ad hoc models. If we look at the ARP scores in conversational domains, then, it becomes evident that sentence encoders fine-tuned for textual similarity fail to take into consideration the longer context and probably to disregard the redundant information that multi-sentence segments might present, leading to a limited ability to encode same segments closer in a coherent embedding space.1) Existing sentence encoders pre-trained on large corpora for textual similarity do not transfer the same good performance to topic segmentation. This is evident by the non-convincing performance of Para-xlm and SimCSE; while TopSeg trained on the target domain is able to perform much more convincingly, still simple methods like CLS token from BERT or RoBERTa perform competively with respect to more 2) Even overcoming the limitations described above would not guarantee better segmentation results, as in certain settings the problem of topic segmentation can partly be solved by text classification. Where specific cues are present that signal a potential topic boundary, the importance of encoding the more global coherence of the underlying text becomes less relevant.If we combine the individual ARP scores with the more general picture that we drew before when looking at the correlation map with segmentation results, two main points can be made:Finally, ARP scores help analysing the encoders under a different light, but could also work as feature selector as they successfully place in the top three encoders the ones that end up having the best topic segmentation results 5 out of 6 times. Future work might explore further the potential of ARP scores in pre-selecting the correct sentence encoders without the need for any additional training; this direction is especially worth considering for domains such as written articles like Wikisection, where ARP scores are strongly correlated with downstream segmentation performance.The evidence thus collected has shown that text similarity benchmarks commonly used in much literature advancing the state of the art of NSEs do not reflect the performance of those same NSEs on the task of topic segmentation. The ARP scores we defined in this section further confirm this evidence: even though the score itself is closely related to text similarity, in fact, it is clear that SimCSE and Para-xlm, both of which have shown good improvements on standard text similarity benchmarks , do not via a number of different factors:1) Regarding performance on QMSUM, standard benchmarks typically do not involve conversational data.2) Notwithstanding this limitation, the semantic granularity needed to solve standard text similarity tasks is usually finer than that needed for topic segmentation. Topic segmentation, in fact, aims to find parts of a text related to a coherent semantic field or domain: even just the fact of identifying such a field is a non-trivial task that might depend on a certain level of world knowledge  and it uFrom a qualitative point of view, we can explain this fact Focusing on this last point, especially, we can qualitatively analyse the similarity assigned by different encoders to specific pair of sentences.From the table we can see that different NSEs tend in fact to manifest a different behaviour in how closely related they consider pairs of sentences. Para-xlm and SimCSE show a behaviour consistent with traditional semantic text similarity tasks, reflecting their pre-training and they assign high scores to paraphrases and entailing sentences, while they assign very low scores to the semantically unrelated pair. Both of these encoders assign very low cosine similarity also to the topically related pair, too, reflecting their relatively poor performance in downstream topic segmentation.On the other hand, The encoder we pre-trained on topic segmentation itself, TopSeg, gives high scores to Paraphrases, NLI and Topic Segmentation pairs and a low score to the unrelated sentences. In particular, it can be seen that even though it still scores paraphrases and even entailing sentences much higher, the score assigned to the pair of sentences from the same topic segment is considerably higher than the one assigned to the unrelated pair, unlike the case of BERT.tallness. The encoder fine-tuned on topic segmentation, then, as well as BERT seem to encode as important certain collocational relationships , whereas encoders fine-tuned just on paraphrases or entailment detection might have forgotten this kind of relationships as part of their specific fine-tuning.Even by just looking at the different pairs in the table it is clear how topic segmentation fundamentally differs from entailing and paraphrasing sentences. While the paraphrases are clearly about a set of specific entities, in the case of the topic segmentation example, the relation between the two sentences mostly concern a semantic field that we could loosely refer to as At the same time, it looks like fine-tuning an encoder on topic segmentation as we did with TopSeg also allows it to recognise totally unrelated sentences in a more convincing way than using a general pre-trained language model such as BERT, suggesting that topic segmentation itself could be a valid pre-training strategy for text similarity-related tasks.Overall, this section does not aim to give an exhaustive analysis, but these initial observations together with the analysis carried out so far are a strong indication that topic segmentation can rely on text similarity, but with a fundamentally different level of granularity such that approaches otherwise effective for text similarity fail when encoding topically related sentences closer in space.Future research can then draw on these initial conclusions in a variety of ways, such as exploring topic segmentation as a pre-training objective for general text similarity or adding the task to existing benchmarks in order to account for such differences in semantic granularity required.1) We established the superiority of models built on NSEs by drawing a comparison of such models against existing baselines across different datasets.2) We presented a systematic comparison of NSEs for the task under a common framework including one supervised and one unsupervised model. As a result, we provided a ranking of different NSEs in the context of topic segmentation.3) We have shown our results on three datasets representing three typical domains in which topic segmentation is used. By doing so, we have highlighted the differences between such domains in terms of topic segmentation performance of different models and NSEs, but also in terms of what the models might focus on in different situations.4) We also introduced a metric, ARP, to directly evaluate the internal coherence of ground truth segments under different NSEs.In this work we have evaluated the influence of pre-trained NSEs on topic segmentation. In doing so, our main contributions have been:1) Using NSEs as feature extractors mostly outperforms similar methods based on hierarchical networks, lexical or topical features.2) The ranking of NSEs for the task does conform to existing literature on text similarity in certain respects (transformer-based models outperform other methods), but seems to diverge from it in others. Especially, models fine-tuned for text similarity and even for topic segmentation do not perform convincingly better than their non fine-tuned counterparts. Also, better pooling strategies failed to yield consistent improvements.3) Following the above conclusion, we have shown that NSEs that perform better for topic segmentation do not always embed sentences from the same segment closer in space as opposed to sentences spanning multiple segments, as might be expected under the assumption that the models capture the underlying topical coherence of the data. Especially for domains like news shows and especially for supervised models that can learn what to focus on, this discrepancy is a strong hint towards the fact that topic segmentation does not always involve recognising the underlying topical coherence. We hypothesise that in these cases, the recognition of recurrent expressions or even individual words might be more relevant for good topic segmentation results.per se an interesting evidence of the limitations of well-established sentence encoders in such domains.4) Notwithstanding the above, our measure of relative topical coherence also highlighted how NSEs do a better job in encoding sentences from the same segment closer in space when applied on natively textual data like Wikipedia articles, while performance drops with news shows transcripts, and their capacity to convincingly reduce inter-segmental embedding variance for meeting transcripts seems practically null. This is probably a result of the high frequency in this latter domain of utterances that may perform important dialogue functions but do not contain topical semantic content, and it is i.e., thematically/topically similar as opposed to be direct paraphrases or entailing each others).5) When text similarity plays a role in recognising topic boundaries, it looks like the type of text similarity involved in such a task is fundamentally different and probably looser than the one accounted for in traditional benchmarks for semantic similarity. This can be inferred by the poor performance both in topic segmentation and ARP scores shown by NSEs that performed very well on traditional sentence similarity benchmarks and it is further confirmed by qualitative analysis. This last piece of evidence, then, frames the task of topic segmentation as a fundamentally different task, such that a research direction worth pursuing is that of including this task in existing benchmarks for text similarity so as to account for the different levels of granularity at which we want to consider two pieces of text similar (Our experiments pointed towards the following conclusions with respect to the use of NSE for topic segmentation:Overall, our contribution is the first to our knowledge to compare different NSEs for topic segmentation in different domains. In doing so, we highlighted interesting convergences and divergences with existing literature and we have empirically shown how the task of topic segmentation might look very different in different domains.In answering our third question, we introduced ARP scores as an indicator of the relative coherence of multi-topic documents under different NSEs, measuring the lexical cohesion of portions of text in the embedding space. This score is a relevant contribution by itself, as it can work as an effective feature selection method for topic segmentation, but it can also help answering a variety of other related questions.Future research might investigate whether the use of ARP scores in conjunction with datasets such as Wikisection, in which the scores are highly correlated with topic segmentation performance, could make a good addition to existing benchmarks for evaluating NSEs performance. The task of topic segmentation, in fact, seems to have peculiar characteristics that make it different from usual text similarity tasks commonly used to evaluate NSEs; under this respect, adding the information of how these features work for topic segmentation could enrich existing benchmarks with a novel angle.From our experiments we could conclude that topic segmentation systems in different domains might focus on things different from lexical cohesion, especially where specific cues are present, which might be useful for segmentation.e.g., dialogue topic segmentation, where cue sentences played a role in a number of proposed systems (This evidence has been indirectly observed by previous literature in,  systems . Similar systems .In this appendix to our main work, we further develop on our hypothesis for which lexical cohesion plays a minor role in automatic topic segmentation systems for domains in which specific segment-terminal cues are present. To do so, we re-run the NSE-BiLSTM experiments using the same settings and encoders we used in our main experiments, but this time we deleted the last sentence of each segment from the training set.For every dataset, then, we applied a pre-processing step for which we delete all sentences or speaker turns Having applied this pre-processing step, we then re-train all the NSE-BiLSTM models following the same approach and parameters described in the main body of this work.We then re-run the correlation analysis presented in “Correlation of ARP Scores with Topic Segmentation Results” to observe how the results from different encoders compare to ARP scores at this point.The results from If we look at Wikisection and QMSUM, instead, we can observe that the correlation coefficient does not vary significantly and this suggest that in these cases the effect of cue phrases was less pronounced.