Distributing File—based Data to Remote sites within the BABAR Collaboration

在线阅读 下载PDF 导出详情
摘要 BABAR[1]usestwoformatsforitsdata:ObjectivitydatabaseandROOT[1]files.Thisposterconcernsthedistributionofthelatter-forObjectivitydatasee[3].TheBABARanalysisdataisstoredinROOTfiles-oneperphysicsrunandanalysisselectionchannel-maintainedinalargedirectorytree,CurrentlyBABARhasmorethan4.5TBytesin200,000ROOTfiles.Thisdatais(mostly)producedatSLAC,butisrequiredforanalysisatuniversitiesandresearchcentresthroughouttheUSandEurope.TwobasicproblemsconfrontuswhenweseektoimportbulkdatafromSLACtoaninstitute'slocalstorageviathenetwork.Wemustdeterminewhichfilesmustbeimported(dependingonthelocalsiterequirementsandwhichfileshavealreadybeenimported),andwemustmaketheoptimumuseofthenetworkwhentransferringthedata,Basicftp-liketools(ftp,scp,etc)donotattempttosolvethefirstproblem.Moresophisticatedtoolslikersync[4],thewidely-usedmirror/synchronisationprogram,comparelocalandremotefilesystems,checkingforchanges(basedonfiledate,sizeand,ifdesired,anelaboratechecksum)inordertoonlycopynewormodifiedfiles,Howeverrsyncallowsforonlylimitedfileselection.Alsowhen,asinBABAR,anextremelylargedirectorystructuremustbescanned,rsynccantakeseveralhoursjusttodeterminewhichfilesneedtobecopied.Althoughrsync(andscp)provideson-the=flycompression,itdoesnotallowustooptimisethenetworktransferbyusingmultiplestreams,abjustingtheTCPwindowsizeorseparatingencryptedauthenticationfromunencrypteddatachannels.
机构地区 不详
出版日期 2001年01月11日(中国期刊网平台首次上网日期,不代表论文的发表时间)
  • 相关文献