Learning With Kernels Support Vector Machines Regularization Optimization And Beyond.pdf

Learning With Kernels Support Vector Machines Regularization Optimization And Beyond.pdf

《Learning With Kernels Support Vector Machines Regularization Optimization And Beyond.pdf》由会员分享，可在线阅读，更多相关《Learning With Kernels Support Vector Machines Regularization Optimization And Beyond.pdf（640页珍藏版）》请在冰豆网上搜索。

LearningwithKernelsAdaptiveComputationandMachineLearningThomasDietterich,EditorChristopherBishop,DavidHeckerman,MichaelJordan,andMichaelKearns,As-sociateEditorsBioinformatics:

TheMachineLearningApproach,PierreBaldiandSrenBrunakReinforcementLearning:

AnIntroduction,RichardS.SuttonandAndrewG.BartoGraphicalModelsforMachineLearningandDigitalCommunication,BrendanJ.FreyLearninginGraphicalModels,MichaelI.JordanCausation,Prediction,andSearch,secondedition,PeterSpirtes,ClarkGlymour,andRichardScheinesPrinciplesofDataMining,DavidHand,HeikkiMannila,andPadhraicSmythBioinformatics:

TheMachineLearningApproach,secondedition,PierreBaldiandSrenBrunakLearningKernelClassifiers:

TheoryandAlgorithms,RalfHerbrichLearningwithKernels:

SupportVectorMachines,Regularization,Optimization,andBeyond,BernhardScholkopfandAlexanderJ.SmolaLearningwithKernelsSupportVectorMachines,Regularization,Optimization,andBeyondBernhardScholkopfAlexanderJ.SmolaTheMITPressCambridge,MassachusettsLondon,Englandc2002MassachusettsInstituteofTechnologyAllrightsreserved.Nopartofthisbookmaybereproducedinanyformbyanyelectronicormechanicalmeans（includingphotocopying,recording,orinformationstorageandretrieval）withoutpermissioninwritingfromthepublisher.TypesetbytheauthorsusingLATEX2PrintedandboundintheUnitedStatesofAmericaLibraryofCongressCataloging-in-PublicationDataLearningwithKernelsSupportVectorMachines,Regularization,OptimizationandBeyond/byBernhardScholkopf,AlexanderJ.Smola.p.cm.Includesbibliographicalreferencesandindex.ISBN0-262-19475-9（alk.paper）1.Machinelearning.2.Algorithms.3.KernelfunctionsI.Scholkopf,Bernhard.II.Smola,AlexanderJ.ToourparentsContentsSeriesForewordxiiiPrefacexv1ATutorialIntroduction11.1DataRepresentationandSimilarity.11.2ASimplePatternRecognitionAlgorithm.41.3SomeInsightsFromStatisticalLearningTheory.61.4HyperplaneClassifiers.111.5SupportVectorClassification.151.6SupportVectorRegression.171.7KernelPrincipalComponentAnalysis.191.8EmpiricalResultsandImplementations.21ICONCEPTSANDTOOLS232Kernels252.1ProductFeatures.262.2TheRepresentationofSimilaritiesinLinearSpaces.292.3ExamplesandPropertiesofKernels.452.4TheRepresentationofDissimilaritiesinLinearSpaces.482.5Summary.552.6Problems.553RiskandLossFunctions613.1LossFunctions.623.2TestErrorandExpectedRisk.653.3AStatisticalPerspective.683.4RobustEstimators.753.5Summary.833.6Problems.844Regularization874.1TheRegularizedRiskFunctional.88viiiContents4.2TheRepresenterTheorem.894.3RegularizationOperators.924.4TranslationInvariantKernels.964.5TranslationInvariantKernelsinHigherDimensions.1054.6DotProductKernels.1104.7Multi-OutputRegularization.1134.8SemiparametricRegularization.1154.9CoefficientBasedRegularization.1184.10Summary.1214.11Problems.1225ElementsofStatisticalLearningTheory1255.1Introduction.1255.2TheLawofLargeNumbers.1285.3WhenDoesLearningWork:

theQuestionofConsistency.1315.4UniformConvergenceandConsistency.1315.5HowtoDeriveaVCBound.1345.6AModelSelectionExample.1445.7Summary.1465.8Problems.1466Optimization1496.1ConvexOptimization.1506.2UnconstrainedProblems.1546.3ConstrainedProblems.1656.4InteriorPointMethods.1756.5MaximumSearchProblems.1796.6Summary.1836.7Problems.184IISUPPORTVECTORMACHINES1877PatternRecognition1897.1SeparatingHyperplanes.1897.2TheRoleoftheMargin.1927.3OptimalMarginHyperplanes.1967.4NonlinearSupportVectorClassifiers.2007.5SoftMarginHyperplanes.2047.6Multi-ClassClassification.2117.7VariationsonaTheme.2147.8Experiments.2157.9Summary.2227.10Problems.222Contentsix8Single-ClassProblems:

QuantileEstimationandNoveltyDetection2278.1Introduction.2288.2ADistributionsSupportandQuantiles.2298.3Algorithms.2308.4Optimization.2348.5Theory.2368.6Discussion.2418.7Experiments.2438.8Summary.2478.9Problems.2489RegressionEstimation2519.1LinearRegressionwithInsensitiveLossFunction.2519.2DualProblems.2549.3-SVRegression.2609.4ConvexCombinationsand1-Norms.2669.5ParametricInsensitivityModels.2699.6Applications.2729.7Summary.2739.8Problems.27410Implementation27910.1TricksoftheTrade.28110.2SparseGreedyMatrixApproximation.28810.3InteriorPointAlgorithms.29510.4SubsetSelectionMethods.30010.5SequentialMinimalOptimization.30510.6IterativeMethods.31210.7Summary.32710.8Problems.32911IncorporatingInvariances33311.1PriorKnowledge.33311.2TransformationInvariance.33511.3TheVirtualSVMethod.33711.4ConstructingInvarianceKernels.34311.5TheJitteredSVMethod.35411.6Summary.35611.7Problems.35712LearningTheoryRevisited35912.1ConcentrationofMeasureInequalities.36012.2Leave-One-OutEstimates.36612.3PAC-BayesianBounds.38112.4Operator-TheoreticMethodsinLearningTheory.391xContents12.5Summary.40312.6Problems.404IIIKERNELMETHODS40513DesigningKernels40713.1TricksforConstructingKernels.40813.2StringKernels.41213.3Locality-ImprovedKernels.41413.4NaturalKernels.41813.5Summary.42313.6Problems.42314KernelFeatureExtraction42714.1Introduction.42714.2KernelPCA.42914.3KernelPCAExperiments.43714.4AFrameworkforFeatureExtraction.44214.5AlgorithmsforSparseKFA.44714.6KFAExperiments.45014.7Summary.45114.8Problems.45215KernelFisherDiscriminant45715.1Introduction.45715.2FishersDiscriminantinFeatureSpace.45815.3EfficientTrainingofKernelFisherDiscriminants.46015.4ProbabilisticOutputs.46415.5Experiments.46615.6Summary.46715.7Problems.46816BayesianKernelMethods46916.1Bayesics.47016.2InferenceMethods.47516.3GaussianProcesses.48016.4ImplementationofGaussianProcesses.48816.5LaplacianProcesses.49916.6RelevanceVectorMachines.50616.7Summary.51116.8Problems.51317RegularizedPrincipalManifolds51717.1ACodingFramework.518Contentsxi17.2ARegularizedQuantizationFunctional.52217.3AnAlgorithmforMinimizingRregf.52617.4ConnectionstoOtherAlgorithms.52917.5UniformConvergenceBounds.53317.6Experiments.53717.7Summary.53917.8Problems.54018Pre-ImagesandReducedSetMethods54318.1ThePre-ImageProblem.54418.2FindingApproximatePre-Images.54718.3ReducedSetMethods.55218.4ReducedSetSelectionMethods.55418.5ReducedSetConstructionMethods.56118.6SequentialEvaluationofReducedSetExpansions.56418.7Summary.56618.8Problems.567AAddenda569A.1DataSets.569A.2Proofs.572BMathematicalPrerequisites575B.1Probability.575B.2LinearAlgebra.580B.3FunctionalAnalysis.586References591Index617NotationandSymbols625SeriesForewordThegoalofbuildingsystemsthatcanadapttotheirenvironmentsandlearnfromtheirexperiencehasattractedresearchersfrommanyfields,includingcomputerscience,engineering,mathematics,physics,neuroscience,andcognitivescience.Outofthisresearchhascomeawidevarietyoflearningtechniquesthathavethepotentialtotransformmanyscientificandindustrialfields.Recently,severalresearchcommunitieshaveconvergedonacommonsetofissuessurroundingsupervised,unsupervised,andreinforcementlearningproblems.TheMITPressseriesonAdaptiveComputationandMachineLearningseekstounifythemanydiversestrandsofmachinelearningresearchandtofosterhighqualityresearchandinnovativeapplications.LearningwithKernels:

SupportVectorMachines,Regularization,Optimization,andBeyondisanexcellentillustrationofthisconvergenceofideasfrommanyfields.Thedevelopmentofkernel-basedlearningmethodshasresultedfromacombi-nationofmachinelearningtheory,optimizationalgorithmsfromoperationsre-search,andkerneltechniquesfrommathematicalanalysis.Thesethreeideashavespreadfarbeyondtheoriginalsupport-vectormachinealgorithm:

Virtuallyev-erylearningalgorithmhasbeenredesignedtoexploitthepowerofkernelmeth-ods.BernhardScholkopfandAlexanderSmolahavewrittenacomprehensive,yetaccessible,accountofthesedevelopments.Thisvolumeincludesallofthemath-ematicalandalgorithmicbackgroundneedednotonlytoobtainabasicunder-standingofthematerialbuttomasterit.Studentsandresearcherswhostudythisbookwillbeabletoapplykernelmethodsincreativewaystosolveawiderangeofproblemsinscienceandengineering.ThomasDietterichPrefaceOneofthemostfortunatesituationsascientistcanencounteristoenterafieldinitsinfancy.Thereisalargechoiceoftopicstoworkon,andmanyoftheissuesareconceptualratherthanmerelytechnical.Overthelastsevenyears,wehavehadtheprivilegetobeinthispositionwithregardtothefieldofSupportVectorMachines（SVMs）.Webeganworkingonourrespectivedoctoraldissertationsin1994and1996.Uponcompletion,wedecidedtocombineoureffortsandwriteabookaboutSVMs.Sincethen,thefieldhasdevelopedimpressively,andhastoanextentbeentransformed.Wesetupawebsitethatquicklybecamethecentralrepositoryforthenewcommunity,andanumberofworkshopswereorganizedbyvariousresearchers.Thescopeofthefieldhasnowwidenedsignificantly,bothintermsofnewalgorithms,suchaskernelmethodsdifferenttoSVMs,andintermsofadeepertheoreticalunderstandingbeinggained.Ithasbecomeclearthatkernelmethodsprovideaframeworkfortacklingsomeratherprofoundissuesinmachinelearningtheory.Atthesametime,successfulapplicationshavedemonstratedthatSVMsnotonlyhaveamoresolidfoundationthanartificialneuralnetworks,butareabletoserveasareplacementforneuralnetworksthatperformaswellorbetter,inawidevarietyoffields.StandardneuralnetworkandpatternrecognitiontextbookshavenowstartedincludingchaptersonSVMsandkernelPCA（forinstance,235,153）.Whilethesedevelopmentstookplace,weweretryingtostrikeabalancebe-tweenpursuingexcitingnewresearch,andmakingprogresswiththeslowlygrow-ingmanuscriptofthisbook.Inthetwoandahalfyearsthatweworkedon