CS
I
Don't use plagiarized sources. Get Your Custom Essay on
Comparison project report
Just from $13/Page
5
1
1
2
-DeepLearningFrameworks
Executive Summary
At theartificial intelligence lab,our lackofa standardizedcomputingenvironment fordeep learningexperiments is
reducingourproductivityaswe increase thenumberof studentsand industrypartnerships. Toresolve this,wewill
selectanAI framework fromeitherTensorfloworPyTorch tobeusedasastandard forall labmembers. This report
detailsour selectioncriteria,ourevaluation,andultimately the framework thatwillbeadopted
.
Theevaluation itself is structuredaccording to threemajorareasofcapabilityandassociated features: Productivity,
Technology,andPerformance. Weevaluateeach framework independently, thenperformamoredetailedanaly-
sis/comparisonbefore tabulating thefinal scoresaccording toamethodologybasedontheworkofBandoretal. [1].
Wealso ranaperformancebenchmarkonhardware typicalofaworkstation in the lab.
Afterperformingtheevaluation,we foundthatboth frameworksperformedwell,but foundthatTensorflowwas
responsive tosomeofourneeds, specifically related tovisualizationanddeploymentcapabilities. Overallwe found
thatTensorflowmetallofour requirementsandachieved thehigher scoreonourevaluation., thoughPyTorchwas
shown to have superior performance on the single GPU benchmark. Based on the evaluation we have selected
Tensorflowtobecomethestandard framework for the lab. Howeverduring theprocesswerealized that thestrict
standardwesought to imposewasnot thebestapproach, sincesomeflexibility in frameworkusewillbenefit the
labandthuswewill encourage theuseofPyTorchandother frameworkswhereappropriate.
I
Contents CSI5
11
2-DeepLearningFrameworks
Contents
1 Introduction 1
1.1 What isanAI framework? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.1 Tensorflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.1.2 PyTorch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 TestingEnvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 Evaluation Criteria
3
2.1 Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.1 QualityofDocumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.2 CommunitySupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.3 Easeof installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1.
4
DebuggingTools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.1.5 Availabilityofpretrainedmodels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 TechnologySupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.1 Algorithmcreation language . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.2 GPUsupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
2.2.3 Computingcluster support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.3 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2.4.1 Benchmarkperformance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Methodology
6
4 Software Evaluation
8
4.1 TensorflowEvaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.1.1 Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
4.1.2 TechnologySupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
4.2 PyTorchEvaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
14
4.2.1 Productivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.2.2 TechnologySupport . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
20
4.3 Performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
22
4.3.1 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.3.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
23
4.4 AnalysisandComparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
4.4.1 Dynamicvsstaticgraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.4.2
V
isualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
26
4.4.3 Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.5 RequirementsSummary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
7
II
CSI51
12
-DeepLearningFrameworks ListofTables
4.6 Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
9
5 Conclusions and Recommendations
30
List of Figures
1 SummaryofStackOverflowquestions taggedwithTensorFlow[5] . . . . . . . . . . . . . . . . . . . . . 9
2 TensorFlowprogrammingenvironment [11] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3 SummaryofStackOverflowquestions taggedwith”pytorch” [
18
] . . . . . . . . . . . . . . . . . . . . .
16
4 Installationconfiguration tool thatprovides theappropriate installationcommandtorun[20] . . . .
17
5 PyTorchGPUvs. CPUperformancecomparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
6 Benchmarkresultsat16-bitprecision,usingasingleGPU.Timesprovided inmilliseconds . . . . . .
24
7 Benchmarkresultsat
32
-bitprecision,usingasingleGPU.Timesprovided inmilliseconds . . . . . . 24
8 RepresentativeexampleofTensorboard, showing3Dhistograms for training [8] . . . . . . . . . . . . . 26
List of Tables
1 Frameworkversionsused inourevaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
2 DecisionSupportSpreadsheet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
3 EvaluationScoringDescription . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 EvaluationConsiderations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
5 Requirements statusunderqualityofdocumentation forTensorFlow . . . . . . . . . . . . . . . . . . . 9
6 Summaryof requirements statusundercommunity support forTensorFlow . . . . . . . . . . . . . . . 9
7 SummaryofTensorFlowinstallationdifficulty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
10
8 Summaryof requirements statusundereaseof installation forTensorFlow . . . . . . . . . . . . . . . . 10
9 Summaryof requirements statusunderdebugging tools forTensorFlow . . . . . . . . . . . . . . . . . 11
10 Summaryof requirements statusunder theavailabilityofpre-trainedmodels forTensorFlow . . . . . 11
11 PretrainedmodelsprovidedbyTensorFlow. Eachmodel indicates the top1and5accuracy.[9] . . . . 12
12 Current statusofTensorFlowfunctionality inother languages [10] . . . . . . . . . . . . . . . . . . . . . 12
13
Summaryof requirements statusofAlgorithmcompatibility inTensorFlow . . . . . . . . . . . . . . . 13
14 Summaryof requirements statusofGPUsupport inTensorFlow . . . . . . . . . . . . . . . . . . . . . . 13
15
Summaryof requirements statusofcomputingcluster support inTensorFlow . . . . . . . . . . . . . . 14
16 Summaryof requirements status fordeployment inTensorFlow . . . . . . . . . . . . . . . . . . . . . . 14
17 Summaryof requirements statusunderqualityofdocumentation forPyTorch . . . . . . . . . . . . . . 15
18 Summaryof requirements statusundercommunity support forPyTorch . . . . . . . . . . . . . . . . . 16
19
Quantitative frameworkadoptionmetrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
20 PyTorch installation times . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
21 Summaryof requirements statusundereaseof installation forPyTorch . . . . . . . . . . . . . . . . . . 18
22 Summaryof requirements statusunderdebugging tools forPyTorch . . . . . . . . . . . . . . . . . . . 19
III
ListofTables CSI5112-DeepLearningFrameworks
23 Summaryof requirements statusunder theavailabilityofpretrainedmodels forPyTorch . . . . . . . 20
24 Pretrainedmodels (andarchitectures)providedbyPyTorch . . . . . . . . . . . . . . . . . . . . . . . . . 20
25 Summaryof requirements status formodelcreation language inPyTorch . . . . . . . . . . . . . . . . . 21
26 Summaryof requirements statusofGPUsupport inPyTorch . . . . . . . . . . . . . . . . . . . . . . . . 21
27
Summaryof requirements statusofmulti-GPUsupport inPyTorch . . . . . . . . . . . . . . . . . . . . 22
28
Summaryof requirements status fordeployment inPyTorch . . . . . . . . . . . . . . . . . . . . . . . . 22
29
Benchmarkresultsat32-bitprecision,usingasingleGPU.Timesprovided inmilliseconds . . . . . . 23
30 Benchmarkresultsat16-bitprecision,usingasingleGPU.Timesprovided inmilliseconds . . . . . . 23
31
Summaryofperformancerequirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
32 Summaryof requirements statusby framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
33
Summaryof scores forourdecisioncharacteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
IV
CSI5112-DeepLearningFrameworks ListofTables
Glossary
Term/Acronym Definition
ASIC Application-Specific IntegratedCircuit. An ICcustomized foraparticularuse.
Anaconda Anopensourcepackageandenvironmentmanagerwritten inpython. Alsoknownas
Conda.
API ApplicationProgrammingInterface
ArXiv A site established by Cornell University to collect electronic preprints of scientific
papers
AWS AmazonWebServices
cFFI CForeignFunction Interface forPython: Facilitatesaccess toalmostanyCcode from
python.
CNN Convolutional Neural Network: A deep network where some layer implement the
convolutionoperation
CUDA Aparallel computingsoftware layer thatgivesdevelopersaccess toNVIDIAGPUvirtual
instructions
cuDNN AGPU-accelerated libraryofprimitives fordeepneuralnetworks.
Estimators Ahigh-levelTensorFlowAPI thatgreatly simplifiesmachine learningprogramming.
Github Acloud-based implementationof thegitprotocol formanagingsoftwarecode,access
andversions (github.com).
GPU GraphicsProcessingUnit. Thecomputerhardware thatprocessesmathematical com-
putations inahighlyparallelway.
Gradient When training a neural network with gradient descent, the gradient is the direction
that theweightsof theneuronsmoveafterapass throughthenetworkduring training.
GraphViz Anopensourcegraphvisualization tool
ICLR InternationalConferenceonLearningRepresentations
IDE IntegrateDevelopmentEnvironment: Thesoftwareusedtocreateandmanagecode
that integrates features suchassourcecontrolanddebugging.
ILSVRC-2012 ImageNetLargeScaleVisualRecognitionChallenge2012
NVIDIA A technology company providing GPU hardware for computing gaming and other
computationally intensive industries
ONNX OpenNeuralNetworkExchange
PIP Apackagemanager forprogramswritten inpythonforusewithpython
QRNN Quasi-RecurrentNeuralNetwork
StackOverflow Acommunitybasedresourceofquestionsandanswers forprogrammers (stackover-
flow.com)
Tensor Ageneralizationofvectorsandmatrices topotentiallyhigherdimensions.
TPU Tensor Processing Unit. A custom ASIC developed by Google specifically for neural
networkmachine learning.
V
CSI5112-DeepLearningFrameworks
1 Introduction
1 Introduction
TheArtificial IntelligenceLaboratory (AIL)within theDepartmentofComputerScienceat the ImaginaryUniversity,
focusesonmachine learningandcomputervisionresearch. Our researchseeks todevelopnewalgorithms thatwill
exceedhumanperformance inavarietyofvisionbased tasks. Inworking towards thisgoalwe intendtostandardize
our technologyplatformbyadoptingasinglemachine learningsoftware framework.
As the lab increasesspaces forstudentsandour industrial fundingcontinues togrow,wemuststreamlineour intake
process fornewstudentswhilealso increasingourproductivityandresearchoutput. Wehave identifiedour lackof
standards regarding software tools, and the lackof acentralizedcomputingenvironment thatmakesuseofour
existinghardwareassets tobeareasof improvement fromwhichwecanderive thegreatestvalue.
To date the faculty and student members of the lab have taken an ad hoc approach to the selection of machine
learning software. At this time, most individuals are using different tools or different versions of the same tools
resulting ina fragmentedcodebaseacross researchprojectsand little tono interoperabilitybetween individualsand
cohorts. Asa result, significantenergymustbeexpendedto reusecode,which increasesourpublicationcycle. This
problemcanextendtomultipleprojectswhenteammembers’ expertise isneededonaprojectusinga framework
that theyarenot familiarwith, especiallywhenconferencedeadlinesareapproaching.
Thereforeweundertake tofinda framework thatmeets the requirementsof the lab,andwill offervalue tostudents
as theyprogress tocareers in industryorcontinuewithacademicwork.
1.1 What is an AI framework?
AnAI framework is a setofprogramming libraries thatprovide themathematical tools needed to structure and
train neural networks, including many of the established algorithms in this space. In some cases some would
describesuch frameworksmoreaccuratelyaspythonpackages (especially in thecaseofPyTorch), since their core
functionalityprovides tensorcomputationaswithotherpackages,butwith focusonGPUacceleration.
Frameworksprovide thecodeusedto implement thealgorithmsandthe fundamentaloperationsonwhich they
rely. Theyarehighlyoptimized forperformanceandthoroughly tested. Modern frameworksalso includesyntax for
structuringnetworksusing their respectiveproprietary formats, aswell as tools fordebuggingandvisualization.
Due to thecomplexityof thesoftware included insuch frameworks thereare few instances forwhichonecould
justify starting such code from scratch. However there are some caveats to framework usage, as each results in
output that is largelyproprietary, including theskills required toperformadvanced tasksandcuttingedgeresearch.
Therefore theselectionof sucha frameworkrequirescareful consideration.
1
1 Introduction CSI5112-DeepLearningFrameworks
Throughourexposure to the literature inartificial intelligence,andthroughaperiod introspectionwithinourgroup,
wehavenarrowedthefieldofpossible frameworksundercontention toTensorflowandPyTorch.
1.1.1 Tensorflow
Tensorflowisanopensource tool forgraphbasedalgorithmdevelopment includingmachine learning tasks. Itwas
designedbyGoogle to runonindividualworkstationsaswell as in largescaleenvironments. Itwasfirst released
inNovember2015underApache2.0 license. Since it is ageneral system, it canbeapplicable for thepurposeof
machine learninganddeepneuralnetworks research [2].
TensorFlowrunsonavarietyofplatforms likeGPUs,CPUs,TPUs,mobile, embedded. By introducingTensorFlow,
Google facilitates theconstruction, traininganddeploymentofdeepneuralnetworks.
1.1.2 PyTorch
PyTorch is a relatively new framework experiencing rapid growth in popularity that describes itself as a python
package thatprovidesGPUaccelerated tensorcomputationanddeepneuralnetworks thataredefinedbasedonthe
dynamicconstructionof thecomputationalgraph[3]. Thesoftware is releasedasabetaversionat the timeof this
writing.
PyTorchputsa lotofemphasisonconformingtothepythonapproachtostructuringcode,andoverallprogramming
style. Sincemanyresearchersanddevelopersarealready familiarwith thepython language, the framework feels
naturaland familiarcompared toothers thatbring theirownparadigm.
1.2 Testing Environment
For thepurposesofourevaluationwehaveusedthe following frameworkversions found inTable1.
Performance
benchmarkswereperformedonaconsumergradePCrunningUbuntuLinux16.04containingasingleNVIDIA
GTX1080TiGPU.Totest installation timesweused threepersonal computers (laptops) runningWindows10,OSX
10.12.6, andUbuntuLinux16.04.
Table 1: Frameworkversionsused inourevaluation.
Framework Version
PyTorch 0.3.1ß
Tensorflow 1.7.0
2
CSI5112-DeepLearningFrameworks 2 EvaluationCriteria
2 Evaluation Criteria
Wehaveselected the followingevaluationcriteria tobestmeasurehowwell thecandidate toolswillmeetourneeds.
We includeasetof requirements foreachcharacteristic toguide theevaluation.
2.1
Productivity
2.1.1
Quality of Documentation
Wewould like tounderstandthecompletenessof thedocumentation includedwith theproduct toensurewecan
quickly learnabout theproductandhaveclear information intheeventofaproblemwithouthavingtoreferdirectly
tosourcecode (if available).
Associated Requirements
• The frameworkmustprovidedocumentation thatdescribesallpublic frameworkAPI functions.
• Frameworkdocumentationmustbeprovidedonline.
• The frameworkdocumentationmustbeupdatedwitheachpoint releaseof thesoftware.
2.1.2
Community Support
Theonlinecommunityhasbeenvery important forproblemsolving,debugging,andgeneral supportwhenusing
suchtools. Wewould like thechosenmachine learning framework tohavea largeactivecommunity.
Associated Requirements
• The framework must have an online community forum with greater than 1000 Stack Overflow
questions.
• The frameworkmusthaveat least30mentionsonarXiv.orgpermonth
• Theremustbeat least1000questions taggedwith the framework’snameonstackoverflow.com
• Of the tagged questions available on stackoverflow.com tagged at least 50% must be marked as
answered.
2.1.3
Ease of installation
Since thesoftwarewillbe installedmanytimesonavarietyof systems,wewould like the framework tobeeasy to
installwithoutcomplexdependence.
Associated Requirements
3
2 EvaluationCriteria CSI5112-DeepLearningFrameworks
• Theframework, ifwrittenusingacompiledlanguage,mustbeinstallableviasource
codecompilation.
• The frameworkshouldbe installablevia theanacondapackagemanager.
2.1.4
Debugging Tools
We would like to be able to quickly understand problems without model definitions, and training procedures
withoutwritingadditional toolsorconductingcondereviews.
Associated Requirements
• The frameworkshouldprovideamethodtovisualizemodelsdefinedusing the framework
• The frameworkshouldprovideamethod for thevisualizationof the trainingprocess
• The frameworkmustprovideamechanismfordebugging thecode thatdefines thenetwork.
2.1.5
Availability of pretrained models
Alotofour learning,work,andexperimentationbuildsontheworkofothers.
Associated Requirements
• The frameworkshouldprovideaccess toasetofpretrainedmodelsandtheir respectivearchitectures.
2.2
Technology Support
2.2.1
Algorithm creation language
Themembersof the labandnewarrivals tendtohave thegreatestexpertiseusingPythonfor scientificandmachine
learningcalculations. Tominimize the learningcurve formembers it isdesirable that the frameworksupport the
Python language through itsAPI.
Associated Requirements
• The frameworkAPImust support thePython language
2.2.2
GPU support
The labusesGPUs forall of its researchcomputations. WehaveanumberofGPUsavailableandweareplanning to
expandthosenumbers. It isessential thatandframeworkselectedsupport trainingand inferenceviasingleand
multipleGPUconfigurations for individualworkstations.
Associated Requirements
4
CSI5112-DeepLearningFrameworks 2 EvaluationCriteria
• The frameworkmustallowtrainingand inferenceviasingleGPUonasinglenode
• The frameworkmustallowtrainingand inferenceviamultipleGPUsonasinglenode
2.2.3
Computing cluster support
TheAIL lab isexploring thepossibilityof creatingadistributedcomputingenvironment tomake thebestuseofour
existingandfuturecompute resources.
Associated Requirements
• The frameworkshouldsupportdistributed trainingacrossmultiple servers.
• The frameworkshouldsupportdistributed trainingand inferenceacrossmultiple servers.
2.3
Deployment
Sincethe lab isseekingtoexpandour industrialpartnerships. Indoingso,weexpect toworkonprojects that require
some level of preliminary productization and/or prototyping. Therefore the chosen framework must support
deploymentofmodels tosupport inferencerequests fromvarioussources.
Associated Requirements
• The framework must support the deployment of models in order to service external requests for
predictions
2.4 Performance
2.4.1
Benchmark performance
Toproperlyevaluatesucha framework,wemustunderstandtherelevantperformanceattributes, sinceapoorly
performing frameworkcouldhaveasignificant timeandenergycost. Whilewedonothaveabaseline forahigh
performingsystemwewill attempt toestablish thisduring theevaluation. Wewillmeasure theperformanceof the
selected frameworksagainst thisbaseline.
Associated Requirements
• The frameworkmustperformbetter thanthebaselinemeasure.
5
3 Methodology CSI5112-DeepLearningFrameworks
3 Methodology
Measurement
For thesoftwareevaluationwewill conductananalysisof theselected frameworks toproduceadetailedcompari-
sonofhoweachaddresseseachrespectivecriteria. Thisanalysiswill includeasubjectiveassessmentofhowwell
the framework performs in the form of a score for each characteristic. In addition we have established a set of
considerations (seeTable4) tobeused toguide theanalysisand toquantifya framework’sability toaddress the
characteristic’s includingmetrics forevaluationperformanceonastandardized task.
Toestablish theperformancebenchmarks,wewill selectat leastonestandardconvolutionalnetworkarchitecture
anddataset tobeused to trainamodel. Under thisapproachwewill take the followingmeasurements:
• Timerequired to train themodel forafixednumberof iterations.
• Timerequired toperforminferenceusing the trainedmodelonasetofnovel images.
Finallywehave identifiedasetof requirements that the future frameworkmustaddress inorder tobeselectedas
outlinedabove. Duringouranalysiswewilldetermine if eachrequirement ismet. If a requirement isnotmet then
wewill attempt toestimate theeffort required toaugment theproduct tobring it intocompliance.
Evaluation
Following theanalysis,wewilluse theresultantdata toevaluate the frameworksusingadecisionanalysis spread-
sheet. The frameworkwith thehighest scorewillbeadoptedby the lab. Inourapproachwehaveasetofcriteria,
eachwitha listof characteristics forevaluation [1].
Whenscoring the frameworkwewill assigneachcharacteristic foragivencriterionaweight suchthat theweights
sumto100%. Wewill alsoassignaweighting toeachcriterionsuchthat theweightsofall thecriteriawill alsosumto
100%. Only thecharacteristics themselveswill receivea rawscore (seeTable2). Thescoringmethod isdescribed in
Table3. Wewillprovideanarrative tosubstantiate thescoresandananalysisdirectlycomparingthemost important
differencesdiscoveredduring theevaluation.
6
CSI5112-DeepLearningFrameworks 3 Methodology
Table 2: DecisionSupportSpreadsheet
Framework Alternatives
Tensorflow Caffe
Item Decision Characteristics Weight Raw Weighted Raw Weighted
A Productivity 40
A.1 Qualityofdocumentation 5
A.2 Community support 40
A.3 Easeof installation 10
A.4 Debugging tools 15
A.5 Availabilityofpretrainedmodels 30
B Technology 40
B.1 Modelcreation language(s) 20
B.2 Framework implementation language 5
B.3 Algorithm/Modalitycompatibility 30
B.4 Multi-GPUsupport 30
B.5 Computingcluster support 15
C Performance 20
C.1 Training time 60
C.2 Inference time 40
Table 3: EvaluationScoringDescription. Scorescanbeanyvaluewithin therangesprovidedbasedonthe
perceivedproportionateabilityofa framework toaddressagivencharacteristic.
Score Range Description
1 Framework fully satisfies thecharacteristic at
anexcellent level
0.75 Frameworksatisfies thecharacteristicatasat-
isfactory level
0.5 Framework requires minimal effort to satisfy
thecharacteristic
0.25 Framework requires significanteffort to satisfy
thecharacteristic
0 Frameworkdoesnot satisfy thecharacteristic
ORUnable toassess
7
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Table 4: EvaluationConsiderations
Criteria Characteristics Sample of Considerations
Productivity Qualityofdocumentation Formatofdocumentation,completeness, examples
Community support Github forks,Papercitations,StackOverflowquestions
Easeof installation Install time,dependency list, systemrequirements
Debugging tools Availability, visualizations, subjective reports
Availabilityofpretrainedmodels Numberofpertrainedmodelsavailable,Models formajorpapers
TechnologySupport Modelcreation language(s) List languages
Framework implementation language Identify language, sourcecodeavailable
Algorithm/Modalitycompatibility Listofnetwork typessupported,extensibility
Multi-GPUsupport TypeandnumberofGPUssupported,efficiencyof scaling
Computingcluster support Complexityof implementation, systemrequirements,user support
Deployment Deploymentoptions, relateddocumentation,qualityof integration
Performance Benchmarks Trainingand inference times fora representativemodel/dataset
4 Software Evaluation
In this sectionwewillprovides the resultsofouranalysisofeach frameworkaccording to themethoddescribed in
section3: Methodology. Wewillprovideananalysis foreach framework thatwill justify theassignedscore.
4.1 Tensorflow Evaluation
4.1.1 Productivity
Quality of Documentation
TensorFlowhas twounitsofAPIdocumentation [4]. Theseunitsareas following:
• HighLevelAPIs
• LowlevelAPIs
High levelAPI introducesestimators that simplifymachine learningprogramming. Estimators encapsulate the
actions like training, evaluation,prediction,export for serving. It alsocontainsdocumentationabouthowtosetup
datapipelines to readdataset intoyourprogram. LowlevelAPIsexplains thebasicsofhowyoucanuseTensorFow
andhowtocreate,manipulateandaccessTensorsandetc.
The APIs in Python is fully documented. This guideline introduces a detailed documentation about all models,
layers, built-in operations, modules, classes, functions and etc. It also provides a brief example for each part.
However,C++, JavaandGOdocumentation isbriefwithsomeexample inmostcases. Thereareanumberofgood
tutorials thatcoversdifferentaspectof imagerecognition,machine learningproblemsdealingwithsequencedata,
various data representation that can be used and other kind of math problems. They are good for learning the
conceptbutnotenoughfor real-worldmodeling tasks. Youcanhaveaccess toallof thesedocumentationviaweb
browser. Thedocumentationat root refers to themost recentversionbut thepreviousversionsdocumentation is
alsoavailableviaTensorFlowversions.
8
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
Figure 1: SummaryofStackOverflowquestions taggedwithTensorFlow[5]
Inconclusion,AIL requirements forqualityofdocumentationare fulfilled (Table5).
Table 5: Requirements statusunderqualityofdocumentation forTensorFlow
ID Requirement Status
1 The frameworkmustprovidedocumentation thatdescribesallpublic frameworkAPI functions.
Met
2 Frameworkdocumentationmustbeprovidedonline. Met
3 The frameworkdocumentationmustbeupdatedwitheachpoint releaseof thesoftware. Met
Community Support
Wecouldsay thateveryone in themachine learningcommunity isawareofTensorFlow,and ithas thecommunity
engagementasanimportant factor to theextent that ithasmentionedin last sevendays (April1,2018–April7,2018)
35 times inarXiv.org. Therearemultiplecommunicationpaths forTensorFlow. TensorFlowdiscussmailing list isa
way to joingeneraldiscussionabout itsdevelopmentanddirections likenewfeatures inupcomingreleases. This
discussmailing list containsabout2700 topics. There isalso theopportunity touseStackOverflowtoaskoranswer
technicalquestions (thecurrentvalue is25848questions taggedwithTensorFlow. SeeFigure1). TensorFlowissues
trackeronGithubcanalsobeused to reportbugsor to requestanewfeature. Soall theassociatedrequirements for
community supportaremet (seeTable6).
Table 6: Summaryof requirements statusundercommunity support forTensorFlow
ID Requirement Status
4 The frameworkauthormustprovideanonlinediscussion forum Met
5 The frameworkmusthaveat least30mentionsonarXiv.orgpermonth Met
6 Thereshouldbeat least1000questions taggedwithTensorFlowonstackoverflow.com Met
7 Of thequestionsavailableonstackoverflow.comtaggedwithTensorFlow,at least50%
mustbemarkedasanswered.
Met
9
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Ease of Installation
There isa fullydetailed instructionguideonTensorFlowdownloadpage [6]. Foreachplatform, theyfirstgivesome
information about supported mechanism by which you can install TensorFlow, it’s prerequisite, a step by step
instruction,howtovalidateyour installationandcommonproblems.
Table 7: SummaryofTensorFlowinstallationdifficulty
Platform Package Manager CUDA Difficulty Time (min.)
Ubuntu Anaconda Yes Easy 16
PIP Yes Easy 10
macOS Anaconda No Easy 19
PIP No Easy 10
Windows PIP Yes Moderate 15
Anaconda Yes Moderate 20
Based on our experiments, using Anaconda makes things easy because it isolates your environments from the
defaultsprovidedonyoursystem. TensorFlowalsosupports installation fromsourcesonlyonUbuntuandmacOS
X.Asa result, the installation iseasyandstraightforwardandmeetsourassociatedrequirements (seeTable8).
Table 8: Summaryof requirements statusundereaseof installation forTensorFlow
ID Requirement Status
8 The framework, ifwrittenusingacompiled language,mustbe installablevia source
codecompilation.
Met
9 The frameworkshouldbe installablevia theanacondapackagemanager. Met
Debugging Tools
TensorFlow provides a specific debugger called TensorFlow debugger (tfdbg) [7]. It gives us a chance to see the
internalstructureandcomputationalgraph,whichisdifficult todebugwithgeneral-purposedebuggers. TensorFlow
also have includedavisualization tools calledTensorBoard [8], it letsusexplore what cango wrongandhow to
haveabetterunderstandingof themodel. TensorBoradalsoprovides theoptionofvisuallymonitor theprogressof
trainingso thatwecancomparedifferentneuralnetworkdesigns tofindthebest result forour trainingdata.
Insummary,puttingaside the fact thatusingaspecializeddebuggercouldbe time-taking,TensorFlowpasses the
associatedrequirements fordebuggingandvisualization tools (seeTable9).
10
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
Table 9: Summaryof requirements statusunderdebugging tools forTensorFlow
ID Requirement Status
10 The frameworkshouldprovideamethodtovisualizemodelsdefinedusing the framework Met
11 The frameworkshouldprovideamethod for thevisualizationof the trainingprocess Met
12 The frameworkmustprovideamechanismfordebugging thecode thatdefines thenetwork. Met
Figure 2: TensorFlowprogrammingenvironment [11]
Availability of Pretrained models
Since trainingmodels fromscratchon largedatasetscanhavean intensive timecomplexity,TensorFlowprovidesa
variouspre-trainedmodelson the ILSVRC-2012-CLS (ImagenetLargeScaleVisualRecognitionChallenge2012)
imageclassificationdataset (Table11)andtherefore theavailabilityofpre-trainedmodels isalsomet (seeTable10).
Table 10: Summaryof requirements statusunder theavailabilityofpre-trainedmodels forTensorFlow
ID Requirement Status
13 The frameworkshouldprovideaccess toasetofpre-trainedmodelsandtheir respec-
tivearchi
tectures
Met
4.1.2 Technology Support
Model Creation Languages
TensorFlow has a programming stack that contains multiples API layers (see Figure 2 ). Although Python is the
firstwell-supportedclient languageprogrammingsupportedbyTensorFlow, It is alsopossible tocreateordevelop
TensorFlowfunctionality inother languages likeC++, Java,Go. Otherclient languagesshoulduse the languages
foreign function interface (FFI) tobeable touseTensorFlowfunctionality.
Moreover,TensorFlowprovidesaCAPIwhichhelp tosupport formoreclient languages. So,newlanguagesupport
has tobebuilton topofCAPI.Youcansee theavailable functionality inC inTable12 [10].
11
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Table 11: PretrainedmodelsprovidedbyTensorFlow. Eachmodel indicates the top1and5accuracy.[9]
Model Top-1 Accuracy Top-5 Accuracy
InceptionV1 69.8 89.6
InceptionV2 73.9 91.8
InceptionV3 78.0 93.9
InceptionV4 80.2 95.2
Inception-ResNet-v2 80.4 95.3
ResNetV150 75.2 92.2
ResNetV1101 76.4 92.9
ResNetV1152 76.8 93.2
ResNetV250 75.6 92.8
ResNetV2101 77.0 93.7
ResNetV2152 77.8 94.1
ResNetV2200 79.9 95.2
VGG16 71.5 89.8
VGG19 71.1 89.8
MobileNet-v1-1.0-224 70.9 89.9
MobileNet-v1-0.50-160 59.1 81.9
MobileNet-v1-0.25-128 41.5 66.3
MobileNet-v2-1.4-224 74.9 92.5
MobileNet-v2-1.0-224 71.9 91.0
NASNet-A-Mobile-224 74.0 91.6
NASNet-A-Large-331 82.7 96.2
PNASNet-5-Large-331 82.9 96.2
Table 12: Current statusofTensorFlowfunctionality inother languages [10]
Feature Python C
RunapredefinedGraph tf.import-graph-def, TF-GraphImportGraphDef
tf.Session TF-NewSession
Graphconstructionwith Yes Yes (TheCAPIsupportsclient
generatedopfunctions languages thatdo this)
Gradients tf.gradients
Functions tf.python.framework.function.Defun
ControlFlow tf.cond, tf.while-loop
NeuralNetwork library tf.train, tf.nn
tf.contrib.layers
tf.contrib.slim
12
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
Algorithm / Modality Compatibility
As our Lab is focused on computer vision related task, it is expected that the framework supports convolution
operation. TensorFlow layersmoduleprovidesanAPI thatmakes it easy toconstructaneuralnetwork. This library
provides a set of neural networks layers and it contains functions for 1D , 2D , 3D convolution layers. So this
Requirement isalsomet (seeTable13).
Table 13: Summaryof requirements statusofAlgorithmcompatibility inTensorFlow
ID Requirement Status
14 The frameworkmust support theconvolutionoperation Met
Multi-GPU support
SinceAI frameworkscanbeuse for intensivecomputations like traininganeuralnetwork, it is really important to
supportGPUtobeable todoall theoperationsat thesametime. TensorFlowsupportsGPUanditsprogramsrun
significantly fasteronaGPUrather thanonaCPU.
TensorFlowdoesnotprovideGPUsupportonmacOSasofversion1.2.The followingrequirements isneeded to run
TensorFlowwithGPUsupport:
• CUDA®Toolkit 9.0.
• TheNVIDIAdriversassociatedwithCUDAToolkit 9.0.
• cuDNNv7.0.
• GPU card with CUDA Compute Capability 3.0 or higher for building from source and 3.5 or higher for
TensorFlow’sbinaries [12].
TensorFlowsupports theGPUofyourmachine, if it isoneormore thanone. It ispossible touseasingleGPUon
a multi-GPU system or using multiple GPUs. In summary, TensorFlow supports GPU computation in different
platformsexcept formacOS.
Table 14: Summaryof requirements statusofGPUsupport inTensorFlow
ID Requirement Status
15 The frameworkmustallowtrainingand inferenceviasingleGPUonasinglenode SomewhatMet
16 The frameworkshouldsupportdistributedGPUtrainingacrossmultiple servers. Met
17 The frameworkmust supportmultipleGPUtrainingonasingle server. Met
13
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Computing cluster support
TensorFlowalsosupportsdistributedcomputation, youcancreateaclusterofTensorFlowserversanddistributea
computationgraphacross thatcluster. ATensorFlowclustercontainsasetof taskparticipating in thedistributed
executionofagraph. Each task is related toaTensorFlow”server”which isable tocreateasessionandeachsession
canbeused toperformadistributedcomputation. youcanuse t f .d e v i c e function tospecify that theoperations
runoneitherCPUorGPU[13].
Table 15: Summaryof requirements statusofcomputingcluster support inTensorFlow
ID Requirement Status
17 The frameworkshouldsupportdistributed trainingacrossmultiple servers. Met
Model Deployment
Although TensorFlow models are regularly characterized and trained using Python, it is also possible to deploy
TensorFlowmodels. TensorFlowprovidesanopen-sourcesoftware librarycalled”TensorFlowServing” thatmakes is
easytodeploynewalgorithmsandexperiments. It iseasytofinddocumentationonTensorFlowServingarchitecture
anduseful tutorialson itswebsite [14]. However, youcannotfindall youneedtodo for servingyourownmodel.
Table 16: Summaryof requirements status fordeployment inTensorFlow
ID Requirement Status
18 The frameworkmust support thedeploymentofmodels inorder toserviceexternal
requests forpredictions.
Met
4.2 PyTorch Evaluation
4.2.1 Productivity
Quality of Documentation
ThePyTorchcoredocumentation [15]offersa thoroughtreatmentof thecompleteAPI.Eachpackage, its associated
data-structures,methodsandalgorithmshasacompletebreakdownofusage includingallparametersandreturn
values. Thedocumentationpagesalso includebriefexamples inmostcases.
In addition to the core Python API documentation there is also a set of tutorials [16] ranging from beginner to
advanced. These tutorials are very well constructed and while they only number sixteen (Tensorflow also has
sixteen) in total, theyarevery focused,withcoverageofall themajorentrypoints into the framework. The tutorials
arealsouseful,withouthaving toactually implement thecodemanually since therearecompleteexplanationswith
14
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
codeandscreencaptures.
All of the documentation is available online, via web browser in HTML format. The API documentation is
archived and searchable via version. All past (and presumably) future versions of the documentation will be
tied to each release. Additionally, since PyTorch is open source software the code is available online via github
(https://github.com/pytorch/pytorch). In our experience, during this evaluation, the ability to review the
code isan importantadjunct to theregularAPIdocumentation.
It is important tomentionthat it ispossible toextendPyTorchviaCAPIbasedoncFFI forPythontosatisfy some
experimental requirementor researchneed. Unfortunately theCAPI isnotyet fullydocumented,andwhile there is
a single tutorialon itsusage, it is challenging tomakeuseof, andrepresentsasourceof frustration forasegmentof
thePyTorchcommunity [17]. Thiscouldbeproblematic if researchprojects requiremanycustomextensions.
In termsof”print”basedpublications, thenumberofbooks,print,PDF,ebook,orotherwise isquite limited. We
wereonlyable tofindtwotitles. This isnot in itselfproblematic,due to theavailabilityofother resources,but it
doesspeak to therecencyofPyTorch’semergenceasanoption.
InsummarythePyTorchdocumentationmeetsourrequirements for frameworkdocumentation. PleaseseeTable17
fora
summaryof the requirements.
Table 17: Summaryof requirements statusunderqualityofdocumentation forPyTorch
ID Requirement Status
1 The frameworkmustprovidedocumentation thatdescribesallpublic frameworkAPI functions. Met
2 Frameworkdocumentationmustbeprovidedonline. Met
3 The frameworkdocumentationmustbeupdatedwitheachpoint releaseof thesoftware. Met
Community Support
Since the releaseofPyTorch in January2017, a significantcommunityhas formedaroundthe tool, contributing
to itsdevelopmentand improvement. Thecontributionshavebeenwideranging including implementationsof
popular research papers, and new packages for specialized networks such as the QRNN released by SalesForce
Research.
ThePyTorchsite includesanonlinediscussion forum(https://discuss.pytorch.org/) thatappears tobevery
active. During the timeof this review,questionsposted to the forumwereansweredwithin less thananhour. The
forumhasalmost7,000users, creatingover40 thousandtopicswith4 thousandof thoseactivewithin the last thirty
days (asofApril 6th,2018). This seemsto indicateastrong,activeandgrowingcommunitywhich isessential forour
use in the lab.
15
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Figure 3: SummaryofStackOverflowquestions taggedwith”pytorch” [18]
ComparedwithTensorflow, thePyTorchcommunity is smaller,with lesswidespreadadoption. Wepartiallyattribute
this to therelatively recentemergenceofPyTorchwhichcontinues toexperiencerapidgrowth. That said, it isnot
clearwhat the impactwouldbe inrealworlduse. Thecommunity support is important since inpastendeavorswe
have foundthat theability toseekhelp fromthecommunitycanmake thedifferencebetweensuccessand failure,
andcanpotentially resolve issues inhours insteadofdays. Thiscouldhaveahuge impactontheLab.
Duringourevaluationwehadtheneed,onmultipleoccasions toconsult thecommunity foranswers. Inall cases
wewereable tofindexistinganswersandcodeexamples tosolve theproblemsthatweencountered.
Insummary thePyTorchmeetsour requirements forcommunity support (withacaveat). PleaseseeTable18 fora
summaryof the requirements. PyTorchdoesnotmeet thespecific requirement that therebeat least1000questions
tagged with “pytorch” on http:\\stackoverflow.com (see Figure 3. The current value is 725 questions. With
115newquestionsappearing in thepast30days,wedoexpect this requirement tobemetbefore the labmoves to
implement thenewframework.
Table 18: Summaryof requirements statusundercommunity support forPyTorch
ID Requirement Status
4 The frameworkauthormustprovideanonlinediscussion forum Met
5 The frameworkmusthaveat least30mentionsonarXiv.orgpermonth Met
6 Thereshouldbeat least1000questions taggedwithpytorchonstackoverflow.com NotMet
7 Of the questions available on stackoverflow.com tagged with pytorch at least 50%
mustbemarkedasanswered.
Met
Inaddition to thosemetrics found inTable18, themetricscaptured inTable19arealsoof interest.
Ease of Installation
ThePyTorch installation isdesignedtobeeasy,butmuchdependsonthe targetmachine’sconfiguration. Onthe
PyTorchdownloadpage there isa tool thatallows theuser toselect theconfigurationof theirmachine inorder to
16
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
Table 19: Metrics toquantifyadoptionofPyTorchvs. Tensorflow. Githubmentionsarebasedonasearchresults
ongithub.com. Redditmentions fromPyTorch [19]
Metric PyTorch Tensorflow
ICLR2018conferenceusage 87 228
Githubrepositorymentions 6018 31,972
Reddit /r/machinelearningmentions 72 273
ensure that thepropercommandtorunfor installationcanbeselected (seeFigure4. Thesuccessof theprovided
command depends on a number of dependencies being installed and working properly. That said the PyTorch
installationselector removesa lotof theguessworkencounteredwhenattempting to installother frameworks.
In the lab, workstations are provided to students and other members. These workstations are standardized to
use Ubuntu Linux 16.04 LTS, using NVIDIA based GPUs currently using CUDA 9 where possible. This is a very
commonenvironmentand is supportedbymost frameworks includingPyTorch. Werecognize that studentswill
sometimeschoose towork independentlyon theirownhardwarewith theconfigurationof their choice, andwe
haveno intentionofmandating thatconfiguration. For labrelatedwork though,contributorsmustprovidework
using theselected framework. PyTorchprovidescomprehensivesupport for thecommonenvironmentsmachine
learningpractitioners inour labareusing. Wenote thatPyTorchdoesnotofficially support installationonMicrosoft
Windows. While it ispossible to installPyTorchonWindows theprocess isnotwelldocumentedandprone toerror.
In much of the lab’s work it is helpful and often required to modify the framework’s source code. As a result we
require that theselectedframeworknotonlybeopensource,butbuilds fromsourcebeofficiallysupported. PyTorch
officially supports installation fromsource.
Weexperimentedwith installationonthreedifferentplatforms (includingMicrosoftWindows). PleaseseeTable20
forasummary. Wefoundthat theuseof theAnacondapackagemanager(PM)simplifiedtheinstallationbyhandling
dependencies in thebase installation,which isourpreferredPM.PIPbased installationswerealsoeffective,when
using thedependencyscriptprovided. Finally,when installing fromsource, againonourestablishedenvironment,
proceededwithout issue.
Figure 4: Installationconfiguration tool thatprovides theappropriate installationcommandtorun[20]
.
17
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Table 20: SummaryofPyTorch installationdifficultybyoperatingsystemandpackagemanager. Timemeasures
areapproximate.
Platform Package Manager CUDA Difficulty Time (min.)
Linux Anaconda Yes Easy 15
Linux PIP Yes Moderate 25
Linux source Yes Easy 45
OSX Anaconda No Easy 10
Windows PIP No Difficult >120
Inconclusionwefoundthe installationprocesswasmucheasier thanexpected. Thevarious installationoptions
wereeasy touseandproceededwithout issue. PyTorchmeetsour requirements foreaseof installation. SeeTable21
forasummaryof the requirements.
Table 21: Summaryof requirements statusundereaseof installation forPyTorch
ID Requirement Status
8 The framework, ifwrittenusingacompiled language,mustbe installablevia source
codecompilation.
Met
9 The frameworkshouldbe installablevia theanacondapackagemanager. Met
Debugging Tools
Code debugging: UnlikeTensorflowthecreationof thecomputationalgraph isdynamic. Asa result each lineof
code isexecuted in lineasopposedtoasynchronously. Therefore,uponerrorsandexceptions, the resultingerror
messageshighlight theerrordirectlyandprovideareadablestack trace. It ispossible touse thestandardpython
debugger toset tracesanywhere (i.e. p d b.se t_t r ac e()) in thecode to rapidly locate thesourceof theerrors.
Makinguseof thepythondebuggeraswell asdebuggersbuilt into IDEssuchasPyCharmmeans the familiarmode
ofdebuggingpythoncodecarriesover toPyTorch forall aspectsof thecode including thecomputationalgraph.
This isunlikeTensorflow,whichrequires theuseofTensorflow’s specializeddebugger (tfdbg) togain insight into the
code,whichcanbe tediousandtime-consuming.
Training visualization: PyTorchdoesnot includeanyvisualization tools,but thereare toolsavailable thatenable
visualizationofPyTorchtraining. Thegeneral solution is touseFacebook’s”Visdom”tool [21]. Visdomisable to
readPyTorch logsandprovidereal-timevisualizations (in2Donly)ofanyvariable, viabrowsermuchthesameas
Tensorflow’sTensorboard.
It isalsopossible,usinga thirdparty tool tomakeuseofTensorboard tovisualizePyTorchvariables.
Gradient visualization: Ofsome interest toour lab is theability tovisualize thegradientwhentrainingconvolu-
tionalmodels. Wenote thatwhile therearemany toolsandcodeavailable toaccomplish this,PyTorchdoesnot
18
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
providesuchvisualization toolsout-of-the-box.
Model visualization: Aswith trainingvisualization,PyTorchdoesnot includea tool formodelvisualizationout-of-
the-box. However,using third-party tools it ispossible toobtainavisualizationof thegraph. Tools suchasGraphViz
havesupportingcode,andrecently itbecamepossible touseTensorboardtovisualizePyTorchgraphs. Inour testing
theTensorboardbindingsexhibitedsomestability issues leading toprogramcrashes.
Tosummarize, inour testing itwas refreshing tohaveaccess toall of thecodewith thestandarddebugger. This
makesabigdifferencewhensolvingcodeproblems,andcouldbeasignificant timesaver forcomplexprojects. For
visualizationPyTorchshows its immaturitywhencomparedwith its longer livedcompetitors. Fortunatelywith the
helpof thirdparty tools, thevisualizationswerequireareavailable, thoughwenote thatworkingwithexternal tools
addsadditional complexity to theworkflow. The fact thatPyTorchcanpiggybackonTensorboard forvisualizations
reducessomeconcernregardingPyTorch’scapabilities in thisarea,but this supportcouldchange in the future. We
believe,basedontheavailabilityof thirdparty toolsPyTorchmeetsour requirements in thisarea. SeeTable22 fora
summaryof the requirements.
Table 22: Summaryof requirements statusunderdebugging tools forPyTorch
ID Requirement Status
10 The frameworkshouldprovideamethodtovisualizemodelsdefinedusing the framework
3rdParty
11 The frameworkshouldprovideamethod for thevisualizationof the trainingprocess 3rdParty
12 The frameworkmustprovideamechanismfordebugging thecode thatdefines thenetwork. Met
Availability of Pretrained models
PyTorch provides a set of architecture definitions for a number of popular networks. Through their model zoo,
networkweightscanbe initializedwiththoseofpreviously trainednetworks (seeTable24). Thesepretrainedmodels
provideddirectlybyPyTorchwere trainedusing the ImageNet [22]dataset. Themodelsareveryeasy touse,making
experimentationveryeasy.
SincePyTorchisarelatively recentrelease thereare lesssuchmodelsavailable fromindividualdeveloperscompared
to those for Tensorflow. However, the community, and overall adoption of PyTorch is growing quickly. In our
investigations thereareanumberof releasesavailableonGithubby individualdevelopers,which isagoodsign for
PyTorch’s future. Furthermore, themodels thatwedownloaded,wewereable to runthemsimplybycopying the
codeprovided,anddidnotexperienceerrors.
Insummary,PyTorchprovides theexpected levelofaccess topretrainedmodels. Thenumberofmodelsavailable
fromthecommunity is growing, and thesentiment in the researchcommunity is this growthwill continue. See
Table23 forasummaryof the requirements.
19
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Table 23: Summaryof requirements statusunder theavailabilityofpretrainedmodels forPyTorch
ID Requirement Status
13 The frameworkshouldprovideaccess toasetofpretrainedmodelsandtheir respec-
tivearchitectures
Met
Table 24: Pretrainedmodels (andarchitectures)providedbyPyTorch. Eachmodel indicates the top1and5error
rate for ImageNetwhere theerror is the fractionof test images for thecorrect label isnotat the topor in the top
five to themostprobable labelspredictedby themodel.
Network Top-1 error Top-5 error
AlexNet 43.45 20.91
VGG-11 30.98 11.37
VGG-13 30.07 10.75
VGG-16 28.41 9.62
VGG-19 27.62 9.12
VGG-11withbatchnormalization 29.62 10.19
VGG-13withbatchnormalization 28.45 9.63
VGG-16withbatchnormalization 26.63 8.5
VGG-19withbatchnormalization 25.76 8.15
ResNet-18 30.24 10.92
ResNet-34 26.7 8.58
ResNet-50 23.85 7.13
ResNet-101 22.63 6.44
ResNet-152 21.69 5.94
SqueezeNet1.0 41.9 19.58
SqueezeNet1.1 41.81 19.38
Densenet-121 25.35 7.83
Densenet-169 24 7
Densenet-201 22.8 6.43
Densenet-161 22.35 6.2
Inceptionv3 22.55 6.44
4.2.2 Technology Support
Model Creation Languages
PyTorch is focused on the python language for model creation. It meets our requirement for model creation
language.
GPU support
PyTorchsupportsGPUcomputationusingCUDA.PyTorchprovidesa simple function – the c ud a() function, to
move tensorsandvariables to theGPUformanipulationusingbasicoperationssuchasaddition, subtraction,and
20
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
Table 25: Summaryof requirements status formodelcreation language inPyTorch
ID Requirement Status
14 The frameworkAPImust support thePython language Met
multiplication(whichcanbecombined intomorecomplexoperationssuchasconvolutionasused inconvolutional
neural networks). It is also possible to configure an application to use CUDA for it’s entire execution using the
C on f i g.C ud a = T r ue parameter. This results insignificant improvements incompute time. GPUcomputation is
supported inmultipleconfigurations, includingsingleandmultipleGPUsonasinglenode.
Provided thatCUDAisconfiguredcorrectlyon the targetmachine,wehave foundthat it is veryeasy to invokeGPU
support. This reduces timewasted troubleshootingmakingworkmoreproductive. Whilenot tested transitioning
between CUDA versions has been a problem for us previously, it is not clear how well PyTorch deals with such
transitions. SeeTable26 forasummaryof the requirements.
Table 26: Summaryof requirements statusofGPUsupport inPyTorch
ID Requirement Status
15 The frameworkmustallowtrainingand inferenceviasingleGPUonasinglenode Met
16 The frameworkmustallowtrainingand inferenceviamultipleGPUaonasinglenode Met
Distributed
GPU Support
PyTorchalsosupportsmulti-GPUconfigurationsbothonsinglemachinesandacrossmultiplenodes inadistributed
clusterconfigurationusingmulti-gpucollectives includingNVIDIANCCL.Thisallowseachbatchduring training to
beparallelizedacrossGPUs,where thegradient iscomputed, thentheresultsaveragedaftereachpass. PyTorch
supportmultiplemodesofcommunicationbetweennodes for training,with limitedbutuseful support forGPU
operations. Overall thephysicallydistributedGPUcomputingrequiresanadvancedunderstandingof theassociated
mechanisms. Documentation is limitedat this timeandso iscommunity support. Weexpect that implementinga
largecluster tosupport the labmembercompute requirementsusing thePyTorch torch.distributedpackagewould
bepremature,at leastuntilPyTorchmovesoutof thebeta releasestage.
OverallPyTorchmeets thebasic requirement fordistributedcomputingbutwewerenotable toverifyPyTorch’s
Figure 5: Resultsofa test comparing themultiplicationof tworandomlygenerated10,000 × 10,000matricesby
CPU, thenusingPyTorch’sapproachusingCUDAontheGPU.
21
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
distributedperformancedirectly,norwereweable to refer to theexperienceofothermembersof thecommunity to
gather informationonthequalityof thePyTorchdistributed implementation. Withrespect tomulti-GPUsupport
on a single machine; we were able to test this with success, and reports from the community also indicate this
feature isworkingwell,butagain ispoorlydocumented. SeeTable27 forasummaryof the requirements.
Table 27: Summaryof requirements statusofmulti-GPUsupport inPyTorch
ID Requirement Status
17 The frameworkshouldsupportdistributedGPUtrainingacrossmultiple servers. Provisional
18 The frameworkmust supportmultipleGPUtrainingonasingle server. Met
Model Deployment
PyTorch’s focusonresearchhasminimized the importanceofdeploymentoptions in itsoffering. Asa result, there is
nomethod fordeployingnativePyTorchmodels, and there isvery littledocumentationaboutwhat theoptionsare
andhowtoexecutesome levelofdeploymenteithernativelyorusing thirdparty tools.
Since the ease of use and flexibility of PyTorch has great appeal beyond the pure research community to those
thathave therequirement todeploymodels thereare thirdpartyoptions. Theseoptionsarebasedonconverting
PyTorchmodels intoa formatcompatiblewithother frameworks thathavemorematuredeployment solutions.
Perhaps themostcommonapproach is touse theOpenNeuralNetworkExchangeFormat (ONNX)byFacebook
andMicrosoft toconvertPyTorchmodels intoCaffe2 format fordeploymentonAmazonWebServices (AWS)and
AWSLambda. Thisallowsconvertiblemodels toservepredictiononline inascalableway [23].
With therelianceonthirdparty solutions,PyTorchmeetsour requirements. However, this solution’sdependence
onapatchworkof toolsaddsadditionalcomplexityandriskcomparedwithTensorflowwhichoffersan integrated
solution (seeTable28 forasummaryof the requirement).
Table 28: Summaryof requirements status fordeployment inPyTorch
ID Requirement Status
19 The frameworkmust support thedeploymentofmodels inorder toserviceexternal
requests forpredictions.
3rdParty
4.3 Performance
4.3.1 Benchmarks
To compare the performance of the candidate frameworks, we have assembled a benchmark to evaluate their
performanceagainstourbaseline framework,Caffe [24],whichweusecurrently in the lab. Thebenchmark includes
22
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
trainingandevaluationmetrics for twocommondeepnetworksat16and32bitprecision levels.
Using thisbenchmarkwetrainedthe theVGG16[25]andResNet152 [26]networksusingasingleNVIDIA1080Ti
GPU.Themodelswere trainedusing imagevolumesof size224 × 224 × 3andwith thebatchsize set at 16. The
trainingmetric is theaverage time it takes to runapairof forwardandbackwardpasses throughthenetwork20
times after an initial 20 passes which are not counted. The evaluation metric is the average time it takes for a
single forwardpass run20 times,also followingan initial 20passes. Thebenchmarkcode isbasedoncode from
(https://github.com/u39kun/deep-learning-benchmark/blob/master/README.md).
While the labcurrentlyusesconsumergradeGPUsusing thePascalarchitecture like theGTX1080TIused for the
benchmark, we intend to move to Titan V GPUs using the Volta architecture both locally and in the cloud. The
VoltabasedGPUsexperienceasignificantperformance improvementwhenfloatingpointprecision ishalved. We
thereforealso includeresultswith16-bitprecision.
4.3.2 Results
From the results (see Table 29 and Table 30, we observe that both frameworks outperform the baseline (Caffe
[24])network. In this singleGPUscenarioPyTorchhas thebestperformance inall tests. This is consistentwith
observations in thecommunitywherePyTorch isgenerally thought tohavebetterperformance. Theperformance
differential isnot significant for this single forward/backwardpass,butextrapolated to tensof thousandsof itera-
tions theperformancedifferencewouldbecomenoticeable. Alsowhilewewerenotable to test theperformance
of the frameworks inamulti-GPUconfiguration,wehavereasontobelieve,basedondiscussionsonthePyTorch
discussion forumsthatPyTorch isnotable tousemultipleGPUsasefficientlyasTensorflow. However thePyTorch
teamisworking to improveefficiency.
Table 29: Benchmarkresultsat32-bitprecision,usingasingleGPU.Timesprovided inmilliseconds
Framework Precision VGG16 Eval VGG16 Train ResNet152 Eval ResNet152 Train
Baseline (Caffe) 32-bit 55.1 160.2 82.3 229.3
PyTorch 32-bit 41.8 139.8 58.8 217.0
Tensorflow 32-bit 45.1 145.6 66.4 260.6
Table 30: Benchmarkresultsat16-bitprecision,usingasingleGPU.Timesprovided inmilliseconds
Framework Precision VGG16 Eval VGG16 Train ResNet152 Eval ResNet152 Train
Baseline (Caffe) 16-bit 57 155.1 66.6 205.3
PyTorch 16-bit 44.1 139.1 49.0 190.5
Tensorflow 16-bit 51.7 140.6 54.2 198.6
23
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Figure 6: Benchmarkresultsat16-bitprecision,usingasingleGPU.Timesprovided inmilliseconds
Figure 7: Benchmarkresultsat32-bitprecision,usingasingleGPU.Timesprovided inmilliseconds
24
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
Overall, both frameworksmeetourperformance requirements (seeTable31 fora summaryof theperformance
requirements).
Table 31: Summaryofperformancerequirements
ID Requirement
Tensorflow PyTorch
20 The frameworkmustperformbetter thanthebaselinemeasure
Met Met
4.4 Analysis and Comparison
Ourreviewof the respective frameworksagainstour requirementshasshownthat the twoframeworks,Tensorflow
andPyTorchhaveverymuch incommon. However thereareanumberofdifferences thatwere revealed thatareat
thecoreofourdecisionmaking. Herewewillhighlightandprovideadiscussionof thosedifferenceswithsome
analysisof the impactonour requirements.
4.4.1 Dynamic vs static graph
Perhaps the most important difference between the two candidate frameworks is the mode of definition of the
computationalgraph. These frameworks represent their variablesandoperationsasadirectedgraphthat forms the
architectureof thenetwork thatwhenexecutedmanytimes trains thenetwork.
Tensorflow: InTensorflow, thegraph is staticallydefined incodebefore runtime. The frameworkcreatesasession
toexecute thegraphwithnoability toadapt thegraphbasedon inputorconsideration. Also, there is lessvisibility
into thesessionobject fordebuggingpurposeswhichcanmakedebuggingnetwork issuesmorechallenging. Of
course thismethodcontinues tobeaverypowerfulapproach, thathassomeadvantages regardingperformance
anddeploymentoptions if thebenefitsofadynamicgrapharenot required.
PyTorch: In PyTorch the graph has an imperative, dynamic definition allowing the graph to be created on the
fly. Thisflexibilitynotonlysimplifiesdealingwithvaryingnetwork inputs,but it allowopensanewdirection for
innovation. Dynamicgraphsarealsoeasier tousesince thecode is linearandexecuteswhenexpected. Thisalso
makesdebuggingeasyeitherwhenusing formaldebugging techniquesor justusingprint statements.
Wedidnot include the formatof thecomputationalgraph inour requirementsbecausewedonothaveaspecific
requirement fordynamicgraphsat this time. Wedo,howeverneedthe framework to tobesufficientlyextensible,
either by the project owners or ourselves to be able to explore the latest research. Since Tensorflow is working
towardsofferingamore robustdynamicgraph implementation,we thinkboth frameworkscover this trend towards
dynamicgraphs. ClearlyPyTorchhas theedge in thisarea,which is sufficientbasedonour impression, tobedriving
someresearchers tomove toPyTorch fromTensorflow.
Advantage: PyTorch
25
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Figure 8: RepresentativeexampleofTensorboard, showing3Dhistograms for training [8]
4.4.2 Visualization
In our previous work, we have made extensive use of visualization in all aspects of our research as an effective
method fordiscoveringhowtoadjustourprocessesaswedevelopourmodels. Herewehighlight thedifferenceswe
discoveredduringourevaluationof the respective tools.
Tensorflow: Tensorflow provides a fully integrated mechanism for realtime visualization of important network
metrics, theycallTensorboard. Thetool’s functionanduser interfaceisrefinedandeasytounderstand(seeFigure8).
PyTorch: PyTorchdoesnotprovideanyvisualizationoptions,andthus third-party solutionsmustbereliedupon.
This includes tools likeGraphviz,Visdom,andevenTensorboard.
Thematurityof theTensorflowframework,andtheresourcesGooglehasdedicated to the tool shows its importance
heremakingTensorflowsuperior in thisarea. Relyingonthirdparty solutions for this importantaspect isundesir-
able.
Advantage: Tensorflow
4.4.3 Deployment
Generally ina researchcontextwedonotneed todeploymodels foruse inproductionenvironmentsaspartofan
applicationon theweb,mobileorother context. However inour labour industrypartnershipsoftenmean that
there is a requirement todeployourwork inaprototype formtosatisfyourpartnersneed. Fromouranalysis it
becameclear thatdeploymentoptionsonPyTorchare limited.
Tensorflow: TensorflowhasadedicatedsystemknownasTensorflowServing todeploymodels. It isa sophisticated
26
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
systemtogiveclientaccess tomodelcomputationwithoutchanging theunderlyingmodelarchitecture.
PyTorch: PyTorch does not natively support deployment. Instead it relies on third party tools, or more likely
conversionofPyTorchmodels toa framework thatdoessupportdeployment. Therearecurrentlyanumberof tools
available thatwill convertPyTorchmodels toTensorflow’s format.
Advantage: Tensorflow
4.5 Requirements Summary
InTable32wesummarize the requirementsand their status for each framework. Ascanbeseen fromthe table
Tensorflowmeetsallof the requirementsout-of-the-box,whereasPyTorch is lesscapable insomeareas. It isalso
clear thatPyTorchdoesnotmeet thenon-mandatory requirement forStackOverflowquestions, throughweexpect
this requirement tobemetbefore theendof theyear.
27
4 SoftwareEvaluation CSI5112-DeepLearningFrameworks
Table 32: Summaryof requirements statusby framework.
ID Requirement Tensorflow PyTorch
Productivity
Quality of Documentation
1 The frameworkmustprovidedocumentation thatdescribesall framework functions. Met Met
2 Frameworkdocumentationmustbeprovidedonline. Met Met
3 The frameworkdocumentationmustbeupdatedwitheachpoint releaseof thesoftware. Met Met
Community Support
4 The frameworkauthormustprovideanonlinediscussion forum Met Met
5 The frameworkmusthaveat least30mentionsonarXiv.orgpermonth Met Met
6 Theremustbeat least1000questions taggedwith the framework’snameonstackoverflow.com Met NotMet
7 Of the taggedquestionsavailableonstackoverflow.comtaggedat least50%mustbemarkedas
answered.
Met Met
Ease of installation
8 The framework, ifwrittenusingacompiled language,mustbe installablevia sourcecodecompilation. Met Met
9 The frameworkshouldbe installablevia theanacondapackagemanager. Met Met
Debugging Tools
10 The frameworkshouldprovideamethodtovisualizemodelsdefinedusing the framework Met 3rdParty
11 The frameworkshouldprovideamethod for thevisualizationof the trainingprocess Met 3rdParty
12 The frameworkmustprovideamechanismfordebugging thecode thatdefines thenetwork. Met Met
Availability of pretrained models
13 The frameworkshouldprovideaccess toasetofpretrainedmodelsandtheir respectivearchi-
tectures
Met Met
Technology Support
Algorithm creation language
14 The frameworkAPImust support thePython language Met Met
GPU Support
15 The frameworkmustallowtrainingand inferenceviasingleGPUonasinglenode Met Met
16 The frameworkmustallowtrainingand inferenceviamultipleGPUaonasinglenode Met Met
Computing cluster support
17 The frameworkshouldsupportdistributed trainingacrossmultiple servers. Met Provisional
18 The frameworkshouldsupportdistributed trainingand inferenceacrossmultiple servers. Met Met
Deployment
19 The frameworkmust support thedeploymentofmodels inorder toserviceexternal requests forpredictions Met 3rdParty
Performance
Benchmark performance
20 The frameworkmustperformbetter thanthebaselinemeasure Met Met
28
CSI5112-DeepLearningFrameworks 4 SoftwareEvaluation
4.6 Scoring
Basedonthefindingsouranalysiswewillnowpresent theresultingscores foreach frameworkbasedonthemethod
described insection3. Eachscorehasbeenassignedbasedona indepthanalysisofhowthe frameworksaddress
eachcharacteristic. Thescoresaresubjectivequantificationsbasedonsaidanalysis (seeTable33).
Table 33: Summaryof scores forourdecisioncharacteristics
Framework Alternatives
Tensorflow PyTorch
Item Decision Characteristics Weight Raw Weighted Raw Weighted
A Productivity 40 37.60 33.00
A.1 Qualityofdocumentation 5 0.75 0.04 1.00 0.05
A.2 Community support 40 1.00 0.40 0.80 0.32
A.3 Easeof installation 10 0.90 0.09 0.95 0.10
A.4 Debugging tools 15 0.75 0.11 0.80 0.12
A.5 Availabilityofpretrainedmodels 30 1.00 0.30 0.80 0.24
B Technology 40 33.60 20.60
B.1 Modelcreation language(s) 10 0.70 0.07 0.75 0.08
B.2 Framework implementation language 5 1.00 0.05 1.00 0.05
B.3 VisualizationOptions 20 1.00 0.20 0.50 0.10
B.4 Multi-GPUsupport 30 0.90 0.27 0.70 0.21
B.5 Computingcluster support 5 1.00 0.05 0.60 0.03
B.6 DeployementOptions 20 1.00 0.20 0.25 0.05
C Performance 20 18.00 20.00
C.1 Training time 60 0.90 0.54 1.00 0.60
C.2 Inference time 40 0.90 0.36 1.00 0.40
Total: 89.2 Total: 73.6
Basedonouranalysisandassociatedevaluationmethodology,TensorflowhasoutperformedPyTorch ineachof
the major areas except performance where PyTorch scored higher due to its superior time in the performance
benchmark.
29
5 ConclusionsandRecommendations CSI5112-DeepLearningFrameworks
5 Conclusions and Recommendations
Inevaluatingandcomparing these frameworks in theareas thataremost importantandrelevant toourwork in the
lab,wehavegainedabetterunderstanding,notonlyof the frameworks themselvesbutalsohowwemightuse them,
andthusourcomputingrequirementsbeyondthe just theAI framework. Hereweofferourconclusionsbasedon
these insights.
Both frameworksare incrediblypowerful, andbothcouldsatisfy thedemandsofour research in the lab. So, the
decisioncomesdowntothespecificways inwhicheachframeworkdiffer inkeyareasasdescribedinsubsection4.4.
PyTorch is an excellent tool who’s strengths were not captured in our requirements. It excels at being easy to
understandandquick to implement -as somewouldsay, it ismore likeanextension tonumpy, the reveredPython
scientificcomputingpackage in itseaseofuse. Wethink that this trait is thedriving forcebehind therapidadoption
ofPyTorchwhether fromnewusersorconverts fromother frameworks, includingTensorflow. Notonly, thatbut
PyTorchmaybe thehighestperformance frameworkavailablewhenoperatingonasingleGPU.However,perhaps
dueto its relative immaturity itdoeshavesomeshortcomings insomeareas thatwethinkare important toourwork;
namelyvisualization,distributedcomputing,anddistributionandTensorflowhasamuchmorerobustoffering in
theseareas.
Thedirectionof the lab is tomakegreateruseofmulti-GPUdistributedarchitectures;notonly tokeeppacewith
what ishappeningat thecuttingedgeof research,butalso to reduce training time inourexperimentsandthusget
answers toquestionsweneedtoprogressour research. Tensorflowhas this in itsDNA,making itmuch less risky to
makesuchamovewithTensorflowthanPyTorch,which is in theearly stagesofhavingaproperlyusable solution in
thisarea. Wehavereliedheavilyononlineanecdotes fromPyTorchusers toarriveat thisconclusion,butwe feel
confident this represents thecurrent stateofPyTorch.
Wealsomakeextensiveuseofvisualization inourworkflowinall stagesofaproject, fromtheearliest experiments,
to thepublication itself,not tomentionaswe-professors, andstudents, educateeachotheraboutourwork. Ten-
sorflow’sTensorBoard tool integratesperfectlywithTensorflowandprovidesexactlywhatweneedtosupportour
work. EventhoughPyTorchcanuseTensorboard, the integration isnot seamless, and inourexperiencesomewhat
clunky toworkwith. TheVisdomtooldoesworkbetter,butwe thinkPyTorchneeds to includeanativevisualization
tool tohave thebest results.
Aswehavediscussed,asour ties to industrygrowwemustbemindfulof theneedtoproductize,orat leastproto-
type interactionswithourmodelsdeployed to theweborotherapplication. This is anafter-thought inPyTorch
in our opinion. Again Tensorflow has deployment options as part of their core offering, which is an important
consideration. While itmaybepossible toconvertPyTorchmodels toTensorflowto takeadvantageof this, that is
notaworkflowthatappeals tous.
Finally,having tabulated thescores foreachkeycharacteristicofour requirements, setbasedonourevaluationof
30
CSI5112-DeepLearningFrameworks 5 ConclusionsandRecommendations
each frameworkwe foundthatTensorflowhadamuchstrongerperformance. Basedonthescoresweconclude that
Tensorflowisbetterable toaddressour requirements.
Thereforewerecommendthat the labadoptTensorflowas theprimary framework forAI research. Interestingly,
whileat theoutsetof thisevaluationourmindsetwas that thechosen frameworkwouldbe the lonestandard to the
exclusionofallothers. Nowasafinalwordwethink therewill continue tobearole forother tools includingPyTorch
inourworkflow. Therearecaseswherewethink thatPyTorchmightbe thebetter tool, andrather thanexclude it,
wearenowinclinedtoencouragemembers toexperimentwithother toolswhereappropriate. Ofcoursewewill
requireeveryone tobecomefamiliarwithTensorflowas theprimary frameworkused in the lab.
31
References CSI5112-DeepLearningFrameworks
References
[1] Michael Bandor. Quantitative Methods for Software Selection and Evaluation. Tech. rep. CMU/SEI-2006-
TN-026. Pittsburgh, PA: Software Engineering Institute, Carnegie Mellon University, 2006. URL: http://
resources.sei.cmu.edu/library/asset-view.cfm?AssetID=7949.
[2] MartínAbadietal. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. Softwareavailable
fromtensorflow.org.2015. URL: https://www.tensorflow.org/.
[3] ManasAgarwal. Deep Learning with PyTorch. A practical approach to building neural network models using
PyTorch. 1st ed.Birmingham:PacktPublishingLtd,Feb.2018. ISBN: 978-1-78862-433-6.
[4] TensorFlow documentation. URL: https://www.tensorflow.org/api_docs/ (visitedon
06/2018).
[5] StackOverflow Questions Tagged TensorFlow. URL: https://stackoverflow.com/questions/tagged/
tensorflow
(visitedon06/2018).
[6] Installing TensorFlow. URL: https://www.tensorflow.org/install/ (visitedon06/2018).
[7] TensorFlow Debugger. URL: https://www.tensorflow.org/programmers_guide/debugger (visitedon
06/2018).
[8] Tensorboard Tutorial. URL: http://edwardlib.org/tutorials/tensorboard (visitedon04/11/2018).
[9] TensorFlow github. URL: https://github.com/tensorflow/models/tree/master/research/slim
(visitedon06/2018).
[10] TensorFlow language-bindings. URL: https://www.tensorflow.org/extend/language_bindings
(visitedon06/2018).
[11] TensorFlow programming stack.URL:https://www.tensorflow.org/get_started/premade_estimators
(visitedon06/2018).
[12] TensorFlow GPU Support. URL: https://www.tensorflow.org/install/install_linux (visited on
06/2018).
[13] Distributed TensorFlow. URL: https://www.tensorflow.org/deploy/distributed (visitedon06/2018).
[14] TensorFlow Serving. URL: https://www.tensorflow.org/versions/r1.1/deploy/tfserve (visitedon
06/2018).
[15] PyTorch documentation. URL: http://pytorch.org/docs/0.3.1/ (visited
on04/06/2018).
[16] Welcome to PyTorch Tutorials. URL: http://pytorch.org/tutorials/ (visitedon04/06/2018).
[17] Document for PyTorch C API. URL:https://discuss.pytorch.org/t/document-for-c-api/287 (visited
on04/06/2018).
[18] StackOverflow Questions Tagged PyTorch. URL: https://stackoverflow.com/tags/pytorch (visitedon
04/20/2018).
[19] PyTorch, a year in… URL: http://pytorch.org/2018/01/19/a-year-in.html (visitedon04/11/2018).
32
CSI5112-DeepLearningFrameworks References
[20] PyTorch. URL: http://pytorch.org/ (visitedon04/06/2018).
[21] Facebook Research – Visdom. URL: https://research.fb.com/downloads/visdom (visitedon04/11/2018).
[22] AlexKrizhevsky, IlyaSutskever, andGeoffreyEHinton. “ImageNetClassificationwithDeepConvolutional
Neural Networks”. In: Advances in Neural Information Processing Systems 25. Ed. by F. Pereira et al. Cur-
ran Associates, Inc., 2012, pp. 1097–1105. URL: http://papers.nips.cc/paper/4824- imagenet-
classification-with-deep-convolutional-neural-networks .
[23] Serving PyTorch Models on AWS Lambda with Caffe2 and ONNX. URL: https : / / machinelearnings .
co/serving- pytorch- models- on- aws- lambda- with- caffe2- onnx- 7b096806cfac (visited on
04/10/2018).
[24] YangqingJiaetal. “Caffe:ConvolutionalArchitecture forFastFeatureEmbedding”. In: Proceedings of the 22Nd
ACM International Conference on Multimedia.MM’14.Orlando,Florida,USA:ACM,2014,pp.675–678. ISBN:
978-1-4503-3063-3. DOI: 10.1145/2647868.2654889. URL: http://doi.acm.org/10.1145/2647868.
2654889.
[25] K.SimonyanandA.Zisserman. “VeryDeepConvolutionalNetworks forLarge-Scale ImageRecognition”. In:
CoRR abs/1409.1556 (2014).
[26] KaimingHeetal. “Deepresidual learning for imagerecognition”. In: Proceedings of the IEEE conference on
computer vision and pattern recognition. 2016,pp.770–778.
33
