Schedule

We meet Mondays at 4:30-6:20 and Thursdays at 5:30-6:20 in SRYE 1002. Recordings in general will not be available. More details discussed on the first day of class.

This schedule is subject to change.

Week	Date	Topics	Preparation	Extra
1	Sep 8	Background Introduction		Bad rabbit feed Roll-away crashes Airbag Bugs Remote Control of Cars
2	Sep 12	Testing Overview Unit Testing	Managing Nondeterminism Fowler on Mocking and Stubs Fowler on Dependency Injection	Fuzz Revisited Attaching a Rocket to the Chainsaw Fuzz Testing Subreddit Google Clean Code Talks 1 JetBrains on GoogleTest Google Clean Code Talks 1 JetBrains on GoogleTest Modeling Readability to Improve Unit Tests TDD: Where did it all go wrong
2	Sep 15	UI Testing
3	Sep 19	Queen's Funeral All classes canceled Testability Practice (optional) Sources	Last day to drop
3	Sep 22	Input Space Partitioning Combinatorial Testing	Combinatorial Software Testing	ISP at Freddie Mac Boundary Value Exploration for Software Analysis Pairwise Testing in the Real World Evaluating Combinatorial Testing NIST on Pairwise Testing Prioritization and Constraints allow High Strength Tests Active Research Counterpoints Hexawise online test tool Microsoft PICT
4	Sep 26	Exercise 1 Due Graph Based Testing		Comparing Non-adequate Test Suites. Statement and branch coverage in practice Can Testedness be Effectively Measured? Coverage and Its Discontents A Survey on Data-Flow Testing Towards Efficient Data-flow Test Data Generation Coverage Criteria for Logical Expressions
4	Sep 29	ALL CS CLASSES CANCELLED
5	Oct 3	Logic Based Coverage		A Practical Tutorial on Modified Condition/Decision Coverage (NASA) Additional NASA discussion on limitations The Effect of Program and Model Structure on MC/DC Test Adequacy Coverage
5	Oct 6	HW 1 Due Mutation Testing		Are mutants meaningful? Efficient mutation analysis Is it helpful? Coverage Is Not Strongly Correlated with Test Suite Effectiveness Code Coverage is a Strong Predictor of Test Suite Effectiveness µDroid: an energy-aware mutation testing framework for Android Exposing library API misuses via mutation analysis Mutation Reduction Strategies Considered Harmful
6	Oct 10	Thanksgiving, NO CLASS Exercise 2 Due Tomorrow
6	Oct 13	Regression Testing Test Planning		Practical Regression Test Selection with Dynamic File Dependencies Refining Interprocedural Change-Impact Analysis using Equivalence Relations Whittaker on Test Plans More from Whittaker The ACC Model jEdit Test Plan LLVM Testing & Quality
7	Oct 17	Examples Program Analysis Tools	CLASS IN SRYE 2016 today only	Infer (from Facebook) Google Error Prone Tricorder: Building a Program Analysis Ecosystem CodeChecker Static analysis in the real world[PDF] Care and Feeding of C++'s Dragons FindBugs and static analysis Getting Rich through Bug Finding The Apple SSL/TLS bug Using FindBugs on Java programs
7	Oct 20	Program Analysis Tools Random Testing	PVS Studio and Chromium (Only read entries for 3 found bugs) Coverity and LibreOffice Keep your code sane with clang-tidy	OSS-Fuzz - Google's continuous fuzzing service for open source software American Fuzzy Lop libfuzzer Peach Fuzzer Structure-aware fuzzing for Clang and LLVM with libprotobuf-mutator Compositional FuzzingAided by Targeted Symbolic Execution Smart Greybox Fuzzing (on GitHub) VUzzer Randoop GRT: Program-Analysis-Guided Random Testing
8	Oct 24	Symbolic Execution Bug Advocacy		KLEE CREST A manifesto for error reporting Writing Clear Bug Reports (especially from 8:00 on) Bug Advocacy in General What makes a good bug report? Where Do Developers Log?
8	Oct 27	Property Based Testing	CLASS IN SRYE 2016 today only	Hypothesis for property based testing DeepState: Symbolic Unit Testing for C and C++ Solving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesis
9	Oct 31	Exercise 3 Due (tomorrow) Debugging Performance		Delta debugging The "Wolf Fence" Algorithm for Debugging A review of reverse debugging The RR Debugger Time Travel Debugging in Visual Studio Tuning C++ Benchmarks C++ Performance Tuning Example A tale of performance debugging Google Native Microbenchmarking Avoiding Benchmarking Pitfalls on the JVM Google Java Microbenchmarking How not to measure latency
9	Nov 3	HW 2 Due Simple Benchmarks Performance		The Fallacy of Premature Optimization Statistically Rigorous Java Performance Evaluation Producing wrong data without doing anything obviously wrong! Coz: Profiler for performance improvements Coz Video Stabilizer: Statistically Rigorous Performance Evaluation Data-oriented design in practice Julia Evans perf tutorial Kernel Perf Tutorial Brendan Gregg's perf resources Writing cache friendly C++ - Jonathan Müller New Improvements to Hash Table Performance - Malte Skarupke Swiss Tables and absl::Hash Vulnerable Dishwashers
10	Nov 7	Security		CERT C++ Coding Standard CERT C Coding Standard Sanitize, Fuzz, and Harden Your C++ Code Responding to DDOS Principles of Security Assurance Actionable research from penetration testing engagements Cybersecurity Laws in Canada HackerOne bug bounties
10	Nov 10	Security	CWE/SANS Top 25 Dangerous Errors (Categories) Mitigation Approaches Secure Coding Practices	Microsoft Digital Defense Report american fuzzy lop Angora: Efficient Fuzzing by Principled Search CERT Security Tools The Frequency of Known Vulnerabilities in JavaScript Avoiding the Top 10 Security Flaws NetSec Subreddit libsodium Security Engineering (Online Book) Java JRE Vulnerabilities Project 0 Legal Liabilities Block Oriented Programming Compiler SMoTherSpectre: exploiting speculative execution through port contention
11	Nov 14	Security A/B Testing		Unbounce guide to A/B testing Most A/B Test Results Are Illusory Optimizely A/B test size calculator Evan's Awesome A/B Testing Tools Regression to the Mean Bandits at Optimizely A Tutorial on Thompson Sampling An Efficient Bandit Algorithm for Realtime Multivariate Optimization
11	Nov 17	A/B Testing
12	Nov 21	Exercise 4 Due Model Checking & TLA+		TLA+ Video Course Protocol and TLA+ lectures Use of Formal Methods at Amazon Web Services Experience of software engineers using TLA+, PlusCal and TLC Why Amazon Chose TLA+ Writing correct lock-free and distributed stateful systems in Rust, assisted by TLA+ Compiling PlusCal into Go
12	Nov 24	Model Checking & TLA+
13	Nov 28	Chaos Engineering		The Eight Fallacies of Distributed Computing Chaos Engineering -- traffic and chaos team @ Netflix Chaos Engineering @ USENIX Rethinking How the Industry Approaches Chaos Engineering Chaos Engineering: the history, principles, and practice Automating chaos experiments in production Chaos Community Broadcast Chaos Monkey Mastering Chaos - A Netflix Guide to Microservices
13	Dec 1	HW 3 Due Chaos Engineering
14	Dec 5	Exercise 5 Due Risk Review / All Questions Answered	Feynman's Minority Report[alternate] Vice article on the report	Challenger Accident Investigation Engineering Ethics Case Study: The Challenger Disaster How Complex Systems Fail Probability-impact assessment S3001: Guidelines for Risk Management The six mistakes executives make in risk management Preparing for High-impact, Low-probability Events Preparing for High-impact, Low-probability Events