Schedule

We meet Tuesdays at 2:30-4:20 and Thursdays at 2:30-3:20 in SUR 3090.

This schedule is subject to change, although the general topics covered shall remain consistent.

Week	Date	Topics	Preparation	Extra
1	Jan 3	Background Introduction Software Quality Slides		Bad rabbit feed Roll-away crashes Airbag Bugs Remote Control of Cars
2	Jan 8	Testing Overview Unit Testing Slides		Fuzz Revisited Attaching a Rocket to the Chainsaw Fuzz Testing Subreddit Google Clean Code Talks 1 JetBrains on GoogleTest
2	Jan 10	More unit testing Testability Slides	Managing Nondeterminism Fowler on Mocking and Stubs Fowler on Dependency Injection	Modeling Readability to Improve Unit Tests
3	Jan 15	Testability MusicManager demo
3	Jan 17	Input Space Partitioning Slides
4	Jan 22	Input Space Partitioning Assignment 1 Due Slides		ISP at Freddie Mac
4	Jan 24	Combinatorial Testing Slides	Combinatorial Software Testing	Pairwise Testing in the Real World Evaluating Combinatorial Testing NIST on Pairwise Testing Prioritization and Constraints allow High Strength Tests Active Research Counterpoints Hexawise online test tool Microsoft PICT
5	Jan 29	Graph Based Testing Slides		Comparing Non-adequate Test Suites. Statement and branch coverage in practice Can Testedness be Effectively Measured? Coverage and Its Discontents
5	Jan 31	Graph Based Testing Data Flow Coverage		A Survey on Data-Flow Testing Towards Efficient Data-flow Test Data Generation Coverage Criteria for Logical Expressions
6	Feb 5	Logic Based Coverage
6	Feb 7	Mutation Testing Assignment 2 Due		Are mutants meaningful? Efficient mutation analysis Is it helpful? Coverage Is Not Strongly Correlated with Test Suite Effectiveness Code Coverage is a Strong Predictor of Test Suite Effectiveness
7	Feb 12	Regression Testing		Practical Regression Test Selection with Dynamic File Dependencies Refining Interprocedural Change-Impact Analysis usingEquivalence Relations
7	Feb 14	Test Planning Program Analysis Tools		Whittaker on Test Plans More from Whittaker The ACC Model jEdit Test Plan LLVM Testing & Quality
8	Feb 19	READING WEEK
8	Feb 21	READING WEEK
9	Feb 26	Program Analysis Tools Examples Homework 1 Slides	Keep your code sane with clang-tidy	Infer (from Facebook) Google Error Prone
9	Feb 28	Program Analysis Tools Examples Automated Test Generation Slides	PVS Studio and Chromium (Only read entries for 3 found bugs) Coverity and LibreOffice	CodeChecker Static analysis in the real world[PDF] Care and Feeding of C++'s Dragons FindBugs and static analysis Getting Rich through Bug Finding The Apple SSL/TLS bug Using FindBugs on Java programs
10	Mar 5	Automated Test Generation		OSS-Fuzz - Google's continuous fuzzing service for open source software American Fuzzy Lop libfuzzer Peach Fuzzer Structure-aware fuzzing for Clang and LLVM with libprotobuf-mutator Compositional FuzzingAided by Targeted Symbolic Execution Smart Greybox Fuzzing (on GitHub) VUzzer KLEE CREST
10	Mar 7	Performance		Tuning C++ Benchmarks C++ Performance Tuning Example A tale of performance debugging Google Native Microbenchmarking Avoiding Benchmarking Pitfalls on the JVM Google Java Microbenchmarking How not to measure latency
11	Mar 12	Performance Simple Benchmarks		The Fallacy of Premature Optimization Statistically Rigorous Java Performance Evaluation Producing wrong data without doing anything obviously wrong! Coz: Profiler for performance improvements Coz Video Stabilizer: Statistically Rigorous Performance Evaluation Data-oriented design in practice
11	Mar 14	Finish Performance		Julia Evans perf tutorial Kernel Perf Tutorial Brendan Gregg's perf resources Writing cache friendly C++ - Jonathan Müller New Improvements to Hash Table Performance - Malte Skarupke Swiss Tables and absl::Hash SMoTherSpectre: exploiting speculative execution through port contention
12	Mar 19	Security Slides	CWE/SANS Top 25 Dangerous Errors (Categories) Mitigation Approaches Secure Coding Practices	Vulnerable Dishwashers CERT C++ Coding Standard CERT C Coding Standard Sanitize, Fuzz, and Harden Your C++ Code Responding to DDOS Principles of Security Assurance Actionable research from penetration testing engagements Cybersecurity Laws in Canada HackerOne bug bounties american fuzzy lop Angora: Efficient Fuzzing by Principled Search CERT Security Tools The Frequency of Known Vulnerabilities in JavaScript Avoiding the Top 10 Security Flaws NetSec Subreddit libsodium Security Engineering (Online Book) Java JRE Vulnerabilities Project 0 Legal Liabilities Block Oriented Programming Compiler
12	Mar 21	Bug Advocacy Slides	A manifesto for error reporting Writing Clear Bug Reports (especially from 8:00 on)	Bug Advocacy in General What makes a good bug report? Where Do Developers Log?
13	Mar 26	Debugging Property Based Testing Slides		Delta debugging The "Wolf Fence" Algorithm for Debugging A review of reverse debugging Mozilla RR Time Travel Debugging in Visual Studio Hypothesis for property based testing DeepState: Symbolic Unit Testing for C and C++ Solving the Water Jug Problem from Die Hard 3 with TLA+ and Hypothesis
13	Mar 28	A/B Testing and Bandits Slides		Unbounce guide to A/B testing Most A/B Test Results Are Illusory Optimizely A/B test size calculator Evan's Awesome A/B Testing Tools Regression to the Mean Bandits at Optimizely A Tutorial on Thompson Sampling An Efficient Bandit Algorithm for Realtime Multivariate Optimization
14	Apr 2	More A/B Testing Model Checking		TLA+ Video Course Protocol and TLA+ lectures Use of Formal Methods at Amazon Web Services Why Amazon Chose TLA+ Writing correct lock-free and distributed stateful systems in Rust, assisted by TLA+
14	Apr 4	Risk and Ethics Project Presentations	Feynman's Minority Report Vice article on the report	Challenger Accident Investigation Report Engineering Ethics Case Study: The Challenger Disaster