Software testing

Notes

• My advice is never to start with unit tests. Start with integration tests and work your way backwards until it stops being worth your time.
• I use TDD when the problem and desired API are both clearly defined. A lot of front-end work is more exploratory in nature, which can make writing tests up front quite challenging. Often (and perhaps even ideally), the desired UI and interactions aren't really set in stone.
• Unit tests are a waste of time when building a product. The only thing that matters is that the product works for your users. The best way to verify that is with e2e tests.
• Almost all the advice you hear about software testing is bad. It's either bad on its face or it leads to bad outcomes or it distracts by focusing on the wrong thing (usually tools).
• People love debating what percentage of which type of tests to write, but it's a distraction. Nearly zero teams write expressive tests that establish clear boundaries, run quickly & reliably, and only fail for useful reasons. Focus on that instead.
• We are moving away from data-testid completely in favor of accessibility attributes like role and aria-label to encourage better semantics and make tests be more human-like.

Links

• Time Travelling and Fixing Bugs with Property-Based Testing (2019)
• Awesome Testing - Curated list of testing software, extensions and resources.
• Software Testing: From Theory to Practice book - Covers the most important testing techniques needed to build high quality software systems.
• Stressgrid - Open-source cloud-native tool for load testing at the scale of millions of simulated devices.
• Storycap - Storybook Addon, Save the screenshot image of your stories via puppeteer.
• React Screenshot Test - Dead simple library to screenshot test React components.
• Baretest - Extremely fast and simple JavaScript test runner.
• fast-check - Property based testing framework for JavaScript (like QuickCheck) written in TypeScript. (Web)
• Puppeteer examples - Puppeteer Headless Chrome examples for real life use cases.
• Tips on testing React components
• TestCheck.js - Generative property testing for JavaScript.
• Awesome Software Testing
• Rainforest - Flexible software testing platform. (CLI)
• Puppetron - Headless Chrome Node API)-based rendering solution.
• Tox - Command line driven CI frontend and development task automation tool.
• How We Test Vector (2020) - Survey of techniques we've found useful.
• puppeteer + dom-testing-library
• How SQLite Is Tested (HN)
• Are Tests Necessary in TypeScript? (2020) (HN)
• Appium - Automation for iOS, Android, and Windows Apps.
• Awesome Test Anything - Useful resources for the Test Anything Protocol.
• Systematically Covering Input Structure
• Compositional Testing of Internet Protocols
• Testing sync at Dropbox (2020)
• How NASA does software testing and QA (2020)
• Headless Recorder - Chrome extension that records your browser interactions and generates a Puppeteer or Playwright script.
• Unit Testing at REKKI (2020)
• Ideology (2015) (Lobsters)
• Gauge - Light weight cross-platform test automation. (Web)
• What are Unit Tests For? (2020) (Lobsters)
• Spark Joy by Running Fewer Tests (2020)
• The lazy programmer's guide to writing thousands of tests: An introduction to Property-Based Tests (2020)
• Against Testing (2020) (HN) (Lobsters)
• Unit Testing is Overrated (2020) (HN) (HN)
• Implementing mutation testing in a niche language with Stryker (2020)
• BrowserStack - App & Browser Testing Made Easy.
• Karate - Open-source tool to combine API test-automation, mocks, performance-testing and even UI automation into a single, unified framework. (HN)
• Thoughts on TDD (2020)
• What’s wrong with snapshot tests (2019)
• Mutation Testing: A Tale of Two Suites (2020)
• In Testing, Scenarios Are King (2020)
• Chaos Engineering book (2020) - Crash test your applications. (Code)
• Ian Cooper - TDD, Where Did It All Go Wrong (2017)
• Is TDD Dead? (2014) (HN)
• Functional Programming and Property-Based Testing
• How They Test - Collection of public resources about how software companies test their software. (Code)
• Test smarter, not harder (2020)
• TestDriven Courses (GitHub)
• Anatomy of a Good Test (2020)
• Assertion Injection : Unit Test Design Pattern
• High-level Language Features and Testing
• I/O Benchmark Suite
• Prefer Fakes Over Mocks (2020) (Lobsters) (HN)
• Awesome Testing Tools
• Waldo - Create reliable automated mobile tests with no-code testing platform.
• OpenFaaS Puppeteer template - Uses docker-puppeteer by buildkite to give you access to Puppeteer.
• Metamorphic Testing (2019)
• Role of testing in software development life cycle (2020)
• UI Testing Best Practices
• Mugshot - Node library for doing visual regression testing.
• Probabilistic flakiness: How do you test your tests? (2020)
• Knapsack Pro - Speed up your tests with optimal test suite parallelization. (GitHub)
• Cross-Branch Testing (2020) (Lobsters)
• Load testing is hard, and the tools are not great. But why? (2021) (HN)
• Ask HN: How should I test all potential combinations of user actions? (2021)
• Teaching and Assessing Property-Based Testing (2021)
• Does It Matter to Test-First or toTest-Last? (Lobsters)
• Writing Environment-Agnostic Functional Acceptance Tests (2021)
• Ruby Conf 12 - Boundaries by Gary Bernhardt
• Effective Property-Based Testing (2021) (Lobsters) (HN)
• Mutation Driven Testing: When TDD Just Isn’t Good Enough (2021) (HN)
• Problems with TDD (2009) (Lobsters)
• Make tests a part of your app (2021)
• Using Relational Problems to Teach Property-Based Testing (2021)
• Kiwi TCMS - Open Source Test Case Management System. (Web)
• Automated Testing with Specifications (2021)
• Core Concepts in Software Testing Workshop
• TDD: Getting Out of My Own Way (2021)
• How to Test (2021)
• Testing in the Twenties (2021)
• start-server-and-test - Starts server, waits for URL, then runs test command; when the tests end, shuts down server.
• On the Diverse And Fantastical Shapes of Testing (2021) (Tweet)
• TDD from the Factorio Team (2021) (HN)
• AtomicJar - Make integration testing easier for developers across the software development lifecycle.
• Rethinking Software Testing: Perspectives from the world of Hardware (2019) (Lobsters) (HN)
• You should want to write unit tests and here is why (2021)
• We need to talk about testing (2021) (HN)
• Property-based testing is not the same as partition testing (2021)
• Choosing properties for property-based testing
• Growth Book - Open Source A/B Testing Platform. (Code) (HN)
• Ponicode - Quick and easy unit testing.
• What is Property Based Testing? (2016) (HN)
• We killed our end-to-end test suite (2021) (HN)
• Coverage is not strongly correlated with test suite effectiveness (2021) (HN)
• Good tests don't change (2021)
• The Fundamentals of Testing with Persistence Layers (2021)
• Skeleton And Principles For A Maintainable Test Suite (2021)
• Tests aren’t enough: Case study after adding type hints to urllib3 (2021) (HN)
• Testing Without Mocks: A Pattern Language (2018)
• FlakyBot - Flaky test management. (HN)
• Awesome Testing Courses & Tutorials
• Interpreting A/B test results: false positives and statistical significance (2021) (HN)
• CI::Queue - Distribute tests over many workers using a queue.
• Reflect - Automated Web Testing.
• Generate All the Things (2021) (Lobsters)
• Tests in a box: Shipping your tests in a container for fun and profit - Federico Paolinelli (2020)
• I test in prod (2021) (HN)
• Argos - Automate visual regression testing. (Web)
• Gnomock - Integration and end-to-end testing toolkit. It uses Docker to create temporary containers for application dependencies, setup their initial state and clean them up in the end.
• Automated, Targeted Testing of Property-Based Testing Predicates
• Testing (2021)
• Overview of Property-Based Testing Functionality
• How the GNU coreutils are tested (2017)
• Fast Test, Slow Test (2012)
• Destroy All Software - Software Testing
• nut.js - Native UI testing / controlling with node.
• split_tests - Utility to split test files into parallel CI containers.
• Write Fewer Tests! From Automation to Autogeneration - David Khourshid (2019)
• granularity of testing (2021)
• The argument against clearing the database between tests (2020) (HN)
• Favor real dependencies for unit testing (2022) (Lobsters)
• Kent - Fake Sentry server for local development, debugging, and integration testing.
• Gonkey - Testing automation tool. Can bomb the service with prepared requests and check the responses.
• Ten-year experience in DBMS testing (2022)
• Three Rules of TDD
• End to End CI Tests in Minutes with Netlify, Playwright & Github Actions (2021)
• How do you check quality of each release (2022)
• ZAP (Report Anything Protocol) - Streamable structured interface for real time reporting of developer tools.
• How to run end-to-end tests 10x faster with firecracker (2022)
• Ask HN: How do you keep track of software requirements and test them? (2022)
• Storybook Test Runner - Turn stories into executable tests.
• Test Anything Protocol
• Eventsim - Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic.
• Some thoughts on naming tests (2022)
• Awesome Visual Regression Testing
• “Don’t Mock What You Don’t Own” in 5 Minutes (2022)
• Ask HN: Do you load test your applications? If so, how? (2022)
• Unit and Integration Tests (2022) (Lobsters)
• Smoke - Designed to test anything that can be wrapped with a command-line interface.
• Test against what won't change (Lobsters)
• Property-Based Testing Against a Model of a Web Application (2022)
• I have complicated feelings about TDD (2022) (Lobsters)