Performance Test Ponderings

Before we start on the bit about ChatGPT, I’d like to introduce todays image which is from DALL-E 3 with the prompt “a group of performance testers planning a stress test”. This one is probably subtly creepier than last weeks but still manages to sit comfortably on the wrong slope of the uncanny valley. Witness the sheer number of graphs they have before the test is run, and the alien arms pointing at, well, what is that exactly on the wall – Performance Initsauce?

Now – onto the good stuff. This week I asked ChatGPT “How should I plan a performance test?” and below is what it said. (Aside, is it an “it”? That almost seems rude…) It actually said rather a lot, so I’ll go a section at a time and add some thoughts.

Planning a performance test is crucial to ensure that your application or system meets the desired performance criteria and provides a positive user experience. Here’s a structured approach to plan a performance test:

No arguments there… lets get the popcorn out and see what it says.

1. Define Objectives:

• Understand the primary goals: Is it for assessing the system’s robustness, scalability, or responsiveness?
• Clearly define the performance criteria: What is the acceptable response time? How many concurrent users should the system handle?

Um…kinda. These to me are second level objectives. The primary objectives should always be business driven. That’s what the robustness, scalability and responsiveness is for. This is rather a technical view. Without that business perspective, the test doesn’t make much sense.

10 points to Hufflepuff though for acceptable response time – I’ve been on a few projects where the client has declined to supply success criteria and just said “test it and tell me what it does”. A few weeks later and you say “average response is 3 seconds” and the client says “Is that good?”. I always feel like Deep Thought in Hitchhikers Guide to the Galaxy… perhaps you should define your question.

Concurrency… is a topic and a half and the other controversial performance tester topic. I’m going to deep dive into this Objectives stuff next week, so I might get into that business next week. What’s next?

2. Gather System Information:

• Architecture of the system (front-end servers, databases, caching layers, etc.)
• Technology stack (languages, frameworks, server types, etc.)
• Deployment environment details (physical/virtual/cloud, networking, etc.)

Ah! Perhaps that’s what the Performance Testers From Beyond are doing above. Yes, no arguments here. You probably don’t need all the little details, but it doesn’t hurt.

3. Determine the Key Scenarios:

• Identify the critical business transactions or user paths. For an e-commerce website, this could be product search, adding items to the cart, or the checkout process.
• Prioritize scenarios based on frequency, importance, and potential risk.

I would say product search, adding items *and* checkout myself. You want the full footprint, not just a big toe. And for priority, add “impact”. You can run something once a week that kills the system.

I also like to map out how the scenarios hit the different bits of kit we’ve found, just in case we need to stub out part of the system. But that’s not always necessary. On we go

4. Identify Test Data:

• User profiles, product details, database entries, etc.
• Ensure data realism: The data should mimic actual production data, but without breaching any privacy norms.

You know, a few little things. This is a huge area, especially for SAP. And I’m not sure you need it all upfront – but oh yes, you need this.

5. Determine Workload Distribution:

• Understand typical user behavior: How many users do task A vs. task B?
• Consider using varied distribution strategies like Uniform, Ramp-up, Peak load, etc.

I’d have done some of that around stage 3 myself, but there’s definitely a task to gather that and sign it off with the business. After all, it’s their system.

6. Set up the Test Environment:

• The test environment should closely mimic the production setup.
• Ensure the environment is isolated so that other activities don’t skew the results.

Again, two little lines with so much information. We’ll deep dive all this stuff later in the series, but a world of yes to both. I’m sure everyone has a story for both these points (the test system that only had 33% of the hardware and no one told me; the live printers that were still attached to the test system and spat out 100s of test prints – oh the horror).

7. Choose the Right Performance Testing Tool:

• Tools like Apache JMeter, LoadRunner, Gatling, and others might be suitable based on your needs and tech stack.
• Ensure the tool can simulate the desired number of users and capture relevant metrics.

Now we’re chosing a tool?? OK. But at least we’re chosing the right one. We’ll see how we do that in the deep dive I presume.

8. Scripting:

• Develop test scripts for the selected scenarios.
• Make scripts modular and reusable.
• Implement parameterization to ensure varied data inputs.

I’m not 100% sure on modular and re-usable, especially for one-off tests. But that’s just an opinion. Otherwise this is fine but like the others – let’s see when we get to the detail.

By-the-by – assuming I’m doing at least one post per sub-section, I hope you’ve all got a lot of popcorn.

9. Baseline Test:

• Execute a test with a minimal load to create a baseline.
• This helps in identifying obvious performance bottlenecks before larger tests.

Yup, I’m good with that. There are other ways as well, especially if you’re making a upgrade, But we can get into that later.

10 Execute the Test:

• Monitor system metrics such as CPU, memory, disk I/O, and network usage.
• Capture application-specific metrics, logs, and any errors.

I agree *someone* should monitor – not sure it’s always going to be me. Like I said last week, you need a multi-disciplinary team for these tests and they probably all have a different view of the world and different tools to capture that view. For me this is more “liaise with technical teams”.

11. Results Analysis:

• Compare the results against the set performance criteria.
• Look for bottlenecks and determine the cause (e.g., slow database queries, inadequate server resources).

“Look for bottlenecks and determine the cause” – ok! Well same point as above really, and this is another huge area – and I’m curious on the detail…

12. Tune & Retest:

• Make necessary optimizations based on the test results.
• Retest to validate improvements.

The thing here is, that it’s not always clear what the “necessary optimizations” are. “Try assumed optimization” and retest might be better. Not all improvements actually make things better. This is one advantage of performance testing, you try this stuff out in a safe environment (and without a downed business screaming at you to just fix it).

13. Report & Documentation:

• Create a comprehensive report detailing the test setup, execution, findings, and recommendations.
• This helps stakeholders understand the performance standing and any necessary actions.

Yes indeed – and add any lessons learned for the next poor soul that has to do it – lest it be you.

14. Review & Feedback Loop:

• Regularly review the performance testing strategy and update it based on system changes, business needs, or any learnings from the past tests.

Remember, performance testing is iterative. As your application evolves, you’ll likely need to revisit and adjust your tests to ensure they remain relevant and effective.

Well indeed, nothing to argue with there.

So, how did ChatGPT do? Not bad for interview questions, I think, and I’ve made a few points as we went. But, this didn’t tell me how to plan, it gave me a sort of plan. From this information I don’t think I can figure out useful trivia like: How long will the test take? How long to execute? How much will it cost (and what do I do if the client says it has to cost less)? Maybe when we get into the detail we’ll see those things.

Anyway – that’s it for this week – next week we’ll look at that objectives setting stage it puts at the start. Have a good week and happy system smashing!