IT System Monitoring Tools

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

System Performance Metrics Every Engineer Should Know Your API is slow. But how slow, exactly? You need numbers. Real metrics that tell you what's actually broken and where to fix it. Here are the four core metrics every engineer should know when analyzing system performance: - Queries Per Second (QPS): How many incoming requests your system handles per second. Your server gets 1,000 requests in one second? That's 1,000 QPS. Sounds straightforward until you realize most systems can't sustain their peak QPS for long without things starting to break. - Transactions Per Second (TPS): How many completed transactions your system processes per second. A transaction includes the full round trip, i.e., the request goes out, hits the database, and comes back with a response. TPS tells you about actual work completed, not just requests received. This is what your business cares about. - Concurrency: How many simultaneous active requests your system is handling at any given moment. You could have 100 requests per second, but if each takes 5 seconds to complete, you're actually handling 500 concurrent requests at once. High concurrency means you need more resources, better connection pooling, and smarter thread management. - Response Time (RT): The elapsed time from when a request starts until the response is received. Measured at both the client level and server level. A simple relationship ties them all together: QPS = Concurrency ÷ Average Response Time More concurrency or lower response time = higher throughput. Over to you: When you analyze performance, which metric do you look at first, QPS, TPS, or Response Time? -- We just launched the all-in-one tech interview prep platform, covering coding, system design, OOD, and machine learning. Launch sale: 50% off. Check it out: https://lnkd.in/euwKh6u8 #systemdesign #coding #interviewtips .

𝗔𝗻𝗸𝗶𝘁𝗮: You know 𝗣𝗼𝗼𝗷𝗮, last Monday our new data pipeline was live in cloud and it failed terribly. Literally had an exhaustive week fixing the critical issues. 𝗣𝗼𝗼𝗷𝗮: Ohh, so don’t you use Cloud monitoring for data pipelines? From my experience always start by tracking these four key metrics: latency, traffic, errors, and saturation. It helps you to check your pipeline health, if it's running smoothly or if there’s a bottleneck somewhere.. 𝗔𝗻𝗸𝗶𝘁𝗮: Makes sense. What tools do you use for this? 𝗣𝗼𝗼𝗷𝗮: Depends on the cloud platform. For AWS, I use CloudWatch—it lets you set up dashboards, track metrics, and create alarms for failures or slowdowns. On Google Cloud, Cloud Monitoring (formerly Stackdriver) is awesome for custom dashboards and log-based metrics. For more advanced needs, tools like Datadog and Splunk offer real-time analytics, anomaly detection, and distributed tracing across service. 𝗔𝗻𝗸𝗶𝘁𝗮: And what about data lineage tracking? How do you track when something goes wrong, it's always a nightmare trying to figure out which downstream systems are affected. 𝗣𝗼𝗼𝗷𝗮: That's where things get interesting. You could simply implement custom logging to track data lineage and create dependency maps. If the customer data pipeline fails, you’ll immediately know that the segmentation, recommendation, and reporting pipelines might be affected. 𝗔𝗻𝗸𝗶𝘁𝗮: And what about logging and troubleshooting? 𝗣𝗼𝗼𝗷𝗮: Comprehensive logging is key. I make sure every step in the pipeline logs events with timestamps and error details. Centralized logging tools like ELK stack or cloud-native solutions help with quick debugging. Plus, maintaining data lineage helps trace issues back to their source. 𝗔𝗻𝗸𝗶𝘁𝗮: Any best practices you swear by? 𝗣𝗼𝗼𝗷𝗮: Yes, here’s what’s my mantra to ensure my weekends are free from pipeline struggles - Set clear monitoring objectives—know what you want to track. Use real-time alerts for critical failures. Regularly review and update your monitoring setup as the pipeline evolves. Automate as much as possible to catch issues early. 𝗔𝗻𝗸𝗶𝘁𝗮: Thanks, 𝗣𝗼𝗼𝗷𝗮! I’ll set up dashboards and alerts right away. Finally, we'll be proactive instead of reactive when it comes to pipeline issues! 𝗣𝗼𝗼𝗷𝗮: Exactly. No more finding out about problems from angry business users. Monitoring will catch issues before they impact anyone downstream. In data engineering, a well-monitored pipeline isn’t just about catching errors—it’s about building trust in every insight you deliver. #data #engineering #reeltorealdata #cloud #bigdata

🔍 Diving into LLM System Metrics: What Really Matters After analyzing six months of LLM deployment data, here are the metrics that actually matter: ⚡ Reliability: 99.99% uptime - because enterprise solutions demand consistency ⏱️ Response Time: 500ms average - crucial for real-time applications 📈 Scale: Processing 10B+ tokens weekly across enterprise workloads 🔒 Security: 256-bit encryption, with <0.001% unauthorized access attempts 💰 Efficiency: Adaptive token allocation reducing operational costs by 30% 🧠 Intelligence: 5 specialized models, each learning from 1M+ daily interactions What stands out is how these metrics are evolving. While response time was the focus couple of years back, we're seeing a clear shift toward efficiency and specialized performance metrics in 2025. 💭 Curious to hear from other AI practitioners: Which metrics are you prioritizing for your LLM systems this year?

👉 Map Dependencies to Find Bottlenecks It is hard for a team to ship fast when it has to wait on other departments, teams, or suppliers to do something they depend on. 💤 For example, when deployments are performed by an external team that is swamped with other work. Or when another department has to perform specialized testing before approval is given. Whatever their dependencies, they are generally outside of the control of a team. 🤷♀️ That makes the delays unpredictable. Even when a team considers something “Done”, weeks or months may pass before their work actually reaches stakeholders. This greatly impedes a team’s ability to work empirically and reduce the risk associated with complex work. This experiment is about creating transparency around dependencies and their effect on your team’s ability to ship fast. It was inspired by the Dependency Spiders in Jimmy Janlén's “96 Visualization Examples” (which contains many other awesome visualizations 🎉 ). To implement this experiment, do the following: 1️⃣ Draw your team in the middle of a big piece of paper. Together, create a list of the teams, people, and departments you frequently need something from to create a Done Increment or release it. Whose approval do you need? 🤔 Who needs to perform an activity for your team to continue? Draw the sources you depend on around your team, like the legs of a spider. 🕷 2️⃣ Whenever your team needs something from someone outside the team, capture the request and the date it was issued on a sticky note and put it next to the source on the canvas. When the request is fulfilled, write the number of days you had to wait on the sticky. At the end of the Sprint, calculate the average wait time in days for all the fulfilled requests and move them to an archive. 3️⃣ Use the Dependency Spider and the average wait time as input for your Sprint Reviews and Sprint Retrospectives. 🤔 What actions can you take to reduce the impact of dependencies on your ability to ship? 🤔 How can you include and collaborate with them to remove or reduce dependencies? 🤔 How can you leverage support from your Product Owner and stakeholders to change your team's environment so that you can ship value to them faster? What are your thoughts after reading this post❓ What other ideas do you have❓

In today’s always-on world, downtime isn’t just an inconvenience — it’s a liability. One missed alert, one overlooked spike, and suddenly your users are staring at error pages and your credibility is on the line. System reliability is the foundation of trust and business continuity and it starts with proactive monitoring and smart alerting. 📊 𝐊𝐞𝐲 𝐌𝐨𝐧𝐢𝐭𝐨𝐫𝐢𝐧𝐠 𝐌𝐞𝐭𝐫𝐢𝐜𝐬: 💻 𝐈𝐧𝐟𝐫𝐚𝐬𝐭𝐫𝐮𝐜𝐭𝐮𝐫𝐞: 📌CPU, memory, disk usage: Think of these as your system’s vital signs. If they’re maxing out, trouble is likely around the corner. 📌Network traffic and errors: Sudden spikes or drops could mean a misbehaving service or something more malicious. 🌐 𝐀𝐩𝐩𝐥𝐢𝐜𝐚𝐭𝐢𝐨𝐧: 📌Request/response counts: Gauge system load and user engagement. 📌Latency (P50, P95, P99):  These help you understand not just the average experience, but the worst ones too. 📌Error rates: Your first hint that something in the code, config, or connection just broke. 📌Queue length and lag: Delayed processing? Might be a jam in the pipeline. 📦 𝐒𝐞𝐫𝐯𝐢𝐜𝐞 (𝐌𝐢𝐜𝐫𝐨𝐬𝐞𝐫𝐯𝐢𝐜𝐞𝐬 𝐨𝐫 𝐀𝐏𝐈𝐬): 📌Inter-service call latency: Detect bottlenecks between services. 📌Retry/failure counts: Spot instability in downstream service interactions. 📌Circuit breaker state: Watch for degraded service states due to repeated failures. 📂 𝐃𝐚𝐭𝐚𝐛𝐚𝐬𝐞: 📌Query latency: Identify slow queries that impact performance. 📌Connection pool usage: Monitor database connection limits and contention. 📌Cache hit/miss ratio: Ensure caching is reducing DB load effectively. 📌Slow queries: Flag expensive operations for optimization. 🔄 𝐁𝐚𝐜𝐤𝐠𝐫𝐨𝐮𝐧𝐝 𝐉𝐨𝐛/𝐐𝐮𝐞𝐮𝐞: 📌Job success/failure rates: Failed jobs are often silent killers of user experience. 📌Processing latency: Measure how long jobs take to complete. 📌Queue length: Watch for backlogs that could impact system performance. 🔒 𝐒𝐞𝐜𝐮𝐫𝐢𝐭𝐲: 📌Unauthorized access attempts: Don’t wait until a breach to care about this. 📌Unusual login activity: Catch compromised credentials early. 📌TLS cert expiry: Avoid outages and insecure connections due to expired certificates. ✅𝐁𝐞𝐬𝐭 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐞𝐬 𝐟𝐨𝐫 𝐀𝐥𝐞𝐫𝐭𝐬: 📌Alert on symptoms, not causes. 📌Trigger alerts on significant deviations or trends, not only fixed metric limits. 📌Avoid alert flapping with buffers and stability checks to reduce noise. 📌Classify alerts by severity levels – Not everything is a page. Reserve those for critical issues. Slack or email can handle the rest. 📌Alerts should tell a story : what’s broken, where, and what to check next. Include links to dashboards, logs, and deploy history. 🛠 𝐓𝐨𝐨𝐥𝐬 𝐔𝐬𝐞𝐝: 📌 Metrics collection: Prometheus, Datadog, CloudWatch etc. 📌Alerting: PagerDuty, Opsgenie etc. 📌Visualization: Grafana, Kibana etc. 📌Log monitoring: Splunk, Loki etc. #tech #blog #devops #observability #monitoring #alerts

“If you haven’t mapped your dependencies, you haven’t mapped your risk.” Because even your most vetted vendor might be your weakest unseen exposure. “The weakest link isn’t always external. Sometimes, it’s the one you trust most.” Yesterday’s compliant partner might not be ready for today’s threat landscape. 📖 STORY: One Vendor. One Missed Patch. One Costly Incident. A critical infrastructure operator recently experienced a brief but high-impact shutdown. The trigger? A third-party supplier had remote access for routine maintenance. But their endpoint hadn’t been patched in over six months. No malware. No breach. Just unmonitored access in a flat network. And just like that, resilience took a hit. 🛑 THE REAL RISK: Shadow Dependencies You can’t mitigate what you don’t see. 🔸 Outdated vendor infrastructure 🔸 Overlapping credentials across suppliers 🔸 No security validation on updates 🔸 Zero visibility into multi-tier dependencies This isn’t just third-party, it's nth-party risk. And when something breaks, you’re the one holding the fallout. 💡 INSIGHT: True Security Posture = Internal + External + Invisible We’ve seen this pattern across OT, IT, and IoT environments. The strongest teams do things differently: ✅ They map integration points not just assets ✅ They validate access controls in real time ✅ They track supplier risk with live dashboards ✅ They treat vendor reviews as a security control, not a formality 🔄 MINDSET SHIFT ❌ “They passed our audit.” ✅ “Audit is history. Visibility is reality.” ❌ “We trust them.” ✅ “Trust is verified continuously.” ✅ TAKEAWAYS 🔸 Run third-party dependency reviews like you run internal assessments 🔸 Extend visibility beyond your walls into supplier ecosystems 🔸 Include vendor breakdowns in red-team scenarios 🔸 Shift from contract confidence to operational assurance 📩 CTA Want to find out which vendors are silently raising your risk profile? DM me for Microminder’s Supply Chain Risk Mapping Kit the same toolset used across infrastructure, healthcare, F&B, and manufacturing to cut external risk without slowing the business. 👇 What’s the biggest “invisible risk” you’ve uncovered? #CyberLeadership #VendorRisk #Microminder #SupplyChainSecurity #OperationalResilience #ThirdPartyRisk #CISO #RiskMapping #ResilienceByDesign #SecurityEcosystem

🔷 Day 14: Reinforcement Learning in 5G Resource Allocation Optimizing spectrum, power, and scheduling through AI that learns from the network itself. 📌 Why Reinforcement Learning (RL) in 5G? Unlike supervised models that rely on labeled data, RL uses trial-and-error — learning from its environment through feedback (rewards). 5G resource allocation is dynamic and context-aware — RL fits perfectly. 📌 Key Resource Challenges in 5G NR Scheduling PRBs under ultra-low latency constraints Power control in dense small cell environments Mobility and handover management Interference-aware resource reuse Slice-specific QoS assurance 📌 How RL Solves These Agent: The network function (e.g., scheduler, SMO, RIC) State: Network KPIs like CQI, buffer size, UE mobility, demand Action: Allocate PRBs, select MCS, adjust transmit power Reward: Higher throughput, lower latency, reduced packet drop Over time, the RL agent learns to take optimal actions to maximize overall network performance. 📌 Practical Use Cases We Covered Dynamic PRB scheduling in congested cells Beam selection based on prior user movement patterns RAN slicing with real-time policy enforcement Intelligent power allocation to balance SINR across users 📌 What Makes RL Ideal for 5G? Operates in real-time environments Learns from unpredictable user behavior Scales across multi-agent setups (e.g., CU-DU split) Adapts to dynamic interference and load patterns 📘 Technical References ITU-T Y.3173 – Framework for ML in future networks O-RAN WG2 – Near-RT RIC AI Training & Inference 3GPP TR 38.891 – Study on AI/ML for 5G NR #5G #AIin5G #ReinforcementLearning #RANOptimization #5GNR #O_RAN #TelecomAI #NitinGupta #Day14 #ResourceAllocation #RIC #SON #5GTraining #WhatsAppLearning

The $6,459 Terraform Lesson: Why Infrastructure Lifecycle Monitoring Matters Our Terraform module CI pipeline failed to clean up test infrastructure, leaving it running since July 2023. I discovered it 28 months later whilst using Claude Code to enhance our AWS integrations; those forgotten db.t3.micro RDS instances and t2.micro EC2 instances behind an ALB had quietly accumulated $6,459 in costs. The breakdown tells an interesting story: the IPv4 charges ($833) actually exceeded the instance costs themselves, reflecting real IPv4 exhaustion economics that incentivised better resource management. The CloudWatch MetricStreamUsage charges ($1,149) were particularly painful, reinforcing that I was right to drive the multiplexing enhancement I was building when I found this mess. This represents a classic infrastructure failure mode: integration tests meant to be ephemeral cattle became expensive pets when terraform destroy silently failed. We had comprehensive application monitoring but no infrastructure lifecycle observability: no alerts for forgotten resources, no granular monthly cost reviews to catch a steady $250/month burn rate. It's team dinner money, not mortgage money, so usually not worth worrying about, unless you stumble into it by accident. The real irony? I only discovered this whilst investigating why our terraform module CI was sluggish, looking for which account was provisioning real RDS instances. Sometimes the best monitoring tool is curiosity combined with the right observability practices. "Huh, WTF is that doing there?!!!" Key takeaways: * Always instrument your infrastructure lifecycle, not just applications * Cost monitoring needs to be granular enough to catch persistent anomalies * Failed cleanup operations are silent failures that compound over time * IPv4 scarcity pricing is actually driving better resource hygiene The savings from this will pay for my team's very nice post-Toronto o11y_x event dinner and then quite a bit more, but the lesson about infrastructure as code reliability was priceless (please learn from our mistake!). When you're managing ephemeral infrastructure at scale, a failed teardown is just as critical as failed deployment.

🔹 Why Monitoring Matters Kubernetes is dynamic (pods come and go), so static monitoring doesn’t work. Monitoring helps ensure cluster health, app performance, and cost efficiency. Critical for debugging, capacity planning, and alerting. 🔹 What to Monitor Cluster Level: - Node health (CPU, memory, disk, network). - Control plane components (API server, etcd, scheduler, controller manager). Pod/Container Level: - Resource usage per pod/container. - Restarts, crash loops, OOM kills. Application Level: - Response times, error rates, request counts. - Business KPIs (custom metrics). Networking: - Latency, dropped packets, failed connections. - Service-to-service traffic flows. Events & Logs: - Kubernetes Events (pod eviction, scheduling failures). - Logs from apps and system components. 🔹 Monitoring Workflow - Collect (metrics, logs, traces via agents like Prometheus + Fluentbit). - Store (time-series DB like Prometheus, logs in Elasticsearch/Loki). - Visualize (dashboards in Grafana/Kibana). - Alert (via Alertmanager, PagerDuty, Slack, email). - Act (debug issues, scale workloads, adjust configs). 🔹 Best Practices - Always monitor control plane health first. - Set resource requests/limits and monitor usage. - Monitor SLIs/SLOs (latency, error rate, uptime). - Enable log aggregation (so pod restarts don’t lose logs). - Use black-box monitoring (synthetic tests for availability). - Keep dashboards + alerts simple (avoid alert fatigue). # Hashtags for Visibility #DevOps #InterviewPreparation #Kubernetes #Docker #CloudComputing #TechCareers #InfrastructureAsCode #CareerGrowth #Monitoring #CICD #Terraform. #Azure #Aws #Gcp #Software #linkedin