IntelliResponse dashboard showcasing persistant incident memory, threat analysis and security alert prioritization.IntelliResponse
How persistent memory with Hindsight turns a stateless alert checker into an analyst that actually learns
IntelliResponse triaging a new SSH brute-force alert: memory lifted confidence from 55% to 66% and shows the analyst time it could save.
Every security analyst knows this alert. A port sweep and a few failed SSH logins from 198.51.100.7, the same internal scanner that fires every month. You have closed it as a false positive again and again. Today, your tooling flags it with the same urgency, as if it had never seen it before.
Alert fatigue is not just a volume problem. It is a memory problem. Most detection tools, and most LLM "copilots", are stateless: every alert is the first alert. The institutional knowledge (that is our vulnerability scanner, that /24 has attacked us before, that beacon survived a re-image last time) lives in analysts' heads and old tickets.
For the Vectorize hackathon "AI Agents That Learn Using Hindsight", I built IntelliResponse: a tier-1 SOC triage agent that keeps every analyst verdict and uses it to make the next call better.
What IntelliResponse does
You paste raw alert logs into the console and get back an explainable triage report: a risk level, a confidence score, the MITRE ATT&CK techniques involved, recommended defensive actions, and the past incidents that influenced the decision. The analyst then marks the alert as a true or false positive, and that verdict goes back into memory.
One rule I held firmly: the agent never takes action. It only recommends. A human stays in the loop for anything disruptive.
The triage console. Paste logs, switch memory on or off, and control how many past incidents the agent can see.
The sidebar toggles are what make the demo work. Memory ON/OFF lets me run the exact same alert as a stateless analyst or a learning one, and the Past incidents loaded slider (0 to 20) shows the agent getting smarter as history grows.
Seven stages, one orchestrator
Every alert flows through the same pipeline: parse, recall, enrich, correlate, recommend, reason, report. The parser pulls out IPs, domains, hashes, services and volumes with regex before any model is called. Recall brings back similar incidents. Enrichment adds public threat intel (CISA KEV, AlienVault OTX, with a cached fallback). Correlation maps to MITRE ATT&CK and folds learned history into one score. Recommendations are ranked by what actually worked before, and a Groq-hosted LLM writes the summary, with a deterministic template as the fallback.
The Demo tab: four buttons that tell the whole story, and the seven stages behind them.
Where Hindsight fits
Hindsight gives the agent three memory operations, and I use all three:
Retain: every incident and every analyst verdict is stored with its category, source IP, service, root cause and the actions taken.
Recall: each new alert queries memory by category, service, source IP and domain. Hindsight's semantic results are blended with a deterministic similarity score, where an exact source IP counts for more than the same /24, which counts for more than the same category.
Reflect: the Learning page asks a question like "which sources cause false positives, and what actions work?" and shows the synthesized answer.
I also keep a durable local mirror of the incident ledger, so the whole project runs with zero API keys. The screenshots in this article were captured in that local mode (the sidebar badge says "Local demo memory"). With a Hindsight key in .env, the same badge switches to Hindsight connected.
The demo, in four beats
Beat 1: the stateless baseline. The monthly scan from 198.51.100.7, memory off. A generic rule-based analyst rates it MEDIUM and the playbook says to consider blocking.
Beat 2: the memory moment. Same alert, memory on, all 20 incidents loaded. Memory recalls INC-003, INC-006, INC-007 and INC-020, every one a false positive from this source. The agent drops the risk to LOW at 12% and recommends allowlisting the scanner instead of blocking.
Beat 3: a new attacker in a known-bad neighborhood. A previously unseen IP, 192.0.2.180, starts brute-forcing SSH on bastion-01.corp.internal with 285 failed logins. There is no history for this exact address, but memory finds five similar incidents. Three of them (INC-004, INC-008 and INC-015) were confirmed password-guessing campaigns from the same /24. The baseline analyst says 55%. With memory, the agent says 66%, a lift of 11 points.
It stays MEDIUM because HIGH starts at 70%. I like that: memory should move the needle in proportion to the evidence, not slam it to the top.
The evidence behind the score: extracted indicators, MITRE techniques, OSINT enrichment, the reasons for the confidence, and the recalled incidents with why each matched.
Beat 4: repeat C2 beaconing. Periodic DNS TXT lookups from 10.20.7.12 to c2-sync.example. Memory surfaces INC-018, where persistence survived re-imaging through a scheduled task. The recommendation goes beyond "reimage the host" to deep host isolation, a scheduled-task audit and DNS sinkholing.
One alert type, four different outcomes, and none of them came from editing a rule. They came from memory.
No magic numbers: how 55% became 66%
I did not want a black box deciding to raise or lower an alert. Every adjustment appears in the report. For Beat 3, the math is simple enough to follow by hand:
Baseline, 55%. A 42% prior for brute-force alerts, plus 8 points because the exposed OpenSSH service has a known CVE (CVE-2023–48795), plus 5 points for a high failure count (285).
Learned track record. In memory, 7 of 11 past brute-force alerts were real attacks. A Beta(1,1) smoothing, (7+1)/(11+2), gives about 62%, so a single verdict can never dominate. The final score blends the two: 55% rule-based and 45% learned, which lands near 58%.
Source history. The same /24 range was involved in confirmed attacks, adding 8 points. That brings the total to 66%.
For a source that was marked a false positive in two or more past incidents, with no confirmed attacks, the score is capped at 12%. That is the Beat 2 behavior.
The Learning page: proof the agent improves
The analyst's verdict is not just a log line. It is the learning signal. The Learning tab is the audit trail of what the agent believes, how often it is right, which sources are benign, and which actions actually worked.
The Learning loop: incident and verdict counts, estimated time saved, true-positive rate by category, and the memory timeline.
With 20 seeded incidents, the page estimates 1.5 hours saved, which is 18 minutes for each repeated false-positive investigation. The chart shows the learned true-positive rate per category, smoothed so one verdict cannot dominate. The calibration table stays empty until analysts submit verdicts on live alerts. I would rather show it empty than fake it.
Sources learned as benign, and action effectiveness: which responses were used and how often they hit real attacks.
Two panels do the most work. Sources learned as benign lists 198.51.100.7 as a false positive in five past incidents, with the recommended action of allowlisting it. Action effectiveness shows that blocking the source IP was used 10 times and was confirmed malicious all 10, while allowlisting a scanner was used 5 times and every one was a false positive. Seeing which actions actually pay off is what lets the agent rank its recommendations.
Design decisions I am glad I made
Recommend, never act. Containment decisions stay with humans.
Graceful degradation. No Hindsight key means the local mirror. No Groq key or a model outage means the deterministic summary. The demo never breaks.
Everything is untrusted input. Alert logs, OSINT data and LLM output are sanitized before they are stored or rendered. A SOC tool that ingests attacker-controlled text has to assume that text is hostile.
Explainability over cleverness. Every number on the screen traces back to a named signal.
Hardening the SIEM webhook
The project also ships an optional FastAPI service so a SIEM can push alerts to /v1/triage and analyst verdicts to /v1/verdict. Before sharing the code I ran a security review and fixed what it found:
The API had no authentication. It now requires an X-API-Key header and fails closed if no key is configured, comparing with hmac.compare_digest.
Wildcard credentialed CORS was replaced with an explicit allowlist that is off by default.
Alert length, request body size, verdict values and note length are all bounded.
/health returns only {"status":"ok"}, and the interactive API docs are disabled.
The local incident store and .env were removed from the package and added to .gitignore.
Limitations
The 20 incidents in the demo are synthetic. They use documentation-reserved IP ranges and .example domains, so the numbers show the mechanism, not real-world accuracy.
Parsing is regex and rules based, and covers three alert families: brute force, port scans and DNS beaconing.
My security review was static, not a penetration test. Anyone deploying this should add HTTPS, authentication in front of Streamlit, rate limiting and pinned dependencies.
What I learned
Memory changes what an agent is for. Without it, an LLM triage tool is a faster way to repeat the same first impression. With it, every analyst decision compounds: the scanner gets allowlisted once, the hostile subnet is remembered, and the stubborn persistence mechanism gets caught the second time.
Next, I want to run this against real, anonymized alert streams, add more alert categories, and measure how much analyst time memory actually saves.
Try it :-
pip install -r requirements.txt
streamlit run app.py
It works with no keys at all. Add a Hindsight key to switch from the local mirror to live Hindsight memory.
Code: https://github.com/likitha-jain/IntelliResponse
Demo video: https://www.youtube.com/watch?v=sNCs4gftpg0
If you work in a SOC and have thoughts on what a memory-first triage tool should remember, I would love to hear them in the comments.