Decision File / Security · Sample
SIEM Replacement: When Selection Criteria Fail to Differentiate
- Who
- A CISO at a payments company
- The decision
- Deciding on a SIEM replacement.
- At stake
- Budget constraints made defending a 28% increase to the CFO untenable.
- They chose
- Dana chose NightOwl mainly for its 3am support despite lacking features.
- What happened
- Migration cost $110k and took six months instead of two due to unforeseen complications.
- Core lesson
- Ensure all critical factors are properly weighted to avoid them becoming de facto decision points later.
What we said vs. what happened
They weighted support at 10%, but the 3am named engineer decided the winner; higher-weighted items either passed for everyone or just knocked one vendor out.
| What we said we’d judge on | Weight | What actually happened | |
|---|---|---|---|
| Threat detection coverage | 30% | All four could detect everything we cared about — it didn’t separate anyone. | |
| Integrations with our existing stack | 20% | Sentrix had 9 of 12; the card switch was “on the roadmap.” The others had all 12 day one. | |
| Query performance at our volume | 15% | Corvid slowed badly on 90-day lookback searches; the others handled our volume fine. | |
| Price | 25% | Price knocked out exactly one vendor — the incumbent. The other three landed within about $30k of each other over three years. | |
| Support | 10% | We bought the 3am phone call — a named engineer in contract. That decided it. | |
| Will the analysts actually use it daily | not on the list | Our SOC stopped opening Sentrix two weeks into the trial — the UI didn’t fit queue-clearing work. |
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (7 quotes)
“We did it properly, or so we thought. April, we wrote a scorecard before we talked to a single vendor.”
“All four could detect everything we cared about.”
“We needed twelve connections… LogHarbor, Corvid and NightOwl had all twelve… Sentrix had nine of twelve. The card switch connector was 'on the roadmap.'”
“Corvid slowed badly on 90-day lookback searches… LogHarbor, Sentrix, NightOwl all handled our volume without complaint.”
“Price knocked out exactly one vendor — LogHarbor. The other three landed within about 30 thousand of each other on a three-year view.”
“The honest truth is we bought the 3am phone call.”
“Sentrix, the analyst darling, our SOC guys just stopped opening it two weeks into the trial.”
Feature comparison
Only NightOwl offered a named 3am engineer at the budgeted tier; that single line decided it.
| Can it do this? | LogHarbor (incumbent) | Sentrix | Corvid | NightOwl |
|---|---|---|---|---|
| A named engineer we can call at 3am | ||||
| Has all 12 integrations we need (incl. card switch) | ||||
| 90-day lookback query performance during incidents |
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (3 quotes)
“When this breaks at 3am during an incident, who picks up the phone?”
“LogHarbor, Corvid and NightOwl had all twelve, day one. Sentrix had nine of twelve.”
“Corvid slowed badly on 90-day lookback searches.”
Decision timeline
Nine months of trials and comparisons ended with one question — “who picks up at 3am?” — and then migration ran four months long.
Renewal quote jumps 28%
Pricing model changed; at our volume it was untenable, so we went to market.
Head-to-head evaluation and trials
Two full head-to-head trials; ~60 hours of team time.
Seriously considered building it
Modeled an open-source stack but killed it due to staffing reality.
The 3am question decides it
One meeting — only NightOwl had a named 3am engineer at our tier.
Migration ran long
Took six months instead of two; four months late.
Go-live on NightOwl
Live in May instead of January.
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (5 quotes)
“The renewal quote… was 28 percent higher than the year before.”
“Nine months. Two full head-to-head trials…”
“We did, seriously, for about six weeks in the middle of the evaluation.”
“When this breaks at 3am during an incident, who picks up the phone?”
“It cost about 110, and it took six months — we went live in May instead of January.”
Who dropped out, and why
The incumbent fell on price; Sentrix fell on real-world use; Corvid fell on support at budget; NightOwl won for the 3am named engineer.
| LogHarbor (incumbent) | Too expensive | 28% renewal hike and pricing model change made it untenable; price knocked out exactly one vendor — the incumbent. | |
| Sentrix | Nobody used it | Analysts stopped opening it two weeks into the trial; UI fit threat hunting, not queue clearing.Also missing the card switch connector and no named 3am engineer. | |
| Corvid | Above budget for support | A named 3am engineer required Enterprise Plus (+$95k/yr); also slowed on 90-day lookbacks. | |
| NightOwl | Chosen | Included a named on-call engineer at our tier — we bought the 3am phone call. |
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (4 quotes)
“Price knocked out exactly one vendor — LogHarbor.”
“Our SOC guys just stopped opening it two weeks into the trial.”
“Corvid — yes, but only on the Enterprise Plus tier, which was 95 thousand a year above what we'd budgeted.”
“The honest truth is we bought the 3am phone call.”
What the money actually buys
Versus the cheapest finalist, they paid about $75k more over three years — ~$40k effectively for the 3am phone call and ~$35k from a migration overrun.
| What we’re paying to solve | Sentrix | Corvid | NightOwl (chosen) |
|---|---|---|---|
| A named engineer we can call at 3amThis line decided it. | not offered | +$95,000/yr | included |
| Migration overrunRan 4 months long | n/a | n/a | +$35,000 (over plan) |
| Three-year total | $735,000 | $765,000 | $810,000 |
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (3 quotes)
“Versus the cheapest finalist we paid about 75 thousand more over three years. 40 of that was the 3am phone call… The other 35 was the migration overrun…”
“Sentrix came to roughly 735 total over three years, Corvid 765, NightOwl 810.”
“Corvid — yes, but only on the Enterprise Plus tier, which was 95 thousand a year above what we'd budgeted. NightOwl — yes, included.”
What we skipped
Skipping reference calls is what bit them — a few calls would have surfaced the 4-month migration overrun risk.
| Formal RFP | Skipped | Cost nothing — we went straight to vendors on procurement’s approved list. | |
| Second trial round across all business units | Cut to 2 weeks in one unit | No specific damage called out; renewal deadline was breathing on us. | |
| Security review | Kept in full | Avoided issues there — nobody was willing to shorten it, and I’m glad. | |
| Reference calls | Skipped | Three phone calls would have saved a four-month migration overrun surprise. |
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (2 quotes)
“Our normal process has a formal RFP — we skipped it… The one that actually cost us: reference calls.”
“It cost about 110, and it took six months — we went live in May instead of January.”
Stakeholder map
The CFO set the constraint, the SOC decided more than the scorecard did, and nobody blocked — the decisive vote came from analysts who simply stopped opening a product.
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (3 quotes)
“Sentrix, the analyst darling, our SOC guys just stopped opening it two weeks into the trial.”
“We went straight to vendors on procurement's approved list.”
“The honest truth is we bought the 3am phone call.”
The CFO view — what the premium buys back
The 3am line cost ~$25k a year. The only other vendor offering it priced the same line at $95k a year — against the real alternative, NightOwl's premium pays back in under a year.
| The path | 3-year cost | The 3am line |
|---|---|---|
| Sentrix — the cheapest finalistThe baseline every premium is measured against | $735,000 | not offered at any price |
| Corvid + Enterprise PlusThe only other way to buy a named 3am engineer | $1,050,000 | +$95,000/yr |
| NightOwl — chosenIncluding the $35k migration overrun | $810,000 | included |
| Premium over cheapest — vs. saving against the real alternative | +$75,000 | −$240,000 |
Source: Dana Whitfield (CISO) and team — interview transcript.
Read what Dana said (3 quotes)
“Versus the cheapest finalist we paid about 75 thousand more over three years. 40 of that was the 3am phone call… The other 35 was the migration overrun…”
“Corvid — yes, but only on the Enterprise Plus tier, which was 95 thousand a year above what we'd budgeted. NightOwl — yes, included.”
“Sentrix came to roughly 735 total over three years, Corvid 765, NightOwl 810.”
How they decided
Ran a nine-month evaluation including trials and created a scorecard.
ConsultedSOC leads · Security team
SkippedSkipped reference calls and a full RFP process.
What they’d do differently
Score migration effort as a criterion and conduct reference calls before deciding.
Unknowns — what one case can’t tell you
This case does not indicate whether NightOwl will continue to meet needs long-term.
“Ensure all critical factors are properly weighted to avoid them becoming de facto decision points later.”
— Dana Whitfield, CISO (illustrative subject)Methodology In the interactive version, underlined claims open the interview moment behind them — a video receipt, timestamped to the words above. Try it in the interactive report ↗
Sample disclosure Generated by UpSight to demonstrate what a Decision File looks like end-to-end — brief, six diagrams, evidence, narrative — from a seed interview transcript. Not one of the real files in the published series; real files use real subjects and subject-approved quotes.
Download PDF